Moderation System Architecture

Overview

The moderation system provides automated content moderation for user-generated content including chat messages and reviews. It uses a multi-layered approach combining OpenAI's Moderation API, custom LLM analysis, and Google's Perspective API to ensure content safety while minimizing false positives.

System Flow

flowchart TD
    A[Message Created] --> B[InstantDB Subscriber]
    B --> C{Already Moderated?}
    C -->|Yes| Z[Skip]
    C -->|No| D[Add to Moderation Queue]
    D --> E[ChatModerationProcessor]
    E --> F[ModerationService.runModeration]
    F --> G[OpenAI Moderation API]
    G --> H{Flagged?}
    
    H -->|No| I[Enqueue Notification]
    I --> J[Mark as Moderated]
    
    H -->|Yes| K[LLM Flagging Analysis]
    K --> L[Determine Action]
    L --> M{Room Type?}
    M -->|Category Room| N[Perspective API Scores]
    M -->|Group/Private| O[LLM Sentiment Analysis]
    N --> P[Update Message with Results]
    O --> P
    P --> Q[Skip Notification]

Components

1. InstantDB Subscriber Service

Location: server/src/instantdb/subscriber.service.ts

Subscribes to real-time message updates from InstantDB and queues unmoderated messages for processing.

Responsibility	Description
Subscription Management	Creates and maintains WebSocket subscriptions to InstantDB
Idempotency	Skips already-moderated messages
Job Deduplication	Uses messageId as BullMQ jobId to prevent duplicates
Health Monitoring	Cron job checks subscription health every minute

2. Moderation Service

Location: server/src/moderation/moderation.service.ts

Orchestrates the moderation pipeline and determines final actions.

async runModeration(type: ModerationType, data: any): Promise<ModerationResultDto>

Moderation Types:

• CHAT - Chat messages
• REVIEW - Product/service reviews

3. Analysis Service

Location: server/src/moderation/analysis.service.ts

Provides AI/ML analysis capabilities:

Method	Provider	Purpose
checkOpenAIModeration()	OpenAI	Binary flag detection (fast, first-pass)
getFlaggingDetailsFromLLM()	OpenRouter	Detailed severity analysis for flagged content
getSentimentFromLLM()	OpenRouter	Sentiment scoring (-1 to 1)
getPerspectiveScores()	Google Perspective API	Toxicity, profanity, and sexual content scores

4. Queue Processors

Location: server/src/moderation/processors/

Processor	Queue	Job Type
ChatModerationProcessor	chat-moderation	MODERATE_CHAT
ReviewModerationProcessor	review-moderation	MODERATE_REVIEW
CommentModerationProcessor	comment-moderation	MODERATE_COMMENT

Decision Flow

Step 1: OpenAI Moderation (Gate)

All content first passes through OpenAI's Moderation API. This is a fast, binary check.

• Not Flagged: Immediately enqueue notification, skip detailed analysis
• Flagged: Proceed to detailed LLM analysis

Step 2: LLM Flagging Analysis

For flagged content, the LLM provides:

• reason: Human-readable explanation
• recommendedAction: RETAIN, WARN, or REMOVE
• needsHumanReview: Boolean for edge cases

Step 3: Context-Specific Analysis

Additional analysis based on content type and context:

Context	Analysis
Category Room Chat	Perspective API (toxicity, profanity, sexual)
Group/Private Chat	LLM Sentiment Analysis
Reviews	LLM Sentiment + Perspective API

Data Model

ModerationResultDto

{
  messageId?: string;
  reviewId?: string;
  isFlagged: boolean;
  action: ModerationAction;
  needsHumanReview: boolean;
  moderationReason?: string;
  sentimentScore?: number;
  sentimentLabel?: SentimentLabel;
  toxicityScore?: number;
  profanityScore?: number;
  sexualScore?: number;
  moderatedAt: Date;
}

ModerationAction Enum

• RETAIN - Content is safe
• WARN - Content is borderline, may need review
• REMOVE - Content should be hidden/removed

SentimentLabel Enum

• POSITIVE
• NEUTRAL
• NEGATIVE

Configuration

Environment variables:

Variable	Description
OPENAI_API_KEY	OpenAI API key for moderation
OPENROUTER_API_KEY	OpenRouter API key for LLM
OPENROUTER_BASE_URL	OpenRouter base URL
PERSPECTIVE_API_KEY	Google Perspective API key
MODERATION_LLM_MODEL	Model ID for LLM analysis
MODERATION_SENTIMENT_PROMPT	Prompt template for sentiment
MODERATION_FLAGGING_PROMPT	Prompt template for flagging

Moderation Flow Implementation & Cleanup

Implemented a robust, sequential moderation-to-notification flow and performed comprehensive code cleanup.

Changes Made

1. Moderation→Notification Flow Restructure

• Sequential Processing: Modified ModerationService to trigger notifications only after OpenAI moderation passes.
• Notification Gate: OpenAI moderation acts as the first filter. If content is flagged, detailed LLM analysis runs but notifications are skipped.
• Decoupled Subscriber: Removed notification logic from instantdb.service.ts and subscriber.service.ts, making them purely responsible for queuing moderation.
• Idempotency: Leveraged BullMQ jobId for robust deduplication.

2. Code Cleanup & Optimization

• Debug Logs: Removed all this.logger.debug statements from moderation-related services.
• Console Logs: Removed console.log statements from analysis.service.ts.
• Comment Stripping: Removed all JSDoc and internal comments from:
  • moderation.service.ts
  • analysis.service.ts
  • instantdb.service.ts
• Protocol Fix: Verified and restored URLs corrupted during comment stripping.

Verification Results

Integration Flow

sequenceDiagram
    participant IDB as InstantDB
    participant SUB as Subscriber
    participant MOD as ModerationService
    participant NOTIF as NotificationService

    IDB->>SUB: New Message (moderated: false)
    SUB->>MOD: Add Moderation Job
    MOD->>MOD: OpenAI Moderation Check
    alt Content is Clean
        MOD->>NOTIF: Enqueue Notification Job
        MOD->>IDB: Mark as Moderated (isFlagged: false)
    else Content is Flagged
        MOD->>MOD: LLM Severity Analysis
        MOD->>IDB: Mark as Moderated (isFlagged: true)
        Note over MOD: Skips Notification
    end

Clean Code Verification

Ran grep checks confirming 0 instances of debug logs and comments in the modified service files.

Queue Configuration

Moderation queues are configured with:

• removeOnComplete: false (for debugging/audit)
• attempts: 3 with exponential backoff
• jobId: chat-mod-{messageId} for deduplication

Integration Points

Inbound

• InstantDB Subscriptions: Real-time message stream
• Direct API Calls: For batch moderation

Outbound

• Notification Service: Enqueues notifications for clean content
• InstantDB Updates: Marks messages as moderated with results

Notification Flow

Notifications are only sent for content that passes OpenAI moderation:

1. OpenAI check returns !flagged
2. enqueueNotificationForChat() fetches message details
3. Validates sender/room metadata
4. Queues notification job
5. Marks message as notificationProcessedAt

Flagged content skips notification entirely.