Moderation pipeline

Why a two-stage pipeline

Clients never write directly to poems. Every new poem goes to pendingPoems: a staging collection invisible to the rest of the app. From there a Cloud Function hands it to Gemini 2.5 Flash and decides whether to promote it or quarantine it. Same logic for comments and profile photos, with different triggers but the same contract.

Splitting staging from publication buys two concrete things: clients never see unapproved content, and Firestore rules can outright forbid direct writes to poems.

Poem flow

client → pendingPoems/{id}
            ↓ onDocumentCreated
         Gemini moderation
         ├─ approved → poems/{id} (moderationPassed: true)
         │              ↓ sendNewPostNotification (follower push)
         └─ blocked  → quarantine + author notification ("contentHidden")

The sendNewPostNotification trigger also fires on poems/{poemId}, but it checks moderationPassed === true before proceeding. This lets moderators force a manual publish without bypassing notifications.

Explicit fail-open

If Gemini doesn’t answer — outage, quota, timeout — the poem is published anyway, but tagged moderationDeferred: true with a timestamp. A user shouldn’t lose content because of infrastructure problems they didn’t cause; admins have a separate flag from moderationPassed to schedule targeted retries, and older clients just ignore the extra field.

The opposite stance — fail-safe — is reserved for images: if a photo can’t be downloaded for analysis, it’s blocked. Letting an unseen image through is riskier than losing one publication.

Moderator override

The moderationOverride: true flag on pendingPoems signals a human approval already happened: the function skips the Gemini call and promotes directly. Used to republish false positives without paying the AI call twice and without risking a different verdict on the second pass.

Comments and profile photos

• Comments — same pattern: written to pending, evaluated, promoted or deleted. The recipient notification only fires after promotion (moderateComment explicitly defers the sendCommentNotification trigger).
• Profile photos — HTTP endpoint called by the client right after the Storage upload. The function downloads the bytes directly from the bucket and feeds them to Gemini as inlineData, with no intermediate public URL. There’s also a moderateLegacyProfilePhoto trigger on users/{uid} to catch direct photoURL changes from older clients.

Human review

The admin panel and user reports plug into the same pipeline:

• Every moderation event writes a log queryable by moderators.
• notifyModerationOutcome messages the author in-app when the verdict flips (e.g. a previously blocked poem gets reinstated).
• notifyModeratorsNewReport alerts moderators when a user report comes in.

The result is a system where Gemini filters the volume, and humans only see ambiguous or reported cases.