PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Foundation Models Moderate Online Content? Evaluating Instruction- vs. Example-Driven Policy Operationalization

Ayan Majumdar, Shounak Paul, Pushpdeep Singh, Ines Abdelaziz, Sayeh Jarollahi, Seungeon Lee, Krishna P. Gummadi, Ingmar Weber, Abhisek Dash

arXiv:2609.10410v1cs.CLcs.AIcs.CY

TL;DR

Complex, platform-specific moderation policies raise the question of whether foundation models can reliably operationalize them. The paper compares instruction- and example-driven VLM guidance on a manually annotated Bluesky benchmark and finds that both can outperform the deployed system, while instruction-driven moderation appears more scalable.

  • Problem

    Complex moderation policies challenge consistent operationalization, while foundation models’ ability to reliably enforce platform norms remains unresolved.

  • Method

    The paper introduces MODERATIONBENCH and systematically compares instruction-driven and example-driven VLM moderation on manually annotated, multimodal Bluesky posts.

  • Results

    0.60 vs. 0.22 F1: foundation models nearly triple Bluesky’s deployed moderation-system score on Random Posts, with the two guidance paradigms reaching comparable peak effectiveness.

  • Takeaways & Limitations

    Foundation models can outperform deployed moderation systems, and instruction-driven moderation appears more suitable for scaling to platform deployment.

  • Takeaways & Limitations

    The study does not assess whether differing VLM policy interpretations produce systematic demographic biases in moderation outcomes.

Abstract

from arXiv · show

The growing complexity of content moderation policies presents a critical challenge for their consistent operationalization. While foundation models possess the basic capabilities needed to confront this challenge, whether they can reliably moderate online content remains an unanswered question. In this paper, we systematically compare two competing paradigms for Vision-Language Model (VLM) guidance: an instruction-driven approach where models reason from policy precepts, and an example-driven approach where they generalize from prior precedents. We ground this investigation in ModerationBench, a new benchmark of 4,000 manually annotated, in-the-wild posts from the Bluesky platform. Our experiments reveal that foundation models can substantially outperform Bluesky's deployed moderation system, nearly tripling its $F_1$ score (0.60 vs. 0.22) on Random Posts in the benchmark, with both instruction- and example-driven paradigms achieving comparable peak effectiveness. Our findings thus chart a path toward reliable and adaptable policy operationalization at scale.

1 Introduction

Content moderation policies have become complex enough to strain human moderators and expose weaknesses in brittle automated systems. This paper evaluates whether foundation-model VLMs can operationalize such policies through instructions or prior examples.

  • Content moderation policies span labels, rationale, and enforcement details, creating an unsustainable cognitive load for human moderators.
  • Foundation models offer richer contextual reasoning than brittle pattern-matching systems, but their reliable use for high-stakes policy enforcement remains an open question.
  • MODERATIONBENCH is a Bluesky-grounded benchmark of multimodal posts manually labeled to reflect human operationalization of the platform’s policies.
  • The study compares instruction-driven moderation from policy frameworks with example-driven moderation that generalizes from random, prototypical, or contextual examples.
  • 0.60 vs. 0.22 F1: instruction-driven foundation models nearly triple Bluesky’s deployed moderation-system score on Random Posts.

2 Related Work

Prior work describes increasingly complex, platform-specific moderation policies and brittle automation, while existing benchmarks leave a gap in evaluating multimodal moderation against deployed systems. This paper addresses that gap by comparing policy instructions with example decisions.

  • Moderation policies vary across platforms, have grown increasingly complex, and increasingly require automation at scale.
  • Existing rule-based moderation pipelines remain brittle, motivating the question of whether foundation models can moderate online content.
  • The paper frames policy operationalization as translating abstract policy goals into concrete moderation decisions.
  • Its two guidance paradigms correspond to codified policy instructions and precedent-based example decisions.
  • Earlier benchmarks targeted narrow harms or modalities, while MODERATIONBENCH uses in-the-wild content and enables direct comparison with a deployed platform system.

3 Curating MODERATIONBENCH

MODERATIONBENCH samples multimodal Bluesky posts across four evaluation settings designed to test detection in natural traffic, recovery of missed harms, and false-positive avoidance. Every post receives independent human safety and harm-label annotations.

  • Bluesky data collection draws on the firehose of platform posts and the label stream of moderation decisions.
  • The benchmark contains four multimodal subsets: Random Posts, Moderated Posts, Near-moderated Posts, and Safe Posts.
  • Dataset subsets: Random Posts preserve natural content distribution, while Moderated Posts sample 1,000 BMS-labeled posts balanced across nine categories.
  • Dataset subsets: Near-moderated Posts select 1,000 unmoderated posts most semantically similar to moderated content to identify potentially missed harmful material.
  • Dataset subsets: Safe Posts sample 1,000 posts from verified organizational accounts where harmful content is expected to be rare, testing false-positive risk.
  • Annotation: Human annotation covers complete multimodal content, assigns safety and harm labels, resolves disagreements through a third annotator, and achieves Cohen’s κ = 0.813.

4 Instruction(Code)-driven Moderation

The section evaluates instruction-driven VLM moderation using progressively granular Bluesky policy guidance, then examines effectiveness, efficiency, and consistency. Richer policy details improve moderation outcomes, while model disagreements expose distinct policy judgments and unresolved subjectivity.

  • 4.1 What, Why, and How to Moderate?: VLMs receive Bluesky policy guidance along three axes: labels identify harms, rationale conveys principles, and details specify labeling rules.
  • 4.1 What, Why, and How to Moderate?: Four prompt configurations vary policy granularity: What, What & Why, What & How, and What, Why & How.
  • 4.2 Effectiveness of Moderation Decisions: Richer instructions consistently improve F1 on Random Posts; policy rationale helps some models, while policy rules drive higher F1 across all models.
  • 4.2 Effectiveness of Moderation Decisions: Most VLM setups outperform BMS on Random Posts, while open-weight gemma4 and qwen3.5 are selected for subsequent analyses alongside frontier models.
  • 4.2 Effectiveness of Moderation Decisions: Detailed instructions reduce unnecessary flagging, bringing gemma4 and qwen3.5 closer to human flagging levels than label-only prompts.
  • 4.2 Effectiveness of Moderation Decisions: On Near-moderated Posts, VLMs achieve high precision, recall, and F1 while BMS scores zero across all metrics despite annotators judging ≈35% policy-violating.
  • 4.2 Effectiveness of Moderation Decisions: Complete policy guidance processes about 49 gemma4 text+image posts per minute versus ≈61 with labels only, adding modest runtime overhead.
  • 4.3 Consistency in Moderation Decisions: When models agree on labels, policy citations align with ρ = 0.82 and JSD = 0.08; disagreements reduce alignment to ρ = 0.58 and JSD = 0.36.

5 Example(Case)-driven Moderation

The example-driven paradigm guides VLMs with prior moderation decisions and safe posts. Prototypical examples generally perform best, while contextual examples reduce performance and increase overflagging; peak performance is comparable to instruction-driven guidance but less efficient.

  • 5.1 Moderated Content as Examples: Example-driven moderation guides VLMs with prior multimodal moderation decisions and safe posts, using random, prototypical, or contextual example-selection strategies.Prototypical examples are closest to the mean embedding for each label, whereas contextual examples are dynamically selected by semantic similarity.
  • 5.2 Effectiveness of Moderation Decisions: Prototypical examples perform the best across all models on Random Posts.The strongest performance appears across VLMs and setups, including weaker models that outperform BMS in some cases.
  • 5.2 Effectiveness of Moderation Decisions: Contextual examples substantially reduce performance because VLMs overflag content as categories such as rude.The performance drop is attributed to increased flagging rates.
  • 5.2 Effectiveness of Moderation Decisions: With prototypical examples, gemini3.5 remains the best-performing model, while gemma4 is the only open-weight VLM described as effectively handling examples.Other open-weight models, including qwen models, show a performance drop relative to instruction-driven prompting.
  • 5.2 Effectiveness of Moderation Decisions: Both policy paradigms improve moderation beyond label definitions, with best-performing models reaching comparable performance under the two forms of guidance.Textual policy details yield more consistent gains, particularly for open-weight VLMs.
  • 5.2 Effectiveness of Moderation Decisions: Example-driven moderation is substantially less efficient because each query includes multiple multimodal in-context examples.With prototypical examples, gemma4 processes around 10 posts/minute, approximately 6.5× slower than with full policy details.

6 AI Safety Models for Moderation

The paper evaluates specialized AI safety models for transfer to Bluesky moderation. These models underperform corresponding general-purpose VLMs, with fixed-taxonomy specialization producing overflagging, under-flagging, or difficulty adapting to the Bluesky policy.

  • 6 AI Safety Models for Moderation: The study tests whether safety models trained for unsafe prompts and AI-generated responses transfer effectively to multimodal social-media moderation.It evaluates llama-guard4-12B and shieldstral under the Bluesky policy or their native taxonomy, alongside corresponding base models.
  • 6 AI Safety Models for Moderation: Safety models are ineffective for content moderation compared with corresponding general-purpose VLMs.llama-guard performs worse than llama4 across all benchmark subsets, while shieldstral is comparable to mistral on harmful subsets but overflags Random and Safe Posts.
  • 6 AI Safety Models for Moderation: Fixed-taxonomy safety models are difficult to steer, although supplying the Bluesky policy reduces some false positives and improves llama-guard’s overall performance.Despite this adaptation, llama-guard remains consistently worse than its base instruct model across all four subsets.

7 Conclusion

The paper introduces MODERATIONBENCH and compares instruction-driven with example-driven VLM guidance for complex policy operationalization. Both paradigms outperform deployed moderation systems, while instruction-driven guidance appears more scalable, although the study acknowledges limitations.

  • 7 Conclusion: The paper introduces MODERATIONBENCH and conducts a systematic comparison of instruction-driven and example-driven VLM guidance.The benchmark is grounded in the Bluesky platform and supports the study of complex moderation-policy operationalization.
  • 7 Conclusion: Foundation models using both paradigms outperform currently deployed moderation systems.The conclusion reports this outcome without restricting it to a single prompting paradigm.
  • 7 Conclusion: Instruction-driven guidance seems more adept at scaling for deployment at platform scale.The conclusion presents this as a supported comparative finding rather than an absolute guarantee.
  • 7 Conclusion: The study is described as a first step and as having limitations discussed elsewhere.The conclusion nevertheless characterizes it as a foundation for transparent and adaptable platform governance.

Limitations

The study identifies scope boundaries involving unsupported modalities, unexamined bias, missing surrounding context and linked content, absent expert annotations, and untested live-scale deployment under evolving policies.

  • The evaluation excludes video and audio, leaving their effects on VLM moderation performance for future work.The benchmark covers text and images but not these additional modalities.
  • The study does not evaluate whether differing policy interpretations produce systematic demographic biases in moderation outcomes.Potential bias may arise from the VLM, platform policy, or example moderation decisions.
  • Each post is treated atomically, so the evaluation omits conversation threads, external fact-checking sources, and platform-wide activity relevant to some labels.The paper specifically identifies broader context as important for labels such as rude, bullying, and spam.
  • The study does not retrieve or incorporate harmful content located in external links.How linked-content retrieval would shape VLM decisions remains open.
  • ModerationBench lacks expert annotations, particularly limiting evaluation strength for political and other subjective categories.The authors identify expert judgments as a direction for extending the benchmark.
  • The work evaluates throughput on a static dataset but does not assess scalability or effectiveness in live streaming settings or under changing policies.Real-time platform-scale moderation also poses computational and practical challenges.

Ethical Considerations

The study uses publicly available text-and-image posts, in-house models and tools, and aggregated reporting while avoiding user interaction and non-public account data.

  • The study relies exclusively on publicly available social-media posts containing text and images.It collects no non-public account data or direct user interactions.
  • All annotations were performed by the co-authors, and the work is not classified as human-subjects research.
  • Data storage and processing used secure institutional infrastructure without sharing data externally.
  • The benchmark used open-weight models and open-source tools in-house without deploying agents or manipulating platform behavior.
  • Because the dataset contains sensitive, explicit, and graphic content, release is gated for noncommercial use.

Generative AI Statement

Generative AI assistants supported grammar, manuscript structuring, helper-function implementation, and visualization work, while the authors retained responsibility for the core implementation and draft.

  • Generative AI assistants were used for grammar correction and manuscript structuring.
  • The assistants also helped implement specific helper functions and produce certain reported visualizations.
  • The authors generated the core implementation and manuscript draft, then reviewed and revised AI-provided text and code changes.

A Data Gathering and Curation

The benchmark and prompt experiments combine large-scale Bluesky collection, multimodal retrieval and annotation with policy- and example-driven VLM evaluation under context and memory constraints.

  • Data Gathering: 1.14B post records were collected from Bluesky’s firehose, including 11.9M posts labeled by the Bluesky Moderation Service.
  • Data Curation: The benchmark focuses on nine post-level harm categories and excludes spam because it reflects account-level behavior.
  • Data Curation: 1,000 labeled posts were randomly sampled with category balance, while safe posts came from verified organizational accounts.
  • Data Curation: Posts were represented with normalized 2048-dimensional multimodal embeddings and indexed for similarity retrieval.
  • Data Curation: For each moderated post, the nearest unlabeled firehose neighbor was selected as a semantically similar candidate for potential false-negative analysis.
  • Annotation: A pilot and independent annotation process established shared edge-case rules, with disagreements jointly reviewed or adjudicated.
  • Baseline: Bluesky’s deployed system combines Hive’s vision classifier with automod’s fixed rule thresholds, making missed violations possible when scores fall below thresholds or categories are uncovered.
  • Instruction-Driven Guidance: Instruction prompts combine Bluesky labels, documented scope details, and community-guideline rationale.

F Instruction-Driven Moderation

Instruction-driven moderation improves when policy specifications become more detailed, generally bringing VLM predictions and flagging rates closer to human judgments. These gains coexist with stable instance-level behavior, structured label shifts, and substantial agreement among models, while richer instructions add measurable image-processing latency.

  • F.1 Moderation Effectiveness: Detailed policy instructions generally improve VLM F1 scores across MODERATIONBENCH subsets, with the largest gains appearing when detailed rules are provided.The improvement is most pronounced on Random Posts; Near-moderated Posts show smaller gains, while Moderated Posts already have high F1 scores.
  • F.1 Moderation Effectiveness: Open models remain competitive with frontier models, with many outperforming gpt5.6 while gemini3.5 is marginally better than the strongest open models.Frontier models also carry usage-policy constraints because they are accessed through APIs.
  • F.1 Moderation Effectiveness: Detailed, rule-grounded policy specifications can surpass Bluesky’s automated moderation system when evaluated against human judgments.The results identify policy detail as an effective lever for improving VLM prediction quality.
  • F.2 Flagging Rates: Richer instructions make VLMs more conservative in flagging and bring their outputs closer to human annotator levels across data subsets.Rationales reduce flagging for most models on Random Posts, while detailed rules reduce flagging across all models.
  • F.2 Flagging Rates: On Moderated Posts, models flag most platform-moderated content, although human annotators mark some of those posts safe; on Near-moderated Posts, models identify some harmful content that Bluesky does not flag.On Safe Posts, detailed rules sharply reduce the modest over-flagging introduced by rationales for some models.
  • F.3 Practical Efficiency: Image count and reasoning configuration materially affect latency, with some thinking models becoming much slower as posts contain more images while non-thinking models remain faster.qwen3vl-th approaches four seconds for posts with four images, whereas qwen3vl remains below 0.5 s and llama4 below one second.
  • F.4.1 Intra-Model Flagging Consistency: 95% of gemma4 instances and 96% of qwen3.5 instances remain consistently flagged or unflagged across instruction levels.The limited shifts account for aggregate flagging-rate differences without indicating large-scale behavioral instability.
  • F.4.2–F.5 Decision Consistency: Instruction-level label changes are concentrated in semantically related groups, while detailed rules generally increase inter-model agreement, especially for gemma4 and qwen3.5.Agreement is lower on Moderated and Near-moderated Posts, but gemma4 and qwen3.5 maintain approximately 0.8 or higher agreement there.

G.1 Moderation Effectiveness

Example-driven moderation performs best with prototypical examples, whereas contextual examples increase false positives, refusals, latency, and cost. Across datasets, examples can improve detection of moderated content but may also cause substantial over-flagging on safe content.

  • Moderation Effectiveness: Prototypical examples improve F1 across most models on Moderated Posts, while contextual examples substantially worsen Random Posts performance.Across ten models, Random Posts false-positive rate rises from 13.6% with Prototypical examples to 28.7% with Contextual examples, compared with 17.2% for Random examples.
  • Moderation Effectiveness: Gemini3.5 has an 11.4% refusal rate on Moderated Posts, concentrated in categories such as sexual, porn, nudity, self-harm, and figurative content, whereas gpt5.6 refuses none.The refusals occur through a prompt-level guardrail before generation.
  • Moderation Effectiveness: Contextual examples can sharply increase flagging on Random Posts, far above the human annotators’ 2.4% rate, with much of the increase attributed to false positives.For example, gemma3 rises from 28.1% to 55.1% and llama4 from 23.1% to 49.2% under contextual examples.
  • Moderation Effectiveness: On Moderated Posts, nearly all models achieve flagging rates comparable to or exceeding human annotators, with performance above 90%.The passage interprets this as examples reinforcing detection of harmful content.
  • Moderation Effectiveness: Example-driven prompting can over-flag Safe Posts, reaching 22.3% for qwen3vl-th with random examples and 30.9% for llama4 with contextual examples.These are the highest reported Safe Posts flagging rates for the respective settings.
  • Practical Efficiency: Contextual example selection is substantially more expensive than instruction-only moderation, raising gemma4 latency from 0.95 s to 14.74 s, approximately 16×.Contextual prefixes change for every query, preventing prefix-cache reuse; fixed random and prototypical sets can be cached.
  • Practical Efficiency: Frontier-model usage costs are higher for example-driven than instruction-driven settings, including $493.87 for gemini3.5 Contextual and $2,553.04 for gpt5.6 Contextual.The reported costs differ across example-selection strategies and models.
  • Decision Agreement: Under prototypical examples, models show greater pairwise agreement, with most moderation decisions matching between compared models.The comparison is quantified using Gwet’s AC1.

H.1 Consistency Across Paradigms

Instruction-driven and prototypical example-driven moderation agree on most Moderated Posts but diverge systematically on boundary cases. The disagreements reveal both example-driven over-classification of safe content and a smaller set of harmful cases that examples recover.

  • Consistency Across Paradigms: Instruction-driven and prototypical example-driven prompting assign the same label to 82.4% of Moderated Posts, with disagreements concentrated on boundary cases.Example-driven prompting reclassifies several porn predictions as figurative or sexual and predicts rude more frequently.
  • Consistency Across Paradigms: Human annotations support the instruction-driven prediction in 22 of 27 cases where example-driven prompting changes Safe Posts to unsafe labels.Example-driven labels in these cases are often rude, intolerance, or graphic-media despite mild profanity, debate commentary, or pop-culture references judged safe.
  • Consistency Across Paradigms: Contextual example selection is the most expensive example-driven setting because it changes the in-context prefix for every query and prevents prefix caching.Random and prototypical settings use fixed example sets whose prefix cost can be amortized.
  • Consistency Across Paradigms: Example-driven prompting correctly identifies harmful content missed by instruction-driven prompting in five remaining cases, including rude, graphic-media, sexual, and figurative content.The examples include a politically charged slur-laden post and a poem invoking violence-related language.
  • Consistency Across Paradigms: Pairwise model agreement is higher with prototypical examples, while prediction differences between models remain concentrated in their label assignments.Gwet’s AC1 is used to quantify consistency between models.
  • Model-Family Comparisons: Llama-guard frequently misses rude and graphic-media content in Moderated and Near-moderated Posts, including targeted insults, profanities, and some nudity instances.The comparison is between llama4 and its safety-specialized counterpart under the Bluesky moderation policy.
  • Model-Family Comparisons: Representative disagreement cases compare gemma4 and qwen3.5 using decision and policy-quote agreement, with only text-based instances shown in the paper.The table indicates whether annotators agree on decisions, quotes, both, or neither.
Loading 2609.10410v1…