Source-linked AI summary
AI vs. Human Moderators: A Comparative Evaluation of Multimodal LLMs in Content Moderation for Brand Safety
Adi Levi, Or Levi, Sardhendu Mishra, Jonathan Morra
TL;DR
Multimodal video moderation remains underexplored, despite the need to interpret visual and textual cues and scale review of growing video volumes. The paper benchmarks GPT, Gemini, and Llama using a professionally labeled, multimodal and multilingual dataset, finding promising cost-reduction potential and stronger performance than text-only methods. It also identifies limitations in language performance and evaluation scope.
Problem
Multimodal content moderation remains relatively underexplored, while text-focused moderation does not capture crucial visual cues in video.
Source: §1 · p. 1Method
The authors build a professionally labeled multimodal, multilingual video dataset and evaluate off-the-shelf and open-source MLLMs against human reviewers.
Source: §1 · p. 2Results
The authors report that multimodal models consistently outperform text-only methods and that compact MLLMs are significantly cheaper without sacrificing accuracy.
Source: §6 · p. 8Takeaways & Limitations
The findings indicate promising cost-reduction potential for hybrid human-AI moderation systems.
Source: §6 · p. 8The evaluation covers three risk categories, the first 60 seconds of videos, and zero-shot performance, leaving broader domains, long-form context, and trained models unassessed.
Source: §5.3 · p. 8
Abstract
from arXiv · showhide
As the volume of video content online grows exponentially, the demand for moderation of unsafe videos has surpassed human capabilities, posing both operational and mental health challenges. While recent studies demonstrated the merits of Multimodal Large Language Models (MLLMs) in various video understanding tasks, their application to multimodal content moderation, a domain that requires nuanced understanding of both visual and textual cues, remains relatively underexplored. In this work, we benchmark the capabilities of MLLMs in brand safety classification, a critical subset of content moderation for safe-guarding advertising integrity. To this end, we introduce a novel, multimodal and multilingual dataset, meticulously labeled by professional reviewers in a multitude of risk categories. Through a detailed comparative analysis, we demonstrate the effectiveness of MLLMs such as Gemini, GPT, and Llama in multimodal brand safety, and evaluate their accuracy and cost efficiency compared to professional human reviewers. Furthermore, we present an in-depth discussion shedding light on limitations of MLLMs and failure cases. We are releasing our dataset alongside this paper to facilitate future research on effective and responsible brand safety and content moderation.
1. Introduction
The paper studies whether MLLMs can classify brand-safety risks accurately and compare with professional reviewers as video moderation grows harder to scale. It introduces a multimodal, multilingual benchmark and evaluates MLLMs against human reviewers.
1. Introduction
- The growth of user-generated video increases demand for scalable, accurate, cost-efficient moderation, while human review requires substantial effort and carries well-being risks.The paper notes financial burdens for organizations and documented negative impacts on reviewers exposed to disturbing content.Source: §1 · p. 1
- The study asks whether MLLMs perform accurately in brand safety and how their accuracy and cost-efficiency compare with professional human reviewers.Source: §1 · p. 1
- Multimodal content moderation remains relatively underexplored, despite the need to interpret nuanced visual and textual cues in videos.Earlier LLM moderation work mainly focused on text, missing visual cues such as gestures, graphic scenes, and contextual backgrounds.Source: §1 · p. 1
- The authors curate a publicly released dataset of thousands of professionally labeled videos, spanning risk categories, modalities, and a dozen languages.Reviewer annotations identify the modality of each risk signal, supporting analysis of modality contributions and model robustness across languages.Source: §1 · p. 2
- The benchmark evaluates off-the-shelf and open-source GPT, Gemini, and Llama models, including prompting techniques, against professional reviewers.The comparison covers accuracy and cost-efficiency and discusses MLLM limitations and failure cases.Source: §1 · p. 2
2. Related Work
Prior work examines LLM annotation, moderation, and brand safety, but largely leaves visual video cues outside content-moderation evaluations. This paper addresses that gap with MLLMs and a multimodal benchmark.
2.1. LLMs for annotation and video understanding
- Annotation research highlights labeled-data scarcity; AnnoLLM uses an explain-then-annotate process and achieves performance comparable to human annotators [11].The paper extends this line of work to video moderation, which requires interpreting interactions between visual and textual elements [24].Source: §2.1 · p. 2
2.2. Content moderation and brand safety methods
- Content-moderation scholarship raises concerns that automation can oversimplify context and introduce bias, while brand-safety work argues for domain-specific datasets.Prior work also explores NLP, computer vision, and sentiment analysis for assessing content quality and context.Source: §2.2 · pp. 2–3
2.3. LLMs for content moderation
- Text-focused moderation research studies rule violations, toxicity, complex reasoning, and user-participatory justifications, but leaves video’s visual aspects largely unaddressed [13] [16].The paper’s proposed evaluation combines video-derived frames, thumbnails, associated text, and transcripts.Source: §2.3 · p. 3
- Prior LLM moderation studies largely neglect visual video cues; this study evaluates MLLMs for brand-safety classification using visual and textual video information.The authors describe this as the first systematic evaluation of MLLMs for brand-safety classification.Source: §2.3 · p. 3
3. Dataset
The paper introduces a public benchmark for multimodal brand-safety research, with professionally reviewed videos across risk categories, modalities, platforms, and languages. Its annotations and distributions support analysis of modality signals and dataset coverage.
3. Dataset
- The dataset addresses the shortage of public benchmarks that capture interactions between visual and textual cues in sensitive video content.Existing datasets are often modality-specific or exclude sensitive content, while this benchmark is intended for responsible brand-safety evaluation and includes protective measures and ethical guidelines.Source: §3 · p. 3
3.1. Data collection
- The benchmark contains 1,500 videos, with 500 each in Drugs, Alcohol and Tobacco, Death, Injury and Military Conflict, and Kids’ Content.The categories provide varied textual and visual signals; Kids’ Content is a risk for some advertisers, including alcohol and pharmaceutical brands.Source: §3.1 · p. 3
- Professional reviewers label videos across text, audio, and visuals, reviewing the full video or its first 60 seconds when it exceeds that length.A senior team member verifies a significant sample of completed reviews.Source: §3.1 · p. 3
3.2. Data characteristics
- 89% of videos have risk signals in multiple modalities; Kids has the largest multimodal share at 92%, while DIMC has the highest visual-only share at 8% and text-only share at 6%.The modality distributions are shown by category in Figure 1; some videos have risk signals only in visuals or only in transcripts.Source: §3.2 · p. 4, Figure 1 · p. 4
- The dataset spans four platforms and a dozen languages, with TikTok at 48.5%, YouTube at 33.1%, Meta at 18.4%, and English at 34.5%.Arabic (18.1%), Spanish (16.7%), Japanese (12.4%), Portuguese (7.9%), and German (5.8%) are also prominent; Figure 2 presents platform and language distributions.Source: §3.2 · p. 4, Figure 2 · p. 4
4. Method
The method prepares multimodal video inputs and evaluates MLLMs through zero-shot brand-safety classification. It uses structured prompts and category-specific binary questions to assess out-of-the-box performance.
4. Method
- The evaluation has two stages: multimodal data preparation followed by zero-shot classification using prompting techniques.Source: §4 · p. 4
4.1. Data preparation
- Videos are trimmed to 60 seconds, key frames are selected at scene changes, and transcripts are generated with Whisper; titles, descriptions, thumbnails, and video files provide additional inputs.Scene-change sampling was chosen after one-frame-per-second sampling produced substantial redundancy; the preparation aimed to reduce overhead.Source: §4.1 · p. 5
4.2. Zero-shot classification
- Zero-shot prompts combine explicit instructions, a brand-safety expert persona, constrained YES/NO output, and step-by-step reasoning guidance.Models receive binary questions for each risk category, and boolean logic combines answers into final category predictions; the authors note further evaluation of prompting effectiveness is needed.Source: §4.2 · p. 5
- Gemini receives MP4 video directly, whereas GPT and Llama receive extracted key frames alongside text, thumbnails, and transcripts.The models are evaluated with API batch requests or locally, with available safety settings configured to BLOCK NONE.Source: §4.2 · p. 5
5. Evaluation
The evaluation compares MLLM accuracy and cost-effectiveness with professional reviewers, then examines modality effects, cost trade-offs, and failure cases.
5.1. Accuracy
- Humans led all MLLMs in F1 across categories (overall 0.98), while Gemini-2.0-Flash and Gemini-2.0-Flash-Lite topped MLLMs at 0.91.Source: Table 2 · p. 6, §5.1 · p. 6
- Compact models show no significant performance degradation versus larger counterparts; Gemini-2.0-Flash-Lite matches Flash, while GPT-4o-mini marginally exceeds GPT-4o overall.Source: §5.1 · p. 6
- Gemini models prioritize recall, whereas GPT-4o and its mini variant maintain higher precision, making the preferred trade-off depend on false-positive versus false-negative costs.Source: §5.1 · p. 6
- Multimodal models generally improve on text-only counterparts, with GPT-4o slightly underperforming its text-only variant in DIMC; Gemini gains more than GPT.Source: Table 3 · p. 6, §5.1 · pp. 6–7
5.2. Cost-Efficiency
- Gemini-2.0-Flash and Gemini-2.0-Flash-Lite achieve F1=0.91 at costs of $56 and $28, while the cheapest model, GPT-4o-mini, costs $25 at F1=0.88.Source: Table 5 · p. 7, §5.2 · p. 8
- Visual inputs account for most model cost, creating a cost trade-off against the classification gains associated with multimodal inputs.Source: §5.2 · p. 7, §5.1 · p. 7
5.3. Limitations and failures
- Failures include language and context errors, such as misclassifying Japanese caffeine-addiction content and ambiguous terms; a reviewer also initially misclassified a fictional serial-killer video.Source: §5.3 · p. 8
- The baseline covers three risk categories, the first 60 seconds of videos, and zero-shot performance, leaving broader domains, long-form context, and fine-tuning unevaluated.Source: §5.3 · p. 8
6. Conclusion
The study introduces a public multimodal, multilingual benchmark and evaluates MLLMs for brand safety, finding promise for lower-cost moderation while professional reviewers remain more accurate on complex classifications.
6. Conclusion
- Human reviewers remain more accurate than MLLMs, particularly on complex or nuanced classifications.The authors identify incorrect associations and language-support challenges as areas for future improvement.Source: §6 · p. 8
- The paper introduces a high-quality, publicly available benchmark of multimodal and multilingual videos spanning platforms and risk categories.The authors plan to expand the dataset with additional risk categories.Source: §6 · p. 8
- Across the evaluated models, multimodal methods consistently outperform text-only methods by combining textual, visual, and auditory signals.Source: §6 · p. 8
- Compact MLLMs offer a significantly cheaper alternative to larger models without sacrificing accuracy, supporting their potential in hybrid human-AI moderation.The authors identify fine-tuning and long-format video cost-efficiency as future research directions.Source: §6 · p. 8