Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderation

Ruotong Wang; Zihao Zhu; Siwei Lyu; Xin Tao; Baoyuan Wu

Source-linked AI summary

Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderation

Ruotong Wang, Zihao Zhu, Siwei Lyu, Xin Tao, Baoyuan Wu

arXiv:2609.00206v1Aug 31, 2026cs.CVcs.AI

TL;DR

MLLM-based video moderation lacks a systematic way to detect harmful meaning that emerges from individually benign components distributed across time or modalities. The paper formalizes DIH, builds a multi-agent-synthesized dataset and benchmark, and finds consistent detection deficits across more than 30 MLLMs, including on real-world videos; targeted post-training substantially mitigates these deficits.

Problem
Existing safety datasets and collection pipelines provide limited coverage of compositional harm that is absent from individual components and evades local visual, keyword, and single-modality signals.
Method
The paper formalizes DIH along temporal and modal decomposition axes and constructs a multi-agent-synthesized dataset with structured reasoning annotations and a dedicated benchmark.
Results
More than 30 MLLMs show substantial and consistent deficits on DIH-T and DIH-M, with fewer than 45% of harmful cases detected by the strongest model; targeted post-training improves accuracy by over 60 percentage points without degrading benign-content performance.
Takeaways & Limitations
DIH is a practical video-moderation challenge rather than merely a synthetic artifact, and targeted post-training provides a strong baseline for improving compositional safety detection.
Takeaways & Limitations
The study focuses on temporal and cross-modal DIH, while other composition axes and social or cultural context remain future-work directions.

Abstract

from arXiv · show

Despite their growing use in video moderation, multimodal large language models (MLLMs) exhibit a compositional safety blind spot: videos composed of seemingly benign components can convey harmful meaning when interpreted as a whole. We refer to this phenomenon as Distributed Implicit Harm (DIH), where harm arises from relations among components distributed along a decomposition axis of the video, rather than from any single explicit cue. Among many possible axes, we study two representative cases: temporally distributed harm across visual segments (DIH-T) and cross-modal harm between audio and visual streams (DIH-M). Studying and mitigating DIH at scale requires data that is difficult to collect: such videos lack compositional harm annotations, evade retrieval based on local visual cues, keywords, or single-modality signals, and are consequently absent from existing safety datasets. To bridge this gap, we develop a multi-agent synthesis framework that composes individually benign components into harmful scenarios and generates diverse DIH videos with explicit reasoning annotations, yielding a dataset of over 9,000 videos spanning visual-only and audio-visual settings. Benchmarking over 30 MLLMs spanning frontier proprietary models and leading open-source systems reveals substantial and consistent deficits in detecting both DIH-T and DIH-M. Notably, this failure persists even among the strongest frontier models: they often correctly assess individual components in isolation but fail to recognize the harmful meaning that emerges from their composition. We further evaluate these models on a manually collected set of real-world DIH videos from social media and observe the same failure mode, highlighting DIH as a practical and underexplored challenge for video moderation.

1 INTRODUCTION

The paper identifies Distributed Implicit Harm (DIH), in which individually benign video components produce harmful meaning only through composition, and studies temporal and cross-modal forms. It introduces a dataset and benchmark showing that MLLMs consistently miss this harm, including on real-world videos, while targeted post-training improves detection.

Problem: DIH arises when benign components convey harmful meaning only when interpreted jointly across temporal or audio-visual relations.The paper distinguishes temporal DIH-T from cross-modal DIH-M and contrasts both with explicitly depicted or uttered harm.
Dataset: Existing safety datasets largely omit DIH because it lacks compositional-harm annotations and evades frame-level, keyword, and visual-cue collection methods.The paper addresses this gap with a multi-agent synthesis framework and reasoning annotations covering 8 major and 28 fine-grained safety categories.
Results: Fewer than 45% of harmful DIH cases are detected by even the strongest model, with failures persisting on held-out real-world social-media videos.The benchmark reports deficiencies across both DIH-T and DIH-M, indicating that the phenomenon is not merely synthetic.
Mitigation: Targeted post-training improves detection accuracy by over 60 percentage points on the strongest models without degrading benign-content performance.The result shows substantial mitigation of the measured deficit through training on the DIH Dataset.
Benchmark: The resulting DIH Dataset contains over 9,000 videos, and DIH-Bench evaluates compositional harm across visual-only and audio-visual settings.The benchmark covers temporal and cross-modal DIH and evaluates more than 30 proprietary, open-source, and omni-modal MLLMs.

2 RELATED WORK

Prior multimodal safety work has expanded from explicit image and image-text harms toward video-level moderation, but existing approaches generally treat harmful content as locally observable. This leaves compositional harm across time and modalities insufficiently addressed.

Multimodal Safety Benchmarks: Early multimodal safety benchmarks primarily evaluate explicit harm in static images or manually paired image-text inputs.These settings do not capture the native temporal and audio-visual structure of video.
Multimodal Safety Benchmarks: Recent video safety benchmarks extend evaluation to content unfolding jointly across time and audio-visual streams.Video is treated as a natively multimodal medium rather than a static image-text pair.
MLLM-Based Moderation: MLLM-based moderation systems commonly sample frames and apply MLLM judgments to flag policy-violating clips.Examples include SafeWatch, KuaiMod, and Filter-And-Refine, while image-text systems emit per-category safety labels.
MLLM-Based Moderation: Across existing MLLM moderation work, harm is generally assumed to be directly observable in portrayed actions or events.This assumption differs from DIH, whose harmful meaning emerges from distributed composition.

3 DISTRIBUTED IMPLICIT HARM

Distributed Implicit Harm is defined by a gap between local safety and global interpretation: every component is benign alone, but a recomposition of components supports harm. The paper formalizes this phenomenon along temporal and modal axes while noting broader axes as future scope.

Problem Definition: A decomposition axis is a dimension along which a video can be partitioned into interpretable components, such as time, modality, speaker, or scene structure.The decomposition provides the basis for testing local components and their recompositions.
Problem Definition: DIH requires local safety, meaning every component is individually non-harmful.Formally, the harm indicator satisfies S(V_i) = 0 for every component V_i.
Problem Definition: DIH also requires compositional harm, meaning some recomposition of at least two components is harmful.The condition is expressed as S(Comp_α(V_i1, ..., V_im)) = 1 for distinct indices with m ≥ 2.
Decomposition Axes: The framework is currently focused on temporal and modal axes, while speaker, subtitle, and scene-structure axes remain future extensions.The paper also notes that whether a composition constitutes DIH may depend on social and cultural context.
Decomposition Axes: In temporal DIH, successive benign visual segments acquire harmful meaning through relations such as analogy, contrast, narrative progression, or off-screen suggestion.The temporal axis is instantiated as DIH-T.
Decomposition Axes: In modal DIH, benign visual and audio streams produce harmful meaning through cross-modal alignment, with an intrinsic temporal dimension.This form is called DIH-M and differs from manually paired image-text harms.

4 DIH DATASET

The DIH Dataset is built with a scalable multi-agent framework that synthesizes videos whose components are individually benign but harmful in composition, with reasoning annotations and broad safety coverage. It includes synthetic and real-world data, plus human verification and perception validation.

Multi-Agent Synthesis Framework: The multi-agent framework synthesizes DIH videos with reasoning-level annotations explaining how harm emerges from component composition.The pipeline is designed for scalable, semantically rich, and structurally diverse benchmarking content.
Multi-Agent Synthesis Framework: Specialized agents plan innocuous components, validate local and compositional safety, generate videos, and produce chain-of-thought explanations of the intended harm pattern.The framework supports temporal composition across visual scenes and cross-modal composition across visual and audio streams.
Multi-Agent Synthesis Framework: The pipeline also generates benign compositions that contain category-relevant sensitive elements but remain globally safe, enabling false-alarm analysis.These benign samples share categorical themes with harmful samples while differing in compositional intent.
Dataset Overview: The dataset contains 9,725 samples organized into DIH-T and DIH-M subsets, covering temporal visual harm and audio-visual harm.The dataset is accompanied by detailed chain-of-thought annotations; the cited passage reports 6,742 DIH-T videos and 2,983 DIH-M videos.
Dataset Overview: The taxonomy spans 8 major safety categories and 28 subcategories, including sexual, violence, discrimination, illegal, political, misinformation, psychological manipulation, and risky activities.The categories were identified from social-media safety policies and existing MLLM unsafe-content taxonomies.
Quality Control and Evaluation: Human verification checks that components are benign in isolation, their composition is harmful, and the reasoning annotation accurately reflects the video and implicit-harm mechanism.MLLM-based quality control was avoided because uncertainty in MLLMs’ DIH recognition could make it unreliable.
Quality Control and Evaluation: In a 20-participant study of 144 sampled videos, harmful videos were judged harmful reliably in full but rarely when components were isolated, while benign videos remained low in both conditions.This supports the dataset’s intended distinction between harm visible after composition and locally benign components.

5 EXPERIMENTS

Experiments show that MLLMs struggle to detect distributed implicit harm across synthetic and real-world data, while prompting and post-training offer different levels of improvement. Ablations indicate that the central bottleneck is compositional reasoning rather than basic visual perception.

5.2 BENCHMARKING MLLMS ON DIH: Accuracy on harmful samples remains below 45% across DIH-T and DIH-M for all evaluated models, including the strongest proprietary systems.Larger models generally perform better, but the improvement is modest and insufficient for reliable DIH detection.
5.3 REPLICATION ON REAL-WORLD DATA: Accuracy on harmful samples does not exceed 40% for any evaluated model on either DIH-T or DIH-M in the Real-World Reference Set.The real-world results closely mirror the synthetic benchmark, indicating the failure mode persists on social-media videos.
5.4 ENHANCING DIH DETECTION VIA COT PROMPTING: CoT prompting consistently improves detection across tested models, but gains diminish for stronger baselines and remain substantially below post-training.The prompt decomposes detection into identifying components, analyzing cross-component associations, and assessing safety implications.
5.5 ENHANCING DIH DETECTION VIA POST-TRAINING: SFT substantially improves Harm. accuracy at every Qwen3.5 scale, while GRPO adds larger gains at ≥9B and Ben. accuracy remains preserved.The post-training procedure uses chain-of-thought annotations and a composite reward covering reasoning, verdict, category, and output format.
5.5 ENHANCING DIH DETECTION VIA POST-TRAINING: Unsafe detection accuracy rises sharply after post-training on real-world videos while Safe accuracy is preserved.This result supports generalization of the compositional reasoning learned from DIH training data to in-the-wild distributions.
5.6 ABLATION STUDIES: From N=8 to N=64 frames, the largest accuracy change is only 1.7%, and accuracy stabilizes from 480P at higher resolutions.These ablations place the bottleneck in reasoning over already-perceived components rather than in frame sampling or fine-grained visual perception.

6 CONCLUSION

This work establishes Distributed Implicit Harm as a practical challenge for multimodal video safety and shows that current models struggle with compositional harm. It also outlines extensions beyond temporal and cross-modal settings, including contextual knowledge and real-time moderation.

DIH detection shows a substantial and consistent gap in current MLLMs, including on real-world social media videos.This indicates the failure mode is not merely a synthetic artifact.
The study develops a scalable multi-agent synthesis pipeline, the DIH Dataset, and DIH-Bench with structured reasoning annotations.The resources cover temporally distributed and cross-modal harm.
Targeted post-training on the DIH Dataset substantially improves DIH detection without degrading benign-content performance.This establishes a baseline for compositional video safety.
Future work should extend DIH beyond temporal and cross-modal axes to speaker, viewpoint, and on-screen text.The paper also identifies social and cultural context as relevant to determining whether a composition constitutes DIH.
Integrating DIH-aware reasoning into real-time moderation pipelines remains an important systems direction.The stated target is scalable deployment on streaming and social media platforms.