Source-linked AI summary
Knowing When to Escalate: Disagreement Patterns for Uncertainty-Aware Content Moderation
Michał Wawer, Jarosław A. Chudziak
TL;DR
Content moderation can involve contested judgments, while standard uncertainty decomposition treats inter-agent disagreement as potentially resolvable epistemic uncertainty. The paper analyzes agents’ justification similarity and conclusion agreement, finding that this structure identifies human annotator disagreement more effectively than magnitude-only and output-only baselines.
Problem
Standard uncertainty decomposition treats inter-agent disagreement as epistemic uncertainty, although moderation judgments can be contested and human annotators may also disagree.
Source: §1 · p. 1, §3.1 · p. 2Method
The paper applies a four-state taxonomy based on justification similarity and conclusion agreement to reasoning traces from perspective-differentiated agents.
Source: §1 · p. 1, §3.1 · p. 2, §4.1 · p. 3Results
On the test set, the structural representation identifies high annotator-disagreement items more effectively than magnitude-only and output-only uncertainty baselines.
Source: §7 · p. 5Takeaways & Limitations
Human-review routing can condition on disagreement structure, with RTDA usable as a post-processing method for systems that collect reasoning traces.
Source: §1 · p. 1, §7 · p. 5Evaluation uses 1,000 English items from one 2022 hate-speech corpus and does not establish robustness across policies, languages, cultures, or domains.
Source: §6 · p. 5
Abstract
from arXiv · showhide
Agentic content-moderation systems increasingly rely on collective judgments from LLM agents, while aggregation methods commonly treat inter-agent disagreement as a reliability problem to be removed through consensus or voting. This framing is inappropriate for value-laden tasks, where disagreement may reflect irreducible value pluralism - a distinction standard uncertainty decompositions cannot recover, since they operate on output distributions and discard the reasoning behind each judgment. We propose Reasoning Trace Divergence Analysis (RTDA), which embeds model-generated justifications from perspective-differentiated agents and classifies each case by justification similarity and conclusion agreement. The resulting four-state taxonomy isolates convergent disagreement, where agents reasoning similarly but concluding differently, as a candidate signature of value-laden conflict. On 1,000 items from the Measuring Hate Speech corpus, this structural signal predicts human annotator disagreement better than magnitude-only and output-only baselines. We contribute a design principle: condition human escalation on the structure of disagreement, not its magnitude alone, with RTDA serving as a post-processing layer for systems that already generate reasoning traces.
1 Introduction
Content moderation spans unambiguous and genuinely contested cases, where disagreement may reflect legitimate value differences or other sources. This paper tests whether the structure of agents’ justifications can help identify cases for human review.
1 Introduction
- Moderation disagreement can reflect legitimate value differences, but may also arise from missing context, unclear policies, item quality, or annotator noise [1].The decisions span content from unambiguous to genuinely contested cases.Source: §1 · p. 1
- Multi-agent moderation commonly uses consensus or weighted voting, implicitly treating disagreement as failure to reach ground truth.The paper questions whether this framing is too restrictive for value-laden classification.Source: §1 · p. 1
- The four-state taxonomy uses justification similarity and conclusion agreement to identify candidate review cases, outperforming magnitude-only and output-only baselines on the evaluated sample.It focuses on agents that express similar considerations yet reach opposing conclusions.Source: §1 · p. 1
- The proposed routing framework can be applied as a post-processing layer when a moderation system already collects structured reasoning traces.The study tests whether semantically similar justifications paired with different conclusions identify content on which human annotators also tend to disagree.Source: §1 · pp. 1–2
2 Related Work
Related work questions whether disagreement should be suppressed, noting that it may reflect annotation pluralism or limitations in multi-agent debate. The paper extends this discussion to moderation by evaluating agent disagreement structure as a predictor of when pluralistic human review may be useful.
2 Related Work
- Debate can improve factuality and reasoning [4], yet shared misconceptions, sycophantic agreement [35], and premature consensus show that agreement is not always reliable.These findings motivate analyzing disagreement structure rather than suppressing disagreement.Source: §2 · p. 2
- Data perspectivism treats annotation disagreement as potentially meaningful, with its interpretation varying between descriptive and prescriptive annotation [20].Prior work finds preserving disagreement generally matches or improves on gold-standard training [14], while annotator composition can alter outcomes [8].Source: §2 · p. 2
- The paper evaluates agent disagreement structure as a predictor of when pluralistic human review may be useful, without equating agent disagreement with human value pluralism.Moderation involves normative tensions, and safety judgments can differ between experts and diverse crowd judgments [1].Source: §2 · p. 2
3 Theoretical Framework
The framework treats multi-agent moderation as judgment under uncertainty and tests whether justification structure can help route cases that output-only uncertainty measures cannot distinguish. It defines four patterns from the combination of trace similarity and conclusion agreement, while interpreting agent diversity as prompt-level rather than model-level.
3.1 Uncertainty in Multi-Agent Judgment
- Standard ensemble decomposition treats inter-agent disagreement as epistemic uncertainty, but output distributions cannot distinguish value priorities from knowledge gaps, missing context, prompt sensitivity, or stochastic variation.The framework tests justification structure as a routing signal rather than a way to recover the latent cause of disagreement.Source: §3.1 · p. 2
3.2 Four-state (dis)agreement taxonomy
- Convergent agreement combines similar reasoning and the same judgment, while divergent agreement combines dissimilar reasoning with the same judgment.Convergent agreement is a candidate for automated decisions; divergent agreement is consistent with conclusion stability across prompted perspectives, but does not establish robustness to other models or policies.Source: §3.2 · p. 2
- Convergent disagreement pairs high justification similarity with different conclusions: agents share a compatible understanding of the content but reach different moderation judgments.The paper treats this as a candidate signature of value pluralism, not proof of its cause.Source: §3.2 · pp. 2–3
- Divergent disagreement combines dissimilar interpretations with different judgments and may reflect epistemic uncertainty, content ambiguity, or inconsistent-interpretation noise.Source: §3.2 · p. 3
3.3 Scope of Agent Diversity
- Agents share one model backend and differ through perspective prompts, a controlled design that may retain shared blind spots and conformity tendencies [33].Source: §3.3 · p. 3
- The framework treats disagreement operationally as sensitivity to prompted perspectives, not as proof of human-like value pluralism; a cross-model ablation tests whether the routing pattern recurs across separate models.Source: §3.3 · p. 3
4 Approach and Methods
The methods pipeline uses perspective-differentiated agents, embeds their reasoning traces, and classifies items by trace similarity and conclusion agreement. It fixes classification thresholds before test evaluation and adds embedding and comparison stages once traces are available.
4 Approach and Methods
- The pipeline combines perspective-differentiated deliberation, reasoning-trace embedding, and four-state classification based on trace similarity and conclusion agreement.Source: §4 · p. 3
4.1 Perspective-Differentiated Deliberation
- Five agents share one base model but receive prompts for harm-focused, context-sensitive, community-norms, free-expression, and legal-framework perspectives.They independently produce moderation decisions and structured justifications covering interpretation, considerations, value trade-offs, and a final judgment with confidence.Source: §4.1 · p. 3
4.2 Reasoning Trace Embedding
- Traces are embedded with all-mpnet-base-v2 and compared using pairwise cosine similarity; high mean similarity with disagreement indicates overlapping verbalized considerations, not direct access to latent reasoning.Source: §4.2 · p. 3
4.3 Taxonomy Classification
- Agreement threshold τ_a was fixed at 0.80, and similarity threshold τ_s was set at 0.72 using the 200-item calibration set; both were fixed before test evaluation.Source: §4.3 · p. 3
- With N=5, the pipeline uses five generations, five embeddings, and ten comparisons per item; the 1,000-item experiment uses 5,000 generations, while B4 and B5 use ten sampled generations per item.Once justifications are available, RTDA adds only embedding and comparison stages.Source: §4.3 · p. 3
5 Experiments
The experiments test whether disagreement structure predicts human annotator disagreement and supports routing, using a held-out sample with preserved individual labels. RTDA outperforms magnitude-only and single-model uncertainty baselines on every reported metric, with cross-model and prompt-composition ablations and a qualitative case examining the scope of that signal.
5.1 Setup and Evaluation Criteria
- The evaluation uses 1,000 held-out Measuring Hate Speech items with individual annotator labels; high disagreement is the top quartile of normalized rating standard deviation, an operational routing proxy.The study compares predictive correlations and routing AUROC/F1 against six baselines, including a supervised classifier trained on 200 items; thresholds are selected on calibration data.Source: §5.1 · pp. 3–4, §5 · p. 3
5.2 Structure vs. Magnitude
- Φ outperforms all magnitude-only and single-model uncertainty baselines on every metric and is competitive with supervised B6; its Pearson association with human disagreement is r=0.41 (95% CI [0.36, 0.46]).Within convergent disagreement, the association is r=0.52 (95% CI [0.38, 0.64]; ρ=0.55); these findings indicate association and routing utility, not the cause of human disagreement.Source: §5.2 · p. 4, Table 1 · p. 4, Table 2 · p. 4
5.3 Ablations
- Across τs∈[0.65, 0.80], the structural method remains at least five F1 points above divergence-only at fixed τa=0.80.This test-set sweep was a post-hoc sensitivity analysis and did not alter the calibration-selected headline operating point.Source: §5.2 · p. 4, Figure 2 · p. 4
- Across three base models, structural ordering over magnitude-only methods is preserved, and routing F1 changes by at most 3 points.The within-state ordering also repeats, supporting cross-backend robustness of the observed association, but not heterogeneity among agents within a panel.Source: §5.3 · pp. 4–5, Table 3 · p. 5
- With five identical prompts and only decoding seeds varied, ΔF1 falls to −0.01, supporting a role for prompt-level perspective differentiation rather than multiple outputs alone.The composition ablation varies both agent count and prompt profiles.Source: §5.3 · p. 5, Table 4 · p. 5
5.4 Qualitative Examination
- In one convergent-disagreement case, agents cited overlapping features but weighted them differently, while normalized human disagreement was 0.82.The case also admits competing interpretations involving protection from group-directed harm, political expression, underspecified reference, and missing context; RTDA cannot determine which explanation is correct.Source: §5.4 · p. 5
6 Limitations
The evaluation supports predictive validity only: justification similarity may reflect post-hoc, prompt-sensitive rationalizations rather than latent decisions, and human disagreement does not reveal its cause. The findings are limited to one English hate-speech corpus and an annotator-disagreement proxy, not tested moderator escalation.
6 Limitations
- Generated justifications may be post-hoc, prompt-sensitive rationalizations, and semantic similarity captures surface content rather than latent decision processes.RTDA therefore cannot determine why humans disagree, since disagreement may reflect competing values, unclear items, missing context, inconsistent labels, or annotator noise.Source: §6 · p. 5
- The evaluation uses 1,000 stratified English items from one 2022 hate-speech corpus, so robustness across policies, languages, cultures, and domains is unestablished.Because the corpus predates the evaluated models, pretraining exposure cannot be excluded; this is not a clean test of uncontaminated zero-shot generalization.Source: §6 · p. 5
- Routing is evaluated against an annotator-disagreement proxy, not a study with professional moderators; run-to-run uncertainty and operational cost also remain unmeasured.The authors call for cross-corpus replication, repeated-run analysis, and an actual escalation study before making deployment claims.Source: §6 · p. 5
7 Conclusion and Future Work
RTDA operationalizes a four-state taxonomy of justification similarity and conclusion agreement, and its structural representation identifies high annotator-disagreement items more effectively than magnitude-only and output-only uncertainty baselines. The authors recommend conditioning escalation on disagreement structure and identify validation and policy adaptation as future work.
7 Conclusion and Future Work
- On the test set, RTDA identifies high annotator-disagreement items more effectively than magnitude-only and output-only uncertainty baselines.The composition ablation indicates that the advantage depends on prompt-level perspective differentiation, while the same qualitative ordering across separate LLMs suggests it is not specific to one backend.Source: §7 · p. 5
- RTDA represents disagreement with a four-state taxonomy over justification similarity and conclusion agreement, as a post-processing method for multi-agent moderation.The practical design principle is to condition human escalation on disagreement structure, not magnitude alone.Source: §7 · p. 5
- Future work should test external validity, compare RTDA-assisted escalation with confidence- and vote-based routing, and measure decision quality, consistency, workload, and review time.Repeated runs, genuinely different model backends, alternative rationale and similarity methods, and learned platform-specific thresholds or state-to-action policies are also proposed.Source: §7 · p. 5
GenAI Usage Disclosure
The authors disclose using a generative AI assistant for phrasing, clarity, and identifying weaknesses in draft text, while attributing the research contributions, methods, results, and interpretations to themselves.
GenAI Usage Disclosure
- A generative AI assistant suggested phrasing and improved clarity in draft text.The disclosure also says it helped identify weaknesses.Source: GenAI Usage Disclosure · p. 6
- The authors state that the research contributions, methods, results, and interpretations are their own.Source: GenAI Usage Disclosure · p. 6
- The disclosed AI assistance concerned draft text rather than research contributions, methods, results, or interpretations.Source: GenAI Usage Disclosure · p. 6