PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation

Danting Zhang, Bei Peng, Robert Loftin

arXiv:2609.03814v1cs.CL

TL;DR

Aggregated moderation labels leave unclear whether LLMs can reliably apply individual criteria. The paper introduces DECO and pairwise evaluation to test criterion-level agreement and conditioning, finding that strong benchmark performance does not ensure reliable criterion-conditioned behaviour.

  • Problem

    Because moderation benchmarks often combine multiple criteria into one label, it remains unclear whether LLMs can accurately apply a single criterion when making decisions.

    Source: §1 · p. 1

  • Method

    The paper introduces criterion-independent DECO content factors to derive criterion-specific labels and compares predictions across criteria for the same input.

    Source: §1 · pp. 1–2

  • Results

    Strong performance on standard benchmarks does not imply reliable criterion-conditioned behaviour, with models often making the same decision when different criteria imply different labels.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Moderation systems should be evaluated not only on aggregate performance but also on whether they apply specific criteria.

    Source: §6 · p. 9

    The criterion functions are minimal executable approximations, not complete or normative definitions of real-world moderation criteria.

    Source: §7 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) demonstrate strong performance on standard content moderation benchmarks. However, these benchmarks often aggregate multiple moderation criteria into a single label, making it unclear whether models can disentangle them and reliably apply each criterion when making decisions. To study whether LLMs exhibit criterion-conditioned behaviour, we introduce Diagnostic Evaluation of COntent (DECO), a criterion-independent factorisation of content that enables controlled, criterion-level evaluation. We also introduce pairwise evaluation to compare model outputs across different criteria for the same input. Across four moderation datasets and four LLMs, we find that strong benchmark performance can hide substantial failures at the criterion level. Models struggle most when correct decisions depend not on overall harmfulness, but on the specific aspect of the content that the criterion requires them to assess. Our results highlight a key limitation of current content moderation benchmarks: strong performance on aggregated labels does not provide sufficient evidence that LLMs can reliably evaluate content with respect to individual moderation criteria. These findings call for the development of evaluation methods that explicitly measure criterion-conditioned behaviour.

1 Introduction

Strong agreement on aggregated moderation labels does not establish that LLMs apply individual criteria reliably. The paper introduces DECO and pairwise evaluation to measure criterion-level agreement and whether predictions change appropriately across criteria.

1 Introduction

  • Aggregated benchmark labels combine multiple criteria, so high overall agreement does not establish reliable application of any one criterion.The introduction asks whether predictions match individual-criterion labels and whether they differ when criteria imply different labels for the same content.Source: §1 · p. 1
  • Pairwise evaluation compares predictions across criteria for the same input, testing whether outputs preserve the label relation implied by each criterion.It complements per-criterion agreement and examines same-label and different-label pairs separately.Source: §1 · pp. 1–2
  • DECO derives criterion-specific labels from content factors scored without access to the evaluated criterion, providing a scalable approximation to individual-criterion labels.On a sampled subset, the derived labels substantially align with judgments from multiple human annotators.Source: §1 · p. 2
  • LLMs struggle more with intent- or actionability-based criteria than with surface-level cues, overreacting to mild expression while missing implicit hostile intent or harmful procedures.Criterion-conditioning is especially inconsistent for intention–facilitation pairs when the criteria imply different labels.Source: §1 · p. 2

2 Related Work

Prior moderation benchmarks commonly combine several criteria into one label, while evaluations rarely test whether models respond to the specified criterion itself. This paper addresses that gap by separating criteria and comparing predictions for the same content.

2 Related Work

  • Moderation benchmarks often combine surface-expression, intention-based, and facilitation-based criteria into a single human-annotated label.Variation in annotators’ identity, task framing, and label interpretation can also affect those judgments.Source: §2 · pp. 2–3
  • High agreement with an individual-criterion label may still reflect shortcuts rather than application of the intended criterion.Prior work describes models exploiting superficial heuristics or annotation artifacts, but evaluations rarely test criterion-conditioned outputs directly.Source: §2 · p. 3
  • The paper addresses this evaluation gap by separating moderation criteria and comparing model outputs for the same content under different criteria.This makes criterion-conditioned behaviour directly observable.Source: §2 · p. 3

3 Evaluation Framework

The framework evaluates moderation predictions against individual criterion-implied labels and tests whether predictions preserve relationships across criteria. It uses three separated criteria, content-only DECO factors, and per-criterion and pairwise metrics.

3 Evaluation Framework

  • The framework treats moderation as binary classification conditioned on one criterion and evaluates both agreement with criterion-implied labels and cross-criterion label relations.The two evaluation properties are single-criterion agreement and whether prediction pairs preserve the relation implied by two criteria.Source: §3 · p. 3

3.1 Selection of Moderation Criteria

  • The three criteria target surface expression, harmful author intent, and actionable facilitation in the input.Appearance Criterion excludes weak or incidental toxic language; Intention Criterion excludes quotation or neutral discussion; Demonstration Criterion targets actionable content rather than mere mention or request.Source: §3.1 · p. 3

3.2 Content Representation with DECO

  • DECO represents content independently of moderation criteria because existing benchmarks lack separate labels and direct human relabeling is costly to scale.Its six interpretable factors are not intended as an exhaustive or uniquely correct representation.Source: §3.2 · p. 3
  • The six DECO factors capture distinct content signals, including lexical expression, harm relevance, author orientation, procedural steps, reproducible detail, and neutral stance.The representation is a six-factor tuple, F(x) = (f1(x), . . . , f6(x)).Source: §3.2 · pp. 3–4
  • An LLM scores each factor from 0–10 using content and factor definitions alone, then predefined criterion-specific heuristics map selected factors to safe or unsafe labels.Thresholds set the strength required for relevant signals, while Δic controls how much harmful orientation must exceed neutral or distancing stance.Source: §3.2 · p. 4
  • The heuristic functions are moderate executable abstractions: they exclude weak lexical mentions, neutral reporting or quotation, and mere mention or general requests.The authors do not present these functions as the only valid formulation.Source: §3.2 · p. 4

3.3 Criterion-Conditioned Evaluation Metrics

  • Per-criterion evaluation measures accuracy for each criterion and also examines recall and the direction of false-positive and false-negative errors.False positives mark safe-to-unsafe predictions; false negatives mark unsafe-to-safe predictions.Source: §3.3 · p. 4
  • Cross-criteria pair accuracy is split into same-label and different-label cases to test whether prediction pairs match criterion-implied label pairs in each subset.SPA and DPA assess preservation of pair structure, not causal attribution.Source: §3.3 · p. 4

4 Experiment Settings

The experiments use four complementary moderation benchmarks and four instruction-following LLMs, then derive criterion-level labels from DECO scores and assess their alignment with benchmark projections and independent human judgments. The validation supports the diagnostic labels, but the authors treat them as controlled targets rather than absolute ground truth.

4.1 Datasets

  • The four benchmarks are Civil Comments, X-Sensitive, OpenAI Moderation Dataset, and ToxicChat, with sampled subsets of 20,000 CC and 5,000 TC examples.The datasets cover complementary harmful-content settings; CC and TC subsets preserve their original label distributions.Source: §4.1 · p. 5

4.2 Models

  • The evaluation covers Qwen2.5-7B-Instruct, Llama-3.1-70B-Instruct, GPT-5.22, and Gemini-2.5-Pro, selected to span different model families rather than exhaustively represent all frontier LLMs.All models are queried independently under each criterion using the same prompt templates and evaluation pipeline, with temperature set to 0 for all models.Source: §4.2 · p. 5

4.3 Annotations

  • A single LLM annotator scores six criterion-independent DECO factors, and the evaluated models receive only the input text and criterion definition, not DECO annotations.The main thresholds encode moderate rather than zero-tolerance interpretations of AC, IC, and DC; sensitivity analyses found the main results consistent under threshold variations.Source: §4.3 · p. 5
  • DECO-derived binary labels agree with binary projections of the OpenAI Moderation Dataset at 86% for AC, 75% for IC, and 70% for DC.The authors describe this comparison as a sanity check, since original benchmark labels do not specify which criterion grounds a decision.Source: §4.3 · pp. 5–6
  • Mean human–DECO agreement was 77%, 80%, and 92%, and majority-vote human–DECO agreement was 82%, 84%, and 98% for AC, IC, and DC, respectively.The independent validation used five annotators and a stratified subset of 200 examples.Source: §4.3 · p. 6
  • The authors treat DECO-derived labels as controlled diagnostic targets rather than absolute ground truth because validation remains limited in scale.They note that larger expert annotation studies would strengthen the connection to real-world moderation practices.Source: §4.3 · p. 6

5 Results

Strong performance on aggregated benchmark labels does not consistently carry over to individual criteria, and pairwise evaluation reveals that models often fail to condition predictions on the specified criterion. The clearest difficulty is distinguishing intention-focused from facilitation-focused criteria, with error patterns also varying by dataset and criterion.

5.1 Benchmark Performance

  • Models perform strongly under original benchmark annotations, with Gemini-2.5-Pro and GPT-5.2 the most stable across datasets and above 0.90 recall across benchmarks.Accuracy on ToxicChat is high for all models, while Civil Comments accuracy is consistently the lowest.Source: §5.1 · pp. 6–7

5.2.1 Overall Performance

  • Criterion-level accuracy does not transfer uniformly: it falls on ToxicChat, especially for IC and DC, but often rises on Civil Comments, X-Sensitive, and OpenAI Moderation.Recall is consistently high under AC, but drops under IC on CC and XS and is variable or often low under DC, especially on XS and TC.Source: §5.2.1 · p. 7

5.3.1 Overall Performance

  • Pairwise performance is substantially lower than benchmark and single-criterion results, with IC–DC different-label pair accuracy below 0.1 for some models on ToxicChat.The IC–DC gap between different-label and same-label pair accuracy indicates difficulty distinguishing harmful intent from actionable facilitation.Source: Table 3 · p. 8, §5.3.1 · p. 8

5.3.2 Qualitative IC–DC Pair Analysis

  • In two ToxicChat examples, Llama-3.1-70B-Instruct extends violent intent or a sensitive keyword to DC-unsafe predictions despite no actionable facilitation.These errors can collapse different-label criterion pairs into the same label, producing over-blocking or under-blocking.Source: Table 4 · p. 8, §5.3.2 · p. 9

5.2.2 Error Analysis

  • Error patterns depend on dataset and criterion: AC produces more false positives, while IC and DC yield false negatives on CC and often false positives for several models on TC.Gemini-2.5-Pro remains more conservative on TC under IC and DC, missing a larger proportion of unsafe cases.Source: Figure 2 · p. 7, §5.2.2 · p. 8

5.4 Discussions and Implications

  • The findings support criterion-specific and pairwise evaluation because accuracy and recall alone do not show whether predictions change appropriately with the criterion.The discussion also recommends making criteria explicit, consistently applied, and grounded in criterion-relevant evidence.Source: §5.4 · p. 9

6 Conclusion

The conclusion finds that strong standard-benchmark performance does not ensure reliable criterion-conditioned moderation. Pairwise evaluation reveals that models may make the same decision even when different criteria imply different labels, particularly for harmful orientation versus actionable facilitation.

6 Conclusion

  • The paper presents a comprehensive evaluation of LLM criterion-conditioned behaviour in content moderation.Source: §6 · p. 9
  • Models often produce the same decision even when different criteria imply different labels, especially for harmful orientation versus actionable facilitation.The conclusion argues moderation evaluation should assess whether models apply specific criteria, not only aggregate performance.Source: §6 · p. 9
  • Strong performance on standard benchmarks does not imply reliable criterion-conditioned behaviour.Source: §6 · p. 9

7 Limitations

The limitations concern the approximations used to represent criteria, the compact scope of DECO’s factors, and ambiguity among natural-language criteria. The authors frame the findings within these boundaries and call for further validation.

7 Limitations

  • DECO’s deterministic criterion functions are minimal executable approximations, not complete or normative definitions of real-world moderation criteria.Validation against human policy experts or platform-specific guidelines could strengthen their connection to real-world enforcement practice.Source: §7 · pp. 9–10
  • DECO uses six factors for the criteria studied, but is not an exhaustive ontology of harmful content.Potentially omitted dimensions include target or speaker identity, audience vulnerability, conversational context, local norms, and platform-specific priorities.Source: §7 · p. 10
  • Appearance-, intention-, and facilitation-based criteria can have ambiguous boundaries, so findings concern model behaviour under realistic criterion ambiguity.The authors study natural-language criteria because real-world moderation policies are often underspecified and require interpretation.Source: §7 · p. 10

8 Ethical Considerations

The ethical discussion distinguishes applying a specified moderation criterion from deciding whether that criterion should be applied. The appendices describe criterion-independent DECO factor prompts, single-criterion prompts, and human annotation guidance that clarifies distinctions among criteria.

8 Ethical Considerations

  • A reliable moderation model should apply legitimate customized criteria while rejecting criteria that enable harm, abuse, or unsafe behaviour.The paper does not resolve this tension, but calls for separating criterion application from whether the criterion itself should be applied.Source: §8 · p. 10

A.1 DECO Generation Prompts

  • DECO prompts define six content-level factors independently of downstream criteria to reduce prediction-driven or post hoc factor scoring.The factor prompts aim to capture observable content properties rather than encode safe or unsafe labels.Source: §A.1 · p. 12, Table 5 · p. 13

A.2 Single-Criterion Evaluation Prompts

  • Single-criterion prompts adapt existing taxonomies, while the AC criterion tolerates non-abusive profanity and tests adherence beyond memorized moderation heuristics.The IC criterion covers protected-trait hate, and DC covers procedural guidance facilitating unlawful acts.Source: §A.2 · pp. 13–14

A.3 Human Annotation Guideline

  • Five annotators independently label inputs under AC, IC, and DC without access to DECO scores, derived labels, or evaluated-model predictions.The guidelines use examples to clarify distinctions among criteria.Source: §A.3 · p. 14

B Human Validation Agreement Metrics

The human-validation metrics compare DECO-derived labels with individual annotators or majority votes, and measure agreement between annotator pairs. Their inclusion rules determine which uncertain judgments contribute.

B Human Validation Agreement Metrics

  • Majority-vote human–DECO agreement compares DECO with the majority human label, excluding uncertain majorities and examples without a unique mode.The metric is calculated only for examples with a non-uncertain majority.Source: B Human Validation Agreement Metrics · p. 14
  • Mean pairwise human–human agreement averages agreement over 10 annotator pairs; uncertain–uncertain pairs are excluded, while uncertain–safe or uncertain–unsafe pairs count as disagreements.The pairwise metric averages agreement across all 10 pairs.Source: B Human Validation Agreement Metrics · p. 14
  • Mean human–DECO agreement compares each annotator’s judgment with the DECO-derived label, excluding examples that annotator marks uncertain.The study averages agreement across five annotators.Source: B Human Validation Agreement Metrics · p. 14

C Individual Label Recalls on Original Benchmarks

Recall on original benchmark labels is generally strong, but several categories and datasets show substantial drops. Isolated criteria can require evidence about stance, intent, or endorsement beyond broad benchmark labels.

C Individual Label Recalls on Original Benchmarks

  • Both GPT-5.2 and Gemini-2.5-Pro achieve only around 0.45 recall on the identity_attack label, with less stable performance on ambiguous labels such as conflictual in X-sensitive.Across the datasets, category-level performance is reasonably strong overall, but several recall drops are non-negligible.Source: C Individual Label Recalls on Original… · p. 14
  • On the OpenAI Moderation dataset, Llama-3.1-70B-Instruct recalls 0.37 on violence, Gemini recalls 0.38 on harassment, and GPT-5.2 recalls 0.35 on violence/graphic.The authors report failures to consistently identify positive instances even with relatively standardised labels and clearer surface evidence.Source: C Individual Label Recalls on Original… · p. 14
  • Separated criteria can require identifying an author’s stance or endorsement, unlike broad labels that may include quoted, reported, or condemned content.This stricter evidential requirement helps explain why strong conventional moderation performance may not carry over to isolated criteria.Source: C Individual Label Recalls on Original… · p. 15

D Threshold Sensitivity of Pairwise Evaluation

Pairwise evaluation’s main qualitative patterns remain stable under permissive and stricter DECO thresholds. Stricter thresholds shift SPA and DPA modestly, while IC–DC different-label cases remain difficult.

D Threshold Sensitivity of Pairwise Evaluation

  • Across two threshold settings, the main pairwise evaluation patterns remain stable, suggesting results are not driven by a single threshold configuration.The settings include a permissive configuration and a stricter one with most factor thresholds increased to 5.Source: D Threshold Sensitivity of Pairwise… · p. 15
  • Under stricter thresholds, same-label pair accuracy often decreases slightly, as stronger evidence is required to judge two examples as sharing a criterion-relevant factor pattern.For GPT, AC–IC SPA drops from 0.49 to 0.45 on Civil Comments, from 0.61 to 0.53 on X-sensitive, and from 0.81 to 0.75 on OpenAI Moderation.Source: D Threshold Sensitivity of Pairwise… · p. 15
  • Different-label pair accuracy does not collapse under stricter thresholds and sometimes improves; Gemini-2.5-Pro’s AC–IC DPA on X-sensitive increases from 0.84 to 0.87.The authors suggest stricter thresholds can reduce false same-pair matches by making the factor representation more selective.Source: D Threshold Sensitivity of Pairwise… · p. 16
  • IC–DC remains imbalanced, with often-high SPA but consistently low DPA, especially on Toxic-Chat and X-sensitive, where distinctions can depend on intent, actionability, or violent escalation.Under stricter thresholds, IC–DC DPA on Toxic-Chat is 0.11 for Gemini-2.5-Pro, 0.13 for GPT-5.2, 0.04 for Llama-3.1-70B-Instruct, and 0.14 for Qwen2.5-7B-Instruct.Source: D Threshold Sensitivity of Pairwise… · p. 16
  • The relative difficulty of datasets and criterion transitions remains largely unchanged, supporting the authors’ conclusion that the observed weaknesses are not artifacts of one threshold choice.ToxicChat and IC–DC different-pair cases remain persistent weaknesses.Source: D Threshold Sensitivity of Pairwise… · p. 16

E Qualitative Examples of False Positives and False Negatives

Qualitative examples show errors tied to applying the specific isolated criterion, alongside broader safety limitations. False positives and false negatives arise when models respond to wording or general safety cues instead of the criterion’s boundary.

E Qualitative Examples of False Positives and False Negatives

  • The examples illustrate that model errors can reflect failures to follow isolated criteria as well as general safety capability limitations.The section examines representative false positives and false negatives from the main experiments.Source: E Qualitative Examples of False… · p. 16
  • AC false positives include nonoffensive content with positive or event-related wording that models incorrectly flag as unsafe, potentially excluding benign content from ad placement.The authors note that overly conservative decisions could lead advertisers to miss opportunities.Source: E Qualitative Examples of False… · p. 16
  • IC false negatives include hostile, exclusionary, or discriminatory intent without canonical threat wording, which can be missed despite high recall on benchmark threat labels.The examples are labelled IC-unsafe by DECO-derived functions but predicted safe by one or more models.Source: E Qualitative Examples of False… · p. 16
  • DC false positives occur when models block requests that mention harmful procedures but provide no actionable procedural details, contrary to the criterion’s boundary.The authors characterize these cases as reliance on broad safety heuristics rather than the exact criterion specified in the prompt.Source: E Qualitative Examples of False… · pp. 16–17

F Computational Budget and Infrastructure

  • The evaluation used API-based inference only, without model training or fine-tuning, and its total cloud/API cost was approximately USD 500.Because the experiments relied on external API inference rather than local model training, GPU-hour usage was not directly applicable.Source: F Computational Budget and… · p. 17
Loading 2609.03814v1…