Source-linked AI summary
VARM-Bench: Benchmarking Verifiable Structured Reasoning in Chinese Abusive Speech Moderation
Mingyu Yuan, Shengtao Wen, Lingbing Guo, Zhen Bi, Xiang Chen
TL;DR
Chinese abusive-speech moderation benchmarks lack a unified, deterministic way to verify the basis of a model’s decision. VARM-Bench addresses this with field-anchored rationales and structured evaluation, showing that strong label performance can coexist with substantial record errors, especially in target identification.
Problem
Existing Chinese moderation benchmarks do not unify verification of targets, stances, and harm categories underlying final-label decisions in context-sensitive social-media text.
Method
VARM-Bench reconstructs complete moderation records from field-anchored rationales and evaluates them deterministically across six moderation decisions.
Results
Strong label-level performance coexists with substantial complete-record errors: GPT-5.5 reaches 97.6% Label Macro-F1 but only 55.4% JREM.
Takeaways & Limitations
Target identification is the primary failure source, making complete-record evaluation necessary for diagnosing Chinese abusive-speech moderation systems.
Abstract
from arXiv · showhide
The widespread circulation of abusive online content has increased the need for reliable moderation of Chinese social-media text. Existing Chinese benchmarks support label classification, fine-grained toxicity categorization, and target-aware extraction, but do not provide a unified representation for deterministically verifying the stated basis of a moderation decision. We introduce VARM-Bench, a benchmark for field-anchored chain-of-thought rationales in Chinese abusive-speech moderation. Each instance contains a concise natural-language rationale with explicit anchors for six decisions: target, target type, target explicitness, author stance, harmfulness label, and fine-grained category. Our deterministic protocol evaluates field correctness, target alignment, output validity, complete-record agreement, and hidden record errors conditioned on correct final decisions, without relying on an LLM judge. Under a common structured-output protocol, we evaluate language models across multiple model families using zero-shot prompting, taxonomy guidance, and structured CoT supervision, and analyze lexical-cue sensitivity and field-level errors. Results show that strong label-level performance can conceal substantial errors in complete moderation records. VARM-Bench provides an auditable and reproducible benchmark for evaluating verifiable moderation rationales in Chinese abusive-speech moderation.
Introduction
Existing Chinese abusive-speech benchmarks and final-label evaluation can miss whether targets, stances, and harm categories are correct in context. VARM-Bench addresses this gap with field-anchored rationales, deterministic moderation records, and an auditable evaluation protocol.
- Motivation: Final-label accuracy can conceal wrong referents, author stances, or harm categories, especially when Chinese discourse uses implicit references, homophones, negation, quotation, or sarcasm.A correct label may still rest on an incorrect moderation basis.
- Motivation: Existing Chinese benchmarks cover classification, bias, implicit toxicity, fine-grained categories, and target extraction, but represent only isolated moderation components.Target spans do not indicate whether authors attack, reject, quote, or neutrally mention targets.
- Benchmark design: VARM-Bench embeds six decisions—target, target type, target explicitness, author stance, harmfulness, and fine-grained category—in one explanation with deterministic record reconstruction.The resulting complete moderation record makes each scored decision traceable to an explanation.
- Benchmark design: 8,000 Chinese social-media comments comprise VARM-Bench, including 1,440 challenging non-harmful examples, with manually corrected and rechecked reference records.The challenging examples include quotation, opposition, neutral identity mentions, behavioral criticism, and other misleading surface cues.
- Evaluation protocol: The unified protocol evaluates decision quality, field correctness, record validity, complete-record agreement, reference sensitivity, explanation quality, and hidden record errors across model settings.The benchmark covers open and closed models under prompting and supervision settings and supports diagnostic analyses of field bottlenecks and rationale quality.
Related Work
Prior work progressed from post-level abusive-language classification toward implicit toxicity, target-aware identification, localized evidence, and fine-grained Chinese benchmarks. Rationale and chain-of-thought research improves inspectability and process evaluation, but extracted spans do not jointly specify all moderation fields.
- Abusive-Language Detection and Chinese Benchmarks: Chinese benchmarks broadened abusive-language evaluation from offensive language and sexism to targeted groups, implied attitudes, toxicity types, expression forms, and target spans.COLD and SWSR cover offensive language and online sexism, while CDial-Bias, ToxiCN, and STATE-ToxiCN extend coverage.
- Rationales and Structured Verification: Rationale benchmarks provide human-annotated evidence that makes predictions easier to inspect, but extracted spans do not jointly show the target, author stance, harmfulness label, and fine-grained category.This limitation motivates structured representations that connect evidence to multiple moderation decisions.
- Rationales and Structured Verification: Chain-of-thought research established natural-language reasoning as a prediction interface and later evaluated intermediate reasoning through chain-level metrics, process supervision, and stepwise verification.Self-consistency additionally studies whether sampling multiple reasoning paths improves prediction reliability.
Preliminaries
VARM-Bench addresses the limits of final-label evaluation by combining six field-anchored moderation decisions with input-specific natural-language justification. A deterministic parser extracts the fields from the same rationale for auditable scoring.
- Motivation: Final-label correctness can conceal wrong targets, explicitness, stances, or incompatible categories, while tuples omit justification and free-form rationales are difficult to compare consistently.These limitations motivate a unified field-anchored moderation CoT.
- Representation: Each input is paired with a six-field reference record and a field-anchored CoT that justifies its moderation decisions.The record is represented as zi = (ti, τi, ϵi, si, yi, ci).
- Structured CoT: The fixed anchor sequence covers target, target type, target explicitness, author stance, harmfulness label, and fine-grained category.A deterministic parser extracts anchored values and reconstructs the predicted moderation record.
- Structured CoT: All field predictions are parsed from one generated CoT, while surrounding prose is retained for rationale auditing rather than generating a separate field tuple.This design supports field-level and joint agreement evaluation with the reference record.
- Rationale requirements: Rationale prose must connect input-specific evidence to decisions involving quotation, negation, opposition, implicit reference, sarcasm, and behavioral criticism.Merely listing or paraphrasing anchor values does not satisfy the rationale requirement.
Benchmark Construction
VARM-Bench comprises 8,000 rigorously split and deduplicated records built from public Chinese social-media text. LLM pre-annotation, independent review, validation, and reannotation established mutually consistent records and high annotation consistency.
- Dataset composition: VARM-Bench contains 8,000 records split into 5,600 training, 800 development, and 1,600 test instances, with a 55/45 harmful–non-harmful ratio and 1,440 difficult non-harmful cases.Normalized duplicates and pairs with RapidFuzz similarity of at least 95% were confined to one split.
- Data collection: Public posts and comments from Bilibili, Zhihu, Baidu Tieba, and Hupu were retrieved with topic queries and lexical cues, while annotators made final decisions from referents, stance, and full context.Difficult non-harmful examples included abusive-language quotations, objections to abuse, neutral identity references, personal accounts, and criticism of behavior.
- Annotation: LLM-assisted pre-annotation produced six-field records and 80–180-character anchored rationales, which three master’s students and one doctoral student independently reviewed in two passes.Accepted rationales contained each anchor once and in order.
- Validation: A record was accepted only when its fields and rationale were source-supported and mutually consistent, and a validator checked completeness, valid values, anchor order, uniqueness, and cross-field constraints.All 8,000 released records passed validation, with no cross-split duplicates remaining at the 95% RapidFuzz threshold.
- Annotation consistency: Three trained reviewers independently reannotated a stratified sample of 200 records without access to frozen references or one another’s decisions, yielding Krippendorff’s α of 0.943 for Label.The sample was category- and phenomenon-stratified.
Experiments
Experiments evaluate multiple model families and prompting regimes on a shared 1,600-item test set, showing that complete-record quality, lexical-cue robustness, and field-level consistency require evaluation beyond final labels. CoT-SFT improves record completeness and makes moderation decisions more inspectable, while target localization remains the main bottleneck.
- Experimental setup: Models are evaluated on the same 1,600-item test set across zero-shot, taxonomy-guided zero-shot, and CoT-SFT settings, with CoT-SFT supervising all six anchors.CoT-SFT uses 5,600 training and 800 development instances with prompt-masked LoRA and development-loss early stopping.
- Complete-record evaluation: Correct category or label decisions can conceal errors in target, explicitness, or author stance, so label-level scores provide only a partial account of record quality.JREM requires target overlap and exact agreement on the other five fields; HER-C/HER-L measure residual errors among parseable outputs with correct decisions.
- Complete-record evaluation: CoT-SFT raises JREM to 56.0–59.5% and lowers HER-C to 34.4–37.2%, while parse success is nearly 100%.These results indicate that remaining errors primarily concern field content rather than output parsing.
- Lexical-cue sensitivity: CoT-SFT’s advantage is concentrated in quotations, rebuttals, neutral mentions, and other non-harmful uses of salient lexical cues.The challenge subsets cover demographic identity, social status or role, body, health or disability, and general abuse.
- Field-level bottlenecks: Target mismatch is highest for every system, followed by target explicitness, whereas the harmfulness label is comparatively stable.Because target localization determines the referent for subsequent fields, boundary or normalization errors can propagate into semantic failures.
- Qualitative analysis: Case analyses show that label/category output omits affected groups, while quoted abuse without author stance can invert harmfulness; VARM-Bench links these dependencies jointly.A quoted group can be retained as the target while opposition, non-harmfulness, and no harm category are recorded.
Conclusion and Future Outlook
VARM-Bench evaluates field-anchored rationales by reconstructing complete Chinese abusive-speech moderation records and applying deterministic metrics across multiple decision fields. Results show that strong label-level performance can coexist with substantial record errors, especially in target identification, while future work will broaden domains and moderation policies.
- Benchmark contribution: VARM-Bench reconstructs complete moderation records from generated CoT rationales and evaluates target, target properties, author stance, harmfulness labels, and categories with deterministic metrics.The benchmark supports systematic analysis of complete moderation decisions and model failure patterns.
- Main finding: Strong label-level performance can coexist with substantial errors in underlying moderation records, with target identification emerging as the primary failure source.
- Future outlook: Future work will extend VARM-Bench to additional domains and diverse moderation policies.