PaperlayerView exact revision on arXiv

Source-linked AI summary

MABPD: Multi-Agent Bias Probing & Detection via Structured Argument Debate

Garvit Joshi, Stavya Dhyani, Jasmine, Arun Chauhan

arXiv:2609.04841v1cs.CLcs.AI

TL;DR

Media bias detection requires sensitivity to subtle presentation cues, yet supervised systems depend on costly annotations and may generalize poorly across taxonomies. MABPD uses three specialized LLM agents and evidence-grounded Structured Argument Debate, achieving performance near supervised SOTA on BABE and transferring zero-shot to HyperPartisan. The main scope boundary is that HyperPartisan evaluation used truncated articles and showed a precision–recall imbalance requiring future adaptation.

  • Problem

    Media bias involves subtle framing and linguistic cues, while supervised detectors require costly annotations and generalize poorly across bias taxonomies.

  • Method

    MABPD uses three specialized agents whose disagreements are resolved by Structured Argument Debate with asymmetric evidence-based weighting and verification.

  • Results

    83.4% macro F1 on BABE was within 0.7 pp of supervised SOTA, while HyperPartisan evaluation achieved 75.0% zero-shot accuracy within 7.2 pp of supervised SOTA.

  • Takeaways & Limitations

    Structured deliberation, rather than mere agent parallelism, is identified as the primary performance driver and supports transfer without task-specific training.

  • Takeaways & Limitations

    HyperPartisan articles were truncated to 600 words, and the run showed 88.8% recall versus 61.4% precision, motivating future threshold adaptation.

Abstract

from arXiv · show

Media bias in news articles operates through subtle linguistic cues---loaded language, selective framing, and strategic omission---that resist single-model detection and have traditionally required large annotated corpora for supervised training. We ask whether structured multi-agent deliberation can serve as a principled, training-free alternative to supervised classification for this task. We introduce MABPD (Multi-Agent Bias Probing & Detection), a pipeline in which three specialized LLM agents analyze an article from complementary perspectives and resolve disagreements through a Structured Argument Debate (SAD) protocol. SAD implements a domain-motivated asymmetric burden of proof---biased claims without grounded textual evidence carry zero weight---combined with role-weighted voting and post-consensus verification, replacing task-specific supervised decision boundaries with explicit deliberative structure. Ablation confirms that this structured deliberation, not mere agent parallelism, drives performance: removing the debate module reduces F1 by up to 10.6 points. On the BABE benchmark (4,121 expert-annotated sentences), MABPD achieves 83.4% macro F1 on the held-out test split---within 0.7 percentage points (pp) of the supervised SOTA (MAGPIE, 84.1% macro F1; Horych et al., 2024)---without any task-specific training or threshold tuning on annotated data. Cross-dataset evaluation on the SemEval 2019 HyperPartisan corpus (644 articles) yields 75.0% zero-shot accuracy, within 7.2 pp of the supervised SOTA accuracy (82.2%; Kiesel et al. 2019), confirming transfer across annotation regimes. We release the full pipeline and evaluation code.

1 Introduction

Media bias is difficult to detect computationally because it arises from presentation choices that can evade factual checks and vary across annotation regimes. MABPD addresses this gap with structured, training-free multi-agent deliberation centered on evidence-grounded debate.

  • Media bias can arise through loaded language, selective framing, ideological labeling, and omission even when sentences remain factually accurate.
  • Supervised bias detectors require costly expert annotation, while models trained on one taxonomy generalize poorly to others.
  • MABPD replaces task-specific supervised decision boundaries with Structured Argument Debate and an explicit deliberative structure.
  • SAD assigns zero weight to biased claims lacking grounded textual evidence and combines role-weighted voting with evidence validation.
  • The architecture isolates Bias, Evidence, and Framing agents, and a heterogeneous-LLM pilot changed F1 by only ∆=−0.1 pp.
  • 83.4% macro F1 on BABE was within 0.7 pp of supervised SOTA, while cross-dataset evaluation achieved 75.0% zero-shot accuracy on HyperPartisan.

2 Related Work

Prior work spans supervised bias detection and multi-agent debate, but media bias remains underexplored in debate-based systems. MABPD positions its contribution against this gap while evaluating architecture rather than raw model differences.

  • BABE is an expert-annotated benchmark across 23 topics, while MAGPIE reaches 84.1% macro F1 through pre-fine-tuning on 59 auxiliary bias tasks.
  • Fine-tuned transformers achieve strong in-domain performance but depend on labeled data and generalize poorly across bias subtypes.
  • Multi-agent debate has addressed factuality, calibration, topology, ablation, and verification, but media bias detection remains largely unexplored in the paradigm.
  • The expanded BABE release contains 4,121 sentences, including additional annotation rounds beyond the original 3,700-sentence dataset.
  • The study uses same-model controls that vary architecture to attribute improvement to evidence grounding, debate, and asymmetric verification rather than raw model capability.

3 Methodology

MABPD combines independent specialist analyses with deterministic scoring, conditional consensus paths, Structured Argument Debate, and asymmetric verification. Its scoring system encodes an evidence-based burden of proof for bias claims.

  • Multi-Agent Analysis: Three specialized agents analyze cleaned text independently: BiasAgent detects bias, EvidenceAgent extracts textual support, and FramingAgent identifies framing patterns.
  • Multi-Agent Analysis: A shared EvidenceStore aggregates bias spans, evidence phrases, and framing spans for subsequent scoring and debate.
  • Scoring: The deterministic scoring agent combines bias, evidence, and framing signals, with weights of 0.50, 0.25, and 0.25 respectively.
  • Consensus: The consensus engine routes articles to neutral veto, strong/moderate bias fast-path, or full debate according to signal strength.
  • Structured Argument Debate: Each SAD argument contains a verdict, up to five evidence spans, confidence, and an anticipated counter-argument.
  • Structured Argument Debate: Biased claims without valid evidence receive zero effective weight, preventing an ungrounded claim from unilaterally forcing biased consensus.
  • Consensus: SAD validates cited evidence against the article and adjusts confidence using evidence quality and inter-agent agreement before resolving conflicts.
  • Verification: The VerifierAgent can override biased decisions to neutral, but never neutral decisions to biased, enforcing conservative post-consensus checking.

4 Experimental Setup

The evaluation uses zero-shot MABPD on the expanded BABE dataset and compares it with supervised and architectural baselines under a held-out protocol. Thresholds were fixed before evaluation rather than tuned on annotated data.

  • Dataset: BABE contains 4,121 sentences across 23 topics, with a 55.8% biased and 44.2% neutral distribution.
  • Dataset: Both BABE splits maintain an approximately 56% bias rate, and MABPD evaluates all sentences without BABE training or threshold tuning.
  • System Configuration: The primary evaluation uses LLaMA 3.3 70B Instruct across agents, with SC = 1 selected for cost efficiency.
  • Baselines: Comparisons include fine-tuned BERT and RoBERTa, MAGPIE, a single-agent baseline, and the full pipeline without debate.
  • Metrics: Metrics include accuracy, precision, recall, binary F1, and macro F1, with significance assessed by paired bootstrap using 10,000 resamples.
  • Held-Out Evaluation: The held-out test protocol compares MABPD and supervised systems on identical data, with MABPD using no BABE training data.
  • Zero-Shot Protocol: All decision thresholds were fixed during system design, including a mathematically fixed 0.50 base classification boundary.

5 Results

MABPD performs competitively under zero-shot evaluation, with structured debate providing the main performance gain over reduced configurations and single-agent baselines. Results also show cross-dataset transfer, human-aligned performance, reproducibility, and identifiable error and cost trade-offs.

  • Main Results: 84.7% binary F1 and 83.4% macro F1 on the held-out BABE test split place MABPD within 0.7 pp of supervised SOTA without labeled training data.The full zero-shot dataset results are 86.2% binary F1 and 84.6% macro F1.
  • Ablation Study: Removing debate reduces F1 by 9.3 points on the full dataset and 10.6 points on the test split, primarily because recall falls while precision remains above 93%.The full-system recall decreases by 19.6 pp on the full dataset and 20.1 pp on the test split.
  • Human Evaluation: MABPD achieves 89.3% accuracy against human majority vote, with Fleiss’ κ = 0.66, across 150 stratified articles.Agreement was 92% on biased articles and 87% on neutral articles.
  • Efficiency: The full SAD system uses approximately 6.8× the tokens and 16.5 seconds per isolated article, while costing $0.0065 per article.Concurrent dispatch lowers amortized latency, and the early-exit cascade is designed to skip secondary analysis for high-confidence neutral articles.
  • Ablation Study: Structured deliberation rescues 716 false negatives while adding 152 false positives, yielding 564 net additional correct decisions versus the single-agent configuration.The corrected cases are overwhelmingly rescued false negatives: 716 of 718, or 99.7%.
  • Generalization: 75.0% zero-shot accuracy on SemEval 2019 HyperPartisan falls within 7.2 pp of supervised SOTA, with recall-heavy transfer across the non-overlapping corpus.The transfer result combines 88.8% recall with 61.4% precision.

6 Conclusion

MABPD uses Structured Argument Debate to replace task-specific fine-tuning with asymmetric evidence grounding and role-weighted deliberation, achieving competitive zero-shot performance on BABE and HyperPartisan.

  • 83.4% macro F1 on BABE’s held-out test split is within 0.7 pp of supervised SOTA without task-specific training.
  • Removing debate costs up to 10.6 F1 points, driven by recall collapse of 20.1 pp.
  • 75.0% zero-shot accuracy on HyperPartisan is within 7.2 pp of supervised SOTA.
  • At an amortized 2.0 s/article under concurrent dispatch, the system is practical for corpus-scale deployment.

Limitations

The evaluation is constrained by shared-model dependence, limited ablations and comparability, unavailable serving infrastructure, and restricted dataset and input scope.

  • Using one LLaMA model isolates architecture but may produce correlated errors and lacks genuine epistemic diversity.
  • Primary evaluation is English-only BABE data covering 23 topics, while HyperPartisan shows 88.8% recall versus 61.4% precision.
  • HyperPartisan truncates 206 of 644 articles to 600 words, making later-occurring bias invisible by construction.
  • The MAGPIE comparison is regime-matched, but supervised baselines train on 3,121 articles while MABPD uses no BABE training or threshold-tuning data.
  • The original LLaMA endpoint is unreachable, so exact reported magnitudes cannot be demonstrated across environments and may vary elsewhere.
  • Per-agent ablations and stricter aggregation comparisons are unavailable because the original environment and required intermediate outputs cannot be regenerated.
  • Despite mitigations, the system remains susceptible to echo-chamber convergence and sycophantic consensus at verification.
  • Dynamic weighting and MBIB evaluation are identified extensions but were not evaluated in this paper.

Ethical Considerations

MABPD is framed as an awareness aid rather than an automated arbiter, with safeguards intended to reduce false positives and fabricated evidence while retaining human oversight.

  • Bias detection can suppress legitimate perspectives or mislabel factual reporting, and the underlying LLM may harbor political biases.
  • MABPD surfaces potentially biased passages for human attention and should not replace journalists, editors, analysts, or consequential human judgment.
  • The SAD evidence factor assigns zero weight to biased claims without evidence while retaining partial weight for unsupported neutral claims.
  • The BiasAgent’s taxonomy covers loaded language, framing, selection, emotional language, political labeling, exaggeration, and omission bias.
  • The taxonomy maps to four meta-categories: biased language, framing bias, unsupported claims, and selective presentation.
  • A protected-terms list is intended to reduce false positives on neutral domain vocabulary without using BABE data in its construction.
  • The paper provides architectural weights and decision thresholds alongside the knowledge-base categories injected into agent prompts.

E Construction of the Bias Lexicon

The lexicon combines author-constructed bias terms, ordinal scoring, runtime code defaults, and released artifacts, with documented reproducibility boundaries. The section also reports evaluation and ablation evidence for the resulting pipeline.

  • Scoring: Scores use four ordinal anchors from 0.60 to 0.90+, with the highest-scoring matching term selected above a 0.45 confidence floor.The scores order terms within bands rather than representing calibrated magnitudes.
  • Scoring caveats: Word-level scores were author-assigned without multirater annotation, while removing score information leaves 90.7% of selected evidence unchanged.This bounds both the annotation basis and the measured contribution of score granularity.
  • Evaluation independence: The lexicon was constructed independently of BABE: 336 of 449 terms occur in none of its 4,121 records, and score correlation with record bias rate was not significant.The reported correlation is r = +0.271, below the stated critical value of 0.285.
  • Lexicon structure: 793 runtime terms combine a released 451-entry, 26-category JSON lexicon with 342 additional in-module terms.The code-only terms are active during detection but absent from the released JSON artifacts.
  • Reproducibility boundary: The 26 released categories do not partition the full runtime lexicon because code-only terms lack machine-readable category labels.Consolidating these terms into the deterministic builder remains the principal outstanding resource task.

N Heterogeneous LLM Architecture Pilot

The heterogeneous pilot assigns different LLM architectures to detection, evidence extraction, and framing analysis. Its near-equivalence to the homogeneous system supports protocol-centered rather than model-family-specific performance.

  • Architecture: The pilot uses LLaMA 3.3 70B for detection and consensus, Mixtral 8x7B for evidence extraction, and LLaMA 3.1 8B for framing analysis.The evaluation uses three distinct architectures on the held-out test split with N=1,000.
  • Pilot result: 83.4% Accuracy and 84.6% F1 were achieved by the heterogeneous pipeline on the test split.These figures are reported for the heterogeneous configuration in Table 5 and Figure 6.
  • Interpretation: The heterogeneous pipeline differed from the homogeneous LLaMA-only system by ∆F1 = −0.1 pp, versus −10.6 pp for debate ablation.The heterogeneous difference was comparable to the reported 0.06 pp run-to-run variation.

O Agent Independence Analysis

The primary agents disagree on a substantial minority of articles, especially ambiguous ones, indicating distinct analytical behavior before consensus. The debate protocol then uses these perspectives to produce a final decision.

  • Disagreement: The three primary agents disagree on the final binary label for 38% of articles.The rate was measured across the 4,121-article dataset before debate.
  • Ambiguity: Disagreement correlates with human annotation uncertainty at point-biserial r = 0.62, linking agent divergence to harder articles.The reported relationship supports the interpretation that prompts elicit diverse perspectives on ambiguous cases.

P MAS Failure Mode Analysis

The failure analysis finds low echo-chamber convergence but a remaining sycophancy failure at consensus. The authors identify explicit chain-of-thought verification as a future mitigation.

  • Observed failures: Echo-chamber convergence occurs in 4% of debates, which the authors attribute to strict role separation and functional agent isolation.These labels are the authors’ own debate-specific descriptions rather than separate MAST failure modes.
  • Observed failures: Sycophancy occurs in 11% of debates when the VerifierAgent sometimes accepts a confident but incorrect BiasAgent without deep reasoning-chain verification.The authors classify this under MAST’s task-verification category and propose explicit chain-of-thought verification as future work.
  • Use boundary: The system is presented as an awareness tool, with human review recommended before consequential decisions.The disclosure explicitly states that AI assistants did not execute experiments or generate results.
Loading 2609.04841v1…