PaperlayerView exact revision on arXiv

Source-linked AI summary

The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection

Yuchen Miao, Zijun Wang, Chang Han, Yurui Shi, Mingtai Zhang, Siyang Xu

arXiv:2610.11136v1cs.CL

TL;DR

Bias detection in government documents needs normative grounding, standpoint review, and interventions that preserve administrative meaning, while existing approaches leave gaps in justification and emerging-target coverage. MARS-Gov combines legal retrieval, open-set screening, juror adjudication, conservative routing, and rewrite verification. It reports stronger performance across baseline families, open-set recovery, transfer, and over-governance measures, but depends on localized normative resources.

  • Problem

    Supervised encoders provide limited evidence for normative justification, while zero-shot LLMs can use generic viewpoints and over-flag group mentions in consequential government language.

    Source: §1 · pp. 1–2

  • Method

    MARS-Gov retrieves legal and administrative context, screens for uncovered standpoints with a dynamic 10th juror, routes uncertain cases to a Judge, and verifies minimal rewrites.

    Source: §1 · p. 2

  • Results

    MARS-Gov reports stronger performance than fine-tuned, zero-shot, and generic debate baselines, with higher open-set recovery and BGS, better transfer, and lower over-governance.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The reported results support the conclusion that reliable governance NLP requires legally grounded evidence, explicit standpoint reasoning, and verification after interventions.

    Source: §5 · p. 9

    MARS-Gov's strongest setting is Dutch administrative text, and other jurisdictions or platforms require localized legal and policy resources.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Presupposing the boundaries of bias is itself a form of bias. We study closed-loop bias governance for Dutch government documents, where a system must detect biased language, ground decisions in legal and contextual evidence, rewrite problematic sentences when intervention is warranted, and verify that the rewrite mitigates harm without distorting meaning. Existing methods face three challenges: (i) discriminative classifiers capture surface regularities but lack normative grounding; (ii) zero-shot LLMs often adopt generic viewpoints and over-flag ambiguous administrative language; and (iii) fixed taxonomies inherit the Closed-World Assumption, missing emerging local targets. We propose MARS-Gov, a standpoint-aware multi-agent framework that combines legal retrieval, open-set target screening, specialized jurors, conservative routing, and rewrite verification. When screening finds an uncovered group, MARS-Gov instantiates a dynamic "10th juror" to deliberate outside the fixed panel. On DGDB, MARS-Gov sets a new SOTA with 0.880 F1, outperforming the strongest zero-shot LLM detector by 20.2 points (29.8% relative) and the best supervised Dutch encoder by 6.8 points, while reducing unnecessary interventions to 2.5%. Leave-One-Category-Out (LOCO) evaluation recovers held-out categories with 85.1% Correct@1 and 93.8% Correct@3.

1 Introduction

Bias detection in government documents requires normative grounding, standpoint identification, and interventions that preserve administrative meaning. MARS-Gov addresses the limits of fixed taxonomies through an auditable, open-set governance loop.

1 Introduction

  • Government language shapes how citizens are treated, and linguistic neutrality matters for institutional trust.Source: §1 · p. 1
  • Reliable review must ground judgments in legal and contextual norms, distinguish eligibility conditions from stigmatizing proxies, and verify that rewrites preserve official obligations.Source: §1 · p. 1
  • Supervised encoders offer limited evidence for normative justification, while zero-shot LLMs can adopt generic viewpoints, over-flag group mentions, and omit supporting norms.Source: §1 · pp. 1–2
  • Fixed taxonomies can miss emerging targets or reduce them to a nearby standpoint, losing the affected group's specific rationale.Source: §1 · p. 2
  • MARS-Gov retrieves legal and administrative context, screens for uncovered groups with a dynamic 10th juror, routes uncertain cases to a grounded Judge, and verifies minimal rewrites.Source: §1 · p. 2
  • The paper frames bureaucratic bias detection as closed-loop governance involving evidence retrieval, minimal rewriting, and verification, rather than one-shot classification.Source: §1 · p. 2
  • The stated contributions include open-set and cross-lingual robustness and governance metrics for mitigation quality and over-governance.Source: §1 · p. 2

2 Related Work

Prior work supplies tools for bias detection, legal retrieval, and open-set recognition, but leaves gaps in emerging-target review and end-to-end governance. MARS-Gov combines these strands with standpoint-specific agents and auditable intervention.

2 Related Work

  • Bias in government documents reflects institutional practice as well as model behavior, and the same phrase can be harmless procedurally but discriminatory in context.Source: §2 · p. 2
  • Supervised encoders and controlled benchmarks remain useful closed-set baselines, but fixed labels can be brittle for emerging administrative targets and do not establish faithful repairs.Source: §2 · p. 2
  • LLMs can apply natural-language criteria and retrieval can add legal evidence, but retrieval alone does not adjudicate standpoints or determine whether a rewrite repairs the problem.Source: §2 · pp. 2–3
  • Open-set recognition and multi-agent systems inform the approach, while MARS-Gov combines legal grounding, target discovery, specialized standpoints, and closed-loop rewriting.Source: §2 · p. 3, Figure 2 · p. 3

3 Methodology

The section frames bias governance as a verified intervention task and specifies a pipeline that grounds judgments in retrieved norms, expands review beyond fixed categories, and measures mitigation alongside side effects.

3.1 Task Definition: Closed-loop Bias Governance

  • MARS-Gov treats governance as detecting bias, proposing a faithful repair when needed, and verifying that the restored sentence is no longer judged biased.The system also stores retrieved evidence and juror reports for audit, while keeping the rewrite narrow enough to preserve the administrative fact pattern.Source: §3.1 · p. 3

3.2 Overview: Pipeline-defined Detector

  • The pipeline separates evidence retrieval, standpoint assignment, routing, adjudication, intervention, and verification into distinct stages.The trace records evidence, affected-group hypotheses, juror reports, routing, the Judge’s rationale, any restoration, and its verification result.Source: §3.2 · pp. 3–4

3.3 Normative Evidence Contextualization via RAG (NEC)

  • NEC retrieves legal and administrative guidance through keyword retrieval, semantic retrieval, and chunk-level context expansion, then summarizes it without issuing a verdict.This gives downstream agents a shared reference frame while leaving later judgments traceable to retrieved context.Source: §3.3 · p. 4, Figure 3 · p. 4

3.4 Open-Set Standpoint Screening (OSS): Scout + Juror Panel

  • OSS maps Scout-identified groups to nine fixed categories and instantiates a temporary, evidence-conditioned juror for each uncovered target.Separate dynamic reports preserve distinct standpoints; the final decision still passes through the same Judge, evidence, and verification steps as known categories.Source: §3.4 · pp. 4–5

3.5 Alert-based Routing Gate (ARG): Risk-sensitive Escalation

  • ARG clears a sentence only when risk signals are weak and well-formed; hard, consistent soft, or fail-safe alerts send it to escalation.The rule-based gate adds no backbone calls; escalated cases go to the Judge and, when needed, restoration.Source: §3.5 · p. 5

3.6 Decision and Restoration: Judge and Verify with Minimal Rewriting

  • For escalated cases, the Judge resolves juror conflicts using the sentence, retrieved evidence, and reports; restoration removes biased framing while preserving meaning and facts, then undergoes verification.A governance action succeeds only when a non-empty restoration is cleared by the same detector; broad paraphrasing is avoided to preserve administrative details.Source: §3.6 · p. 5

3.7 Pipeline-aware Governance Metrics

  • BMR, SF, and BGS measure mitigation and meaning preservation, while OGR measures unnecessary intervention on non-biased inputs.BGS combines recall, mitigation, and fidelity; these measures complement F1 by capturing over-rewriting and rewrites that fail verification.Source: §3.7 · pp. 5–6

4 Experiments

The experiments address five questions spanning baseline performance, recovery of held-out categories, component and efficiency effects, and cross-domain and cross-lingual transfer. Across these evaluations, MARS-Gov is compared with fine-tuned, zero-shot, and agentic systems on Dutch and external benchmarks.

4.1 Experimental Setup

  • The evaluation uses DGDB and DGDB-Unseen alongside DALC, SBIC, and KoBBQ, comparing fine-tuned encoders, zero-shot LLMs, agentic controls, and matched Gemini prompting baselines.DGDB-Unseen replaces selected bias terms with unseen expressions while preserving labels and administrative context.Source: §4.1 · p. 6

4.2 Overall Performance (RQ1)

  • MARS-Gov exceeds 0.85 F1 across four backbones, with Gemini 2.5 Pro achieving the best mean F1, 0.880.The best configuration improves BGS to 0.808; the full staged procedure is 6.8 F1 points and 7.3 BGS points above Single-Call, while reducing OGR from 6.7% to 2.5%.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • DGDB-Unseen F1OOD is 0.823–0.857 for MARS-Gov, versus 0.554 for BERTje and 0.499 for RobBERT; Gemini 2.5 Pro also reaches 0.895 recall, 0.865 precision, and 2.5% OGR.MARS-Gov's ∆Drop is −3.6% to −2.4%, compared with −31.8% for BERTje and −38.5% for RobBERT; Grok-3 has 0.920 recall and 19.1% OGR.Source: §4.2 · p. 7

4.3 Controlled Open-set Evaluation with Held-out Categories (RQ2)

  • 85.1% Correct@1 and 93.8% Correct@3 recover held-out categories with Scout-guided review, which raises average F1 to 0.875 from 0.807 without the Scout.The LOCO evaluation removes one juror–watchlist pair at a time; gains are pronounced for categories with indirect cues, including Colonialism and Education/Class.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 Ablation & Efficiency Analysis (RQ3–RQ4)

  • Removing the Scout lowers F1 by 0.035 and recall by 5.5%; removing NEC yields F1 0.824, while removing verification drops BGS from 0.808 to 0.646 and raises BMR to 98%.The ablation identifies Scout-guided dynamic review, legal grounding, and rewrite verification as distinct contributors to detection or governance quality.Source: Table 3 · p. 8, §4.4 · p. 8
  • At τ = 2, Figure 4 shows a stable elbow where over-governance falls with little recall loss across backbones.Figure 4 plots recall against Over-Governance Rate as the threshold increases.Source: Figure 4 · p. 8, §4.4 · p. 8
  • With Gemini 2.5 Pro, ARG clears 64.8% of inputs early and reduces calls from 28.6 to 12.4, latency from 41.5s to 19.2s, and cost from $18.30 to $7.35 per 1,000 sentences.The gate adds no LLM call, and removing it leaves detection unchanged while roughly doubling cost.Source: Table 4 · p. 8, §4.4 · p. 8

4.5 Cross-Domain and Cross-Lingual Generalization (RQ5)

  • MARS-Gov remains strongest on DALC, SBIC, and KoBBQ, adding 20–30 precision points while keeping recall comparable; absolute F1 remains below in-domain DGDB.The external benchmarks shift genre, label assumptions, and language; transfer supports portability of the review procedure, not Dutch law, so another jurisdiction needs local legal and policy evidence.Source: Table 5 · p. 9, §4.5 · p. 9

5 Conclusion

MARS-Gov combines legal retrieval, open-set screening, juror adjudication, conservative routing, and rewrite verification for Dutch government documents. Experiments report stronger performance across several baselines and conclude that reliable governance NLP requires legally grounded evidence, standpoint reasoning, and post-intervention verification.

5 Conclusion

  • Experiments show stronger performance than fine-tuned, zero-shot, and generic debate baselines, with higher open-set recovery and BGS, better transfer, and lower over-governance.The conclusion also reports better cross-domain and cross-lingual transfer.Source: §5 · p. 9
  • MARS-Gov combines legal retrieval, open-set target screening, juror-based adjudication, conservative routing, and rewrite verification rather than treating bias detection as one-shot classification over a fixed taxonomy.Source: §5 · p. 9
  • DGDB-Unseen results show that compact encoders remain competitive on familiar terms but lose reliability when public language shifts.Source: §5 · p. 9
  • Future work will extend MARS-Gov to additional jurisdictions and reduce inference cost for broader evaluation and lower-cost deployment.Source: §5 · p. 9

Limitations

The framework’s strongest setting is Dutch administrative text, and its multi-agent pipeline carries inference costs; BGS also compresses distinct governance trade-offs into one score.

Limitations

  • Applying MARS-Gov to other jurisdictions or platforms requires localized legal and policy resources, not prompt translation alone.Without those resources, its justifications may be linguistically plausible but normatively incorrect.Source: Limitations · p. 9
  • The multi-agent pipeline is more expensive and slower than lightweight discriminative models, limiting high-throughput or real-time use despite the routing gate.The authors identify reducing model calls without weakening legal grounding or verification as future work.Source: Limitations · p. 9
  • Similar BGS scores can conceal different trade-offs between detection coverage, mitigation success, and meaning preservation, so component-level analysis remains necessary.BGS is reported as a compact summary, not a substitute for its individual components.Source: Limitations · p. 9

Ethics Statement

The ethics statement frames MARS-Gov as human-reviewed decision support, not an autonomous decision-maker, and calls for preserving records and scrutinizing the system’s normative resources and dual-use risks. The appendix index lists extended benchmarks, LOCO prototypes, decision traces, retrieval details, sensitivity results, and prompt templates.

Ethics Statement

  • MARS-Gov should support qualified human review rather than make decisions or rewrite official records autonomously.The original text should remain in an auditable record, and neutralized wording does not establish that discriminatory practice has ended.Source: Ethics Statement · pp. 9–10
  • Watchlists, criteria, and protected-group standpoints should be documented, versioned, audited, and revised with affected communities, experts, and legal stakeholders.High-risk outputs require manual verification, error analysis, and disparate-impact monitoring; model confidence is not correctness.Source: Ethics Statement · p. 10
  • Because rewriting could sanitize discriminatory policy without changing it, the authors release research artifacts rather than a turnkey live tool and do not recommend fully automated government decision pipelines.Source: Ethics Statement · p. 10

Appendix Index

  • The appendix index covers extended model benchmarks and results, cross-lingual reasoning, LOCO prototypes, end-to-end traces, retrieval resources, routing sensitivity, and prompt templates.Source: Appendix Index · p. 12

A Dataset Overview

The dataset overview directs readers to Tables 6 and 7 for the datasets used in the experiments.

A Dataset Overview

B Extended Benchmarks across Model Families

Across 18 LLMs, MARS-Gov gains vary with backbone reasoning capacity, and native Dutch prompting gives a consistent F1 advantage over English prompts. The extended evaluation also finds that administrative terms can lose meaning in translation.

B Extended Benchmarks across Model Families

B.1 Analysis of Extended Results

  • MARS-Gov broadly improves results across backbones, but only models around the GPT-4 Turbo or Llama-3-70B tier reliably exceed the fine-tuned BERTje baseline.Open-weight Qwen-2.5-72B and DeepSeek-V3 also exceed that baseline.Source: §B.1 · p. 13

B.2 Cross-Lingual Reasoning

  • Dutch prompts yield a +0.8% to +2.3% F1 advantage across all four backbones, as administrative terms lose meaning after translation.The comparison uses Dutch prompts and English prompts produced through DeepL translation.Source: §B.2 · p. 13

C LOCO Prototypes

The appendix details LOCO prototype mapping, representative decision traces, and the NEC knowledge base. It also reports extended model-family results and a consistent advantage for native Dutch prompts.

C LOCO Prototypes

  • Native Dutch prompting consistently outperforms English prompts across backbones, avoiding semantic loss in translated administrative terminology.The comparison reports a statistically significant difference (p < 0.05).Source: Figure 5 · p. 13

C.1 Fixed Prototype Texts for Deterministic Mapping

  • LOCO uses clips/e5-base-trm-nl to rank fixed prototypes, mapping Scout-generated groups and rationales to calculate Correct@1 and Correct@3.The prototype texts were fixed before the experiment using dataset statistics and definitions; NEC uses a separate retrieval embedding model.Source: §C.1 · p. 13, C LOCO Prototypes · p. 13
  • Across model families, MARS-Gov averages +0.20 ∆F1, with larger gains for frontier backbones; about GPT-4 Turbo-level reasoning is needed to surpass supervised baselines.The table uses fine-tuned BERTje at F1 = 0.812 as the performance threshold.Source: Table 8 · p. 14

Case T1: Intersectional welfare-fraud profiling

  • In the welfare-fraud trace, jurors flag group-based fraud-risk and incapacity claims, while the rewrite preserves the fraud-control goal and uses objective risk indicators.The Scout identifies migration, gender, and neighborhood/class proxies.Source: Case T1: Intersectional welfare-fraud… · p. 14

Case T2: Language proficiency as intent inference

  • In the language-proficiency trace, jurors flag the unsupported link to integration failure and intentional deception; the rewrite reframes the observation as a communication-support need.The original administrative fact is preserved.Source: Case T2: Language proficiency as… · p. 14

Case T3: Objective income threshold

E.1 DGDB NEC Retrieval Setting

  • DGDB NEC retrieval combines legal and policy foundations, accessibility and administrative guidance, discrimination and racism reports, inclusive-language resources, and category reasoning notes.The appendix describes this as the DGDB retrieval setting.Source: §E.1 · p. 15

E.2 Knowledge Base Composition

  • The NEC knowledge base has legal-policy, administrative-accessibility, and sociocultural-inclusive-language layers, including official guidance and DGDB reasoning entries.Table 9 summarizes the inventory’s support for DGDB retrieval.Source: §E.2 · p. 15

E.3 Implementation Parameters

NEC uses complementary retrieval stages and preserves source context for inspection. ARG escalation thresholds are tuned around the two parameters identified as most influential.

E.3 Implementation Parameters

  • The default NEC setting selects k = 5 seed fragments before local context expansion, and Table 9 describes its prioritization of normative sources and official writing guidance..Source: §E.3 · p. 15, Table 9 · p. 15
  • NEC combines keyword matching, dense semantic retrieval, and local chunk expansion; retained source identifiers and chunk positions make evidence inspectable, while the Summary Agent cannot classify.Keyword retrieval preserves precise terminology, dense retrieval recovers paraphrases, and expansion restores nearby definitions or caveats.Source: §E.3 · p. 15

F ARG Hyperparameter Sensitivity Grid

  • ARG tunes five escalation parameters; its heatmap varies τhigh and Nsoft_bias, the two parameters identified as most influential.The validation-set tuning is introduced with Gemini 2.5 Pro.Source: F ARG Hyperparameter Sensitivity Grid · p. 15

F.1 Performance Heatmaps

The selected ARG setting balances detection performance and over-governance. The appendix’s prompt templates keep pipeline stages and agent roles separately specified.

F.1 Performance Heatmaps

  • F1 = 0.880 with OGR = 2.5% at τhigh = 0.6 and Nsoft_bias = 2; lower thresholds bypass jurors too often, while higher ones increase calls without improving F1.Figure 6 plots F1 and OGR across τhigh and juror-vote thresholds.Source: §F.1 · p. 15, Figure 6 · p. 16

G English and Dutch Prompt Templates

  • Prompt boundaries restrict each agent to a stage and output contract, keeping evidence summarization, standpoint discovery, adjudication, restoration, and verification separately inspectable.Malformed outputs are exposed to routing logic rather than interpreted heuristically.Source: G English and Dutch Prompt Templates · p. 16

G.1 English Prompt Templates

  • The context summarizer returns neutral, relevant evidence fragments with labels such as definition, criterion, guideline, caveat, or scope, without making a bias decision.When no relevant evidence exists, it returns one bullet stating that none was found.Source: §G.1 · p. 16
  • The Scout identifies explicitly mentioned or implied groups, especially those outside the fixed panel, and returns structured group labels, evidence, and a prompt hint for newly surfaced groups.It is instructed not to judge whether the sentence is biased.Source: §G.1 · p. 16
Loading 2610.11136v1…