PaperlayerView exact revision on arXiv

Source-linked AI summary

JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification

Tianxin Zhou, Ruixi Lin

arXiv:2608.20607v1cs.CLcs.AIcs.LG

TL;DR

Reference-free judge agreement can conceal shared false-negative errors, making majority acceptance risky in factuality evaluation. JuryProbe measures panel-level consensus risk from calibration and routes high-risk accepts to trusted-reference verification; its supported contribution is an empirical diagnostic, not a formal safety guarantee or reliable natural-panel stand-down rule.

  • Problem

    Reference-free factuality panels lack evidence that agreement reflects independent judge errors, despite their use in accept-or-escalate decisions.

  • Method

    JuryProbe estimates panel-level consensus risk from a labeled calibration probe using FN-only judge correlation and false-consensus lift, then routes high-risk majority accepts to grounded judges.

  • Results

    In flagged settings, routed evaluation matches grounding every reference-free majority accept in 34/34 splits, while the fixed rule stands down on a negative control and avoids roughly 28% of reference acquisitions.

  • Takeaways & Limitations

    JuryProbe supports using calibration-based consensus risk to decide when reference-free agreement should activate accept-conditioned grounding rather than serve as sufficient evidence.

  • Takeaways & Limitations

    JuryProbe provides no formal risk guarantee and does not establish when natural factuality-panel agreement can be trusted without grounding.

Abstract

from arXiv · show

Panels of inexpensive LLM judges increasingly make accept-or-escalate decisions. In factuality settings, accepting a claim because several reference-free judges agree can create a hidden risk: agreement may reflect shared false-negative blind spots rather than independent evidence. We introduce JuryProbe, an empirical consensus-risk diagnostic for reference-free factuality judge panels, paired with a calibration-based routing policy. JuryProbe estimates consensus risk from a labeled calibration probe using false-negative-only (FN-only) judge correlation and false-consensus lift; when flagged high-risk, reference-free majority accepts are routed to the same judges with trusted references. On audited FEVER corruptions, reference-free panels show correlated false negatives (FN-only correlations 0.402 and 0.368; lifts 3.13x and 18.13x), while unanimous false consensus drops to zero under a trusted-reference best-case diagnostic on both minimal-pair and non-minimal-pair evidence. In flagged settings, the routed policy is by construction equivalent to grounding every reference-free majority accept (verified in 34/34 splits): improvement comes from accept-conditioned grounding, while the diagnostic determines whether to activate it. A fixed, pre-specified rule flags 8-10 of 10 splits across synthetic, benchmark-authored, and scientific families and 0 of 10 on a negative control, where standing down avoids 28% of reference acquisitions at a 0.004 increase in false accepts. False-accept reduction persists under weak BM25 retrieval at substantial coverage cost, while stale stand-down labels require periodic recalibration. JuryProbe provides no formal risk guarantee and does not establish reliable stand-down on natural panels; its supported contribution is an empirical diagnostic of high-risk panel error dependence.

1 Introduction

JuryProbe treats reference-free judge agreement as a consensus-risk routing problem rather than inherently reliable evidence. It measures correlated false negatives and routes high-risk accepts to trusted-reference verification.

  • Motivation: JuryProbe asks when agreement from a cheap reference-free factuality panel should be trusted or routed to grounded verification.The paper frames the issue as a measurement-and-routing problem.
  • Consensus-risk diagnostic: FN-only correlation measures shared failures, while false-consensus lift measures their unanimous-acceptance consequence.Together, the statistics define a panel-level risk regime rather than a sample-level classifier.
  • Routing rationale: Disagreement-based escalation cannot catch unanimous false acceptance, so JuryProbe routes accepts when calibration identifies a high-risk consensus regime.The diagnostic is empirical and does not certify that an unflagged panel is safe.
  • Grounding diagnostic: Trusted-reference evaluation finds no unanimous false consensus for the same judges on minimal-pair and non-minimal-pair evidence.This is presented as a best-case trusted-reference diagnostic, not causal isolation of grounding’s effect.
  • Policy evaluation: In every flagged split, JuryProbe-Routed matches Ground-All-Accepts in 34/34 splits, making the diagnostic determine whether accept-conditioned grounding activates.The fixed rule yields both labels across eight claim families, while negative-control stand-down avoids approximately 28% of reference acquisitions.
  • Conclusion: The paper argues that risk-aware grounding is preferable to relying on reference-free agreement or simply adding more reference-free judges.Its supported contribution is an empirical diagnostic of high-risk panel error dependence.

2 Related Work

JuryProbe builds on work about LLM judges, judge-panel independence, factuality verification, and selective prediction. Its distinctive contribution is routing reference-free factuality-panel accepts using measured consensus risk rather than disagreement or uncertainty.

  • LLM evaluation panels: LLM-as-a-judge research established scalable evaluation, while later work motivated panels of smaller judges as alternatives to a single expensive judge.JuryProbe studies when such reference-free panel agreement should no longer be treated as reliable evidence.
  • Judge independence: Correlated judge errors reduce the number of effective independent votes, motivating measurement of shared failures in reference-free factuality panels.JuryProbe narrows this concern to correlated false negatives on corrupted factual claims.
  • Factuality verification: Existing factuality work includes atomic fact decomposition, hallucination detection, sampling-based consistency, and retrieval-augmented verification.JuryProbe uses grounding as an intervention and escalation target rather than proposing a new factuality verifier.
  • Selective escalation: Selective prediction and related LLM-evaluation methods defer uncertain or disagreement-heavy cases, whereas JuryProbe routes based on measured consensus risk.This distinction matters because unanimous false acceptance is invisible to disagreement-based escalation.

3 JuryProbe Framework

JuryProbe treats correlated false negatives as a panel-level consensus risk and routes reference-free accepts to grounded verification when calibration flags that risk. Its asymmetric policy protects against unsafe acceptance while keeping the risk label fixed during deployment.

  • 3 JuryProbe Framework: JuryProbe decides when reference-free panel agreement is insufficient evidence to accept a factuality claim.
  • 3 JuryProbe Framework: False-consensus events occur when all judges accept the same corrupted claim, which disagreement-based escalation cannot detect.
  • 3.4 JuryProbe-Routed Policy: Consensus risk is estimated once on calibration data and is not recomputed for individual deployment claims.
  • 3.2 Consensus Risk: FN-only correlation measures shared misses, while false-consensus lift measures excess unanimous false acceptance relative to independent errors.
  • 3.2 Consensus Risk: A panel is high-risk when calibration satisfies ρFN > 0.15, LFC > 1.5, and p < 0.05.The thresholds define a panel-level risk regime rather than an item-level classifier and are fixed before deployment evaluation.
  • 3.4 JuryProbe-Routed Policy: The routed policy grounds reference-free majority accepts when the fixed calibration label indicates high risk, while reference-free rejects remain unchanged.This one-sided design cannot increase the false-accept rate relative to reference-free judging by construction.

4 Experimental Setup

The evaluation uses frozen, audited factuality corruptions, fixed three-judge panels, held-out calibration and deployment splits, and repeated seeds. It measures risk, grounding behavior, policy quality, and verification cost against routing and grounding baselines.

  • 4 Experimental Setup: The controlled setting uses trusted references for clean claims and known non-factual labels for corrupted claims.
  • 4 Experimental Setup: Number and Entity corruption families each contain 300 clean and 300 corrupted examples, with budgets fixed before judge evaluation.
  • 4 Experimental Setup: Five additional frozen families extend evaluation across FEVER controls, benchmark-authored claims, and scientific evidence.
  • 4 Experimental Setup: The same three LLM judges perform both reference-free judging and grounded verification, keeping judge composition constant.
  • 4 Experimental Setup: Policy evaluation reports false-accept and true-accept rates, residual false consensus, verifier items, and verifier calls.
  • 4 Experimental Setup: Each split contains 150 clean and 150 corrupted calibration examples and an equally sized deployment set, with the risk label fixed before deployment evaluation.
  • 4 Experimental Setup: Ten split seeds test stability across calibration and deployment partitions, alongside baselines including Ground-All-Accepts, Disagreement-Routed, and Random-Routed.

5 Results

The results show correlated false negatives in reference-free panels and evaluate trusted-reference grounding, routing identity, stand-down, and cross-family behavior. Grounding removes observed false consensus in audited settings, while the fixed rule selectively activates grounding with control-specific trade-offs.

  • 5.1 Consensus Risk in Reference-Free Judging: 0.402 and 0.368 FN-only correlations, with 3.13× and 18.13× false-consensus lifts, indicate substantial consensus risk in both confirmatory families.The statistics quantify dependence among judges’ failures and the excess of unanimous false acceptance over independence expectations.
  • 5.1 Consensus Risk in Reference-Free Judging: Reference-free agreement can reflect correlated false-negative failures rather than independent confirmation.Attribute reproduces the signal, but its low absolute false-consensus rate limits its use for paired grounding analysis.
  • 5.2 Trusted-Reference Grounding Diagnostic: 0.000 grounded false-consensus rates occur for both Number and Entity, compared with 0.159 and 0.031 under reference-free judging.The same judges and aggregation rule were evaluated on the same detectable corrupted claims with and without trusted references.
  • 5.2 Trusted-Reference Grounding Diagnostic: 0/190 three-judge false consensuses occur with either SciFact reference, versus 10/190 reference-free, while pairwise FN correlations remain nonzero.The non-minimal-pair evaluation used benchmark rationales and full published abstracts; mean correlations were 0.247 and 0.108, respectively.
  • 5.3 Policy Evaluation: Baselines, Identity, and Stand-Down: 34/34 flagged splits make JuryProbe-Routed identical to Ground-All-Accepts, so improvements arise from grounding reference-free majority accepts.The diagnostic’s distinct operational role is deciding whether to activate accept-conditioned grounding.
  • 5.3 Policy Evaluation: Baselines, Identity, and Stand-Down: 49.6% fewer verifier calls for Number and 62.1% fewer for Entity accompany elimination of false accepts, while clean-claim acceptance falls to 58–64%.Disagreement routing leaves residual false consensus, and random routing does not eliminate false accepts at the same verifier-call budget.
  • 5.3 Policy Evaluation: Baselines, Identity, and Stand-Down: 84.7 ± 4.0 references per split, about 28% of deployment claims, are saved on the negative control, while false accepts decrease from 0.013 to 0.009.The rule stands down in 10/10 splits and overturns 1 of 4 reference-free accepted corruptions without overturning accepted clean items.
  • 5.4 Cross-Family Evaluation of the Fixed Rule: 8–10/10 signal-family splits are flagged, compared with 0/10 for the negative control and 6/10 for the boundary control.The fixed thresholds are applied without per-family tuning, though evaluation remains replication when calibration comes from the evaluated family.

6 Discussion

JuryProbe is useful only in a conditional operating regime where false accepts are costly, trusted references are available but expensive, and accept coverage can be traded for fewer acquisitions. Its labels can become stale under distribution shift, and the study provides no formal guarantee or reliable natural-panel stand-down.

  • Operating regime: JuryProbe is useful when false accepts are costly, trusted references are available but expensive to acquire universally, and some true-accept coverage can be traded for fewer references.It is not presented as an unconditional improvement over Always Grounded.
  • Operating regime: A one-time labeled calibration probe is amortized over deployment volume, while trusted-reference acquisition is the relevant scaling cost.For a three-judge evaluation, LLM-call cost is about 10^-5 USD per claim; the calibration probe uses 300 labeled items per family.
  • Distribution shift and recalibration: Stand-down labels can become stale under distribution shift, so the tested shifts support periodic labeled recalibration rather than zero-shot drift detection.At 100% shift, false-accept rates rise to 0.429 for Number, 0.111 for Entity, and 0.198 for SciFact without recalibration.
  • Limitations: JuryProbe has no distribution-free guarantee: an unflagged split does not certify zero error, but only indicates that calibration criteria for grounded routing were not met.Users cannot specify a target error rate and obtain a rule guaranteed to meet it.
  • Limitations: The study does not establish reliable stand-down for natural panels and is limited to short, self-contained binary factuality claims evaluated by small open-weight panels.Natural-data evaluations include a boundary case and scientific folds that are flagged throughout.
  • Limitations: Grounded results outside the retrieval stress test are trusted-reference best-case diagnostics, while behavior with degraded or adversarial references remains untested.The BM25 stress test covers one deliberately weak retriever on a single benchmark.
  • Limitations: The paired comparisons diagnose adding trusted references rather than isolating reference wording, prompt framing, or other protocol-level effects.Pairwise false-negative correlation also persists under grounding on SciFact.
  • Limitations: JuryProbe protects accepts but does not recover additional true accepts rejected by the reference-free panel.Detectable-subset analysis reduces but cannot fully eliminate latent item-difficulty sources.

7 Conclusion

JuryProbe diagnoses whether reference-free factuality-panel agreement reflects correlated false negatives and routes high-risk accepts to grounded verification. Across the evaluated settings, it reduced false acceptance under the stated conditions, while remaining an empirical diagnostic without formal guarantees or demonstrated natural-panel stand-down.

  • Conclusion: JuryProbe assesses correlated false-negative errors that can produce false consensus, a failure mode disagreement-based escalation cannot detect by construction.The paper treats consensus risk as a routing problem rather than assuming agreement implies reliability.
  • Conclusion: In flagged splits, routing was identical to grounding every reference-free majority accept in 34/34 splits, while the diagnostic determined whether to activate that policy.The resulting improvement therefore comes from accept-conditioned grounding.
  • Conclusion: A fixed rule produced both high-risk and stand-down labels across eight claim families, stood down on every negative-control split, and avoided roughly 28% of reference acquisitions at a 0.004 false-accept increase.False-accept reduction persisted with weak BM25 retrieval, but with substantial coverage loss.
  • Conclusion: The conclusions remain bounded by no formal risk guarantee, no reliable natural-panel stand-down result, and trusted-reference best-case grounding outside retrieval stress testing.Agreement should not be treated as sufficient evidence for acceptance unless judge-error dependence is understood.

8 Broader Impact Statement

The paper frames deployment cautions around recalibration and reference quality, emphasizing that automated factuality judging requires validation before high-stakes use.

  • Deployment cautions: Stand-down decisions are specific to the evaluated panel and data distribution, so changes in models, prompts, retrieval, or data require a fresh labeled audit.A stale label cannot detect open-world distribution shift.
  • Deployment cautions: Trusted references may be incomplete, incorrect, outdated, or manipulated, and the weak-retriever stress test does not establish robustness to adversarial or production sources.The stress test measures degradation under one deliberately weak retriever.
  • Deployment cautions: Automated factuality judging should not replace expert review in high-stakes applications without domain-specific validation.

B Additional Robustness and Artifact Details

The additional analyses document evaluation integrity, corruption construction, detectability variants, and robustness checks for JuryProbe's consensus-risk conclusions.

  • Robustness checks: The robustness checks test threshold choice, grounded specificity, random-routing variance, evaluation integrity, and judge-panel composition using frozen datasets and cached outputs.Grounded-specificity checks test whether the estimator is an always-on trigger, while the negative control evaluates reference-free stand-down.
  • False-consensus cases: Representative cases show all three reference-free judges accepting corrupted claims that grounded judging rejects when trusted references contradict the modified facts.These cases directly illustrate why disagreement-based routing misses unanimous false acceptance.
  • Evaluation protocol: Judges receive binary decisions from claims alone in the reference-free condition, while grounded judging adds a trusted reference and tests consistency with it.
  • Corruption construction: Number corruptions modify numerical facts and Entity corruptions replace named entities with plausible alternatives; both families were frozen before evaluation.Number and Entity were selected as confirmatory families after author audits.
  • Corruption construction: Relation corruptions were excluded before main evaluation because audits found unstable fluency, syntax, and semantic-naturalness artifacts.
  • Detectability analysis: False-consensus lift and paired grounding analyses use a GPT-4o-detectable corrupted subset, with alternative all-corrupted and grounded-detectable subsets checking filter dependence.GPT-4o is neither a JuryProbe judge nor the grounded verifier or routing policy.
  • Detectability analysis: Across detectability definitions, reference-free panels show excess false consensus while grounded false consensus is not observed.
  • SciFact diagnostic: The SciFact grounding diagnostic evaluates 190 contradicted and 190 supported claims using benchmark rationales and full published abstracts.The abstracts average 274 tokens and encode semantic rather than single-token contradictions.

F Cross-Family Evaluation: Split-Level Results and Full Statistics

The cross-family statistics quantify consensus-risk lift while highlighting instability when the independence null is near zero. CREAK also shows measurable reference-free deployment performance, but some grounded comparisons are unavailable or analytic only.

  • Near-zero independence nulls can make cross-split mean lifts unstable and incomparable across families.A single false-consensus event may produce an extreme ratio when marginal false-negative rates are very low.
  • On CREAK, reference-free deployment yields a 0.029±0.010 false-accept rate and a 0.767±0.024 true-accept rate.
  • FEVER-Refutes lacks a trusted reference, so neither grounded policy is evaluable there.
  • The Obvious-Number boundary control has no grounded cache, making its grounded-policy rates analytic intervals rather than measured policy performance.

G Distribution-Shift Recalibration Grid

The recalibration stress test measures how often a labeled audit flags shifted target families across false-claim shares and audit sizes. It does not establish zero-shot safety under arbitrary open-world distribution shift.

  • 0/10 splits are flagged by the initial negative-control-only calibration across all audit sizes.Separate labeled recalibration audits report flags as a function of target false-claim share and budgets of 25, 50, or 95 corrupted items.
  • The recalibration grid reports flagged-split counts out of 10 across target-family false-claim shares and labeled-audit budgets.
  • The stress test evaluates listed benchmark shifts after a new labeled audit, not zero-shot safety under arbitrary open-world drift.

H SciFact Panels: Fold-level Results and Wilson Intervals

The SciFact evaluation combines pooled and fold-level panel statistics with grounded-reference stress tests, held-out deployment checks, and a utility analysis. Results show family-dependent operating bands, while the analysis remains bounded by empirical panel, source, and utility assumptions.

  • H SciFact Panels: Fold-level Results and Wilson Intervals: All SciFact folds are flagged, so routed outcomes exercise grounded verification rather than the stand-down branch.Ground-All-Accepts uses the full published abstract as reference, not gold-label substitutions.
  • H SciFact Panels: Fold-level Results and Wilson Intervals: The SciFact evaluation reports pooled out-of-fold rates on 190 contradicted and 190 supported claims with Wilson 95% intervals.
  • H SciFact Panels: Fold-level Results and Wilson Intervals: BM25@3 verifier-majority false-accept rate is 0.012 on retrieval hits and 0.047 on misses, an association rather than a mechanism test.
  • H SciFact Panels: Fold-level Results and Wilson Intervals: The held-out check recomputes routing on disjoint deployment halves while estimating the risk flag from calibration data only.Every full-subset figure falls within one split-level standard deviation of the held-out mean, and no policy parameters are fitted beyond the binary flag.
  • H SciFact Panels: Fold-level Results and Wilson Intervals: The utility model assumes linear, risk-neutral, per-claim-additive utility and stable class-conditional rates when reweighting balanced-split results.Boundary constants are means over 10 splits, and rates are specific to the evaluated panels, claim families, and reference sources.
  • H SciFact Panels: Fold-level Results and Wilson Intervals: In flagged Number and Entity settings, routed and Ground-All-Accepts policies are identical, with false-accept rates of 0.000 and reference use of 0.504 and 0.379 per claim, respectively.Reference-free majority has false-accept rates of 0.427 for Number and 0.119 for Entity.
  • H SciFact Panels: Fold-level Results and Wilson Intervals: The executed negative-control Always Grounded condition has a 0.138 false-accept rate, correcting an earlier oracle-label simulation reported as 0.000.The correction does not change the other negative-control results.
  • H SciFact Panels: Fold-level Results and Wilson Intervals: The flagged operating band is 0.29–0.42 vTP < cref < 0.16–0.42 cFA, and it is non-empty only when false-accept cost is approximately 1× to 1.9× true-accept value.
Loading 2608.20607v1…