PaperlayerView exact revision on arXiv

Source-linked AI summary

A Jury, Not an Electorate: Label-Free Reliability Estimation for Ensembles of Language Models

Alessandro Rozza, Davide Mambelli, Steven Motta, Juri Taresch

arXiv:2610.21085v1cs.CLcs.AIcs.MA

TL;DR

Ensembles may contain useful answers beyond those of any one model, but existing methods often assign reputations using labels or before hearing the answers. The paper estimates expert reliability from unlabelled answer configurations, then finds comparable performance to label-fitted fusion and better results than single-model and label-free alternatives.

  • Problem

    Ensembles can contain answers missed by individual models, but many aggregation methods assign experts reputations learned from labels or inferred before hearing their answers.

    Source: §1 · p. 1

  • Method

    A one-coin Dawid–Skene model estimates expert reliabilities by EM from unlabelled answer configurations, then uses them as weights for linear fusion.

    Source: §1 · p. 2

  • Results

    76.9 ± 1.3%: the jury exceeds the strongest label-free alternative on the full benchmark by 5.5 points and performs comparably to supervised linear fusion.

    Source: §5 · p. 12

  • Takeaways & Limitations

    The method provides a label-free estimator of expert weights for linear fusion.

    Source: §5 · p. 12

    Most of the gain over the best single expert comes from GSM8K, and the results do not establish equivalence or generalisation beyond the evaluated setting.

    Source: Limitations · p. 12

Abstract

from arXiv · show

An ensemble of language models is a jury, not an electorate. The answers are testimony from witnesses of unknown reliability, and the question is not what the majority said but who is credible. Most ensemble methods assign each expert a fixed reputation, often learned from labels. We ask whether it can instead be inferred from the pattern of agreement and disagreement among the experts. We model the experts as a jury: each question has a latent true answer, each expert reports it with unknown reliability, and those reliabilities are estimated by Expectation Maximisation (EM) from unlabelled answer configurations alone. We show that its decision rule reduces to a log-odds weighted linear score over the experts' answer distributions. What changes is not how the jury votes, but how it learns whom to trust. On a 1560-question benchmark, the jury, with nine experts of up to 3B parameters, performs comparably to the same fusion rule with label-fitted weights, showing no statistically significant advantage from supervision. Instead, it outperforms the best single model with matched self-consistency and the strongest label-free alternative on the full benchmark.

1. Introduction

The paper treats model answers as testimony and estimates expert reliability from answer agreement rather than relying on reputations learned from labels. On the benchmark, the jury performs comparably to label-fitted fusion while outperforming the best single model and the strongest label-free alternative.

1. Introduction

  • The motivating challenge is using model diversity when experts disagree: an answer missed by one model may already appear elsewhere in the pool.Existing approaches often assign reputations learned from labels or inferred from the query before hearing the answers.Source: §1 · p. 1
  • 8.4 points: retaining answer distributions improves majority voting, and weighting experts adds a further 5.5 points.A matched control attributes the first gain to the answer distributions rather than additional test-time sampling.Source: §1 · pp. 1–2
  • The jury adapts a one-coin Dawid–Skene model, estimating each expert’s reliability by EM from unlabelled answer configurations without seeing gold answers.The contribution separates the effect of retaining answer distributions from expert weighting and additional sampling.Source: §1 · p. 2
  • For single answers, the decision rule is the log-odds weighted vote; for answer distributions, it is a signed linear fusion rather than a new family of ensemble methods.The paper also gives a moment-based identifiability result for K≥3 that does not require any expert to be better than chance.Source: §1 · p. 2
  • 76.9 ± 1.3%: the jury beats the best single expert by 2.0 points and the same expert with eight-sample self-consistency on GSM8K by 1.6 points, while performing comparably to supervised linear fusion.It also exceeds the strongest label-free alternative on the full benchmark.Source: §1 · p. 2

2. Related work

Related work spans methods that route queries before inference, combine outputs during generation, or aggregate completed answers afterward. The paper places its method in post-hoc aggregation and distinguishes its agreement-based reliability estimator from related label-free and supervised approaches.

2. Related work

  • Routing selects whom to ask before inference, while this method decides whom to believe after all experts have answered.Routing approaches use correctness labels, reward-model scores, human preferences, bandits, or mutual-information selection.Source: §2 · p. 3
  • During-inference fusion methods use logits or partial generations, whereas this method uses final answers; cascades instead address the accuracy-cost trade-off.Examples include vocabulary alignment, speculative decoding, and process-reward search [14] [23] [37] [52].Source: §2 · pp. 3–4
  • Post-hoc aggregation combines completed answers by weighting experts or scoring candidates; this method weights experts using answer distributions and reliabilities estimated from unlabelled configurations.The closest label-free counterparts include Smoothie [17], ISP [1], and Optimal Weight variants OW-L and OW-I [1].Source: §2 · p. 3
  • SkillAggregation [46] learns context-dependent judge skills, whereas this method fits a restricted generative model to produce log-odds weights for fixed linear fusion.SkillAggregation accepts binary and probabilistic estimates and extends Crowdlayer [41].Source: §2 · p. 3
  • The approach draws on Dawid and Skene’s joint recovery of latent truth and annotator reliability and the log-odds weighted vote for conditionally independent voters.With non-negative weights, the distributional rule is Stone’s linear pool; related alternatives include logarithmic pooling and DeGroot’s iterative consensus model.Source: §2 · pp. 3–4

3. Method

The method estimates expert reliability from unlabelled answer configurations, then uses those estimates in a linear fusion rule. It separates the information in answer distributions from the choice of expert weights.

3. Method

  • The method’s core is a generative model for answer configurations whose EM updates estimate reliabilities without gold labels.The estimates are fit on unlabelled training configurations and frozen before evaluation.Source: §3 · p. 4, §3.2 · p. 6

3.1. What a weighted vote over argmax answers can and cannot do

  • A weighted vote over argmax answers can change which observed answer wins, but cannot select an answer absent from the experts’ proposals.Retaining answer distributions can enlarge the reachable set, while weighting experts better is a separate source of improvement.Source: §3.1 · p. 4

3.2. A working generative model of the answer configuration

  • The model treats each question’s observed candidate set as fixed support and assumes experts are conditionally independent given a latent true answer.Its one-reliability-per-expert Dawid–Skene variant uses a uniform error distribution over the other candidates; answers outside the observed support receive zero probability.Source: §3.2 · p. 5
  • EM estimates each expert’s reliability from unlabelled answer configurations, averaging posterior-weighted mass placed on candidates across questions.The model is identifiable under the stated support and observation assumptions when at least three experts have reliability different from 1/K, though EM may reach a local maximum.Source: §3.2 · pp. 5–6

3.3. What the rule actually is

  • With single answers, the jury reduces to the log-odds weighted vote; with distributions, it is signed linear fusion, where a non-top-choice candidate can win.The learned reliability vector sets the expert-specific weights, making the same fusion rule with label-fitted weights a natural supervised baseline.Source: §3.3 · pp. 6–7

3.4. Modelling choices

  • For sampled distributions, α=n recovers the multinomial likelihood; the default is α=8 for eight samples and α=1 for a greedy answer.Changing α leaves the fixed-reliability argmax unchanged but can alter the E-step and fitted reliabilities; an extension also tests shared-lure errors.Source: §3.4 · pp. 7–8

4. Results

Across the benchmark, the jury’s distribution-based fusion exceeds equal weighting and label-free alternatives, while its advantage over label-fitted weights is not statistically significant after refitting variability is included. It also improves on the strongest single expert and remains robust across sub-pools, but its posterior confidence requires recalibration.

4.1. Setup

  • The benchmark comprises 1,560 questions and nine instruction-tuned experts of up to 3B parameters, evaluated over ten stratified train/test splits.The jury is fitted on unlabelled configurations from the training half; comparisons use a paired bootstrap over questions.Source: §4.1 · pp. 8–9

4.2. The comparison

  • +8.4 points: retaining answer distributions raises equal-weight voting from 63.0 ± 1.6% to 71.4 ± 1.8%; log-odds weighting adds +5.5 points to reach 76.9 ± 1.3%.A sample-matched control and comparisons across weighting schemes support the distinction between distributional information and weighting.Source: §4.2 · p. 9, Table 2 · p. 11
  • After refitting variability is included, the jury’s +0.6-point difference from supervised linear fusion is not statistically significant (95% interval [−0.6, +2.0]).The jury and supervised baseline use the same fusion form but estimate reliabilities differently.Source: §4.2 · pp. 9–10, §4.1 · p. 9
  • 76.9% accuracy: the jury is +5.5 points over the equal-weight pool and +7.9 over ISP on the full benchmark.With greedy answers, the jury scores 72.3% versus 69.1% for ISP.Source: §4.2 · p. 10
  • The jury reaches 76.9%, +2.0 points over the best single expert, and remains +1.6 points ahead when that expert uses matched eight-sample self-consistency on GSM8K.The gain is concentrated on GSM8K; on the sources where qwen3b dominates, the jury essentially matches it.Source: §4.2 · p. 11
  • Across 246 sub-pools, the jury beats the equal-weight pool on 79%, 98%, and 100% of three-, five-, and seven-expert pools, respectively.Nested resampling leaves its advantages over equal-weight pooling and majority voting intact.Source: §4.2 · p. 11

4.3. Model checks and calibration

  • Mean top-1 posterior confidence is 0.991 against 76.9% accuracy, and the authors say recalibration is required before using it as a confidence score.Expected calibration error is 0.223 over ten equal-width bins.Source: §4.3 · pp. 11–12

5. Conclusion

The jury learns expert weights from unlabelled testimony and performs comparably to supervised linear fusion. It outperforms the best single expert and the strongest label-free alternative in the reported benchmark.

5. Conclusion

  • Keeping answer distributions improves majority voting by 8.4 points, and weighting experts adds a further 5.5 points.Source: §5 · p. 12
  • A generative model fitted by EM on unlabelled answers learns expert weights without reading a gold answer.The paper describes this model as simple and marks its inspiration with.Source: §5 · p. 12
  • At 76.9 ± 1.3%, the jury beats the best greedy expert by 2.0 points, matched eight-sample GSM8K by 1.6, and the strongest label-free alternative by 5.5.The benchmark covers 1,560 questions.Source: §5 · p. 12
  • The jury performs comparably to supervised linear fusion; its small edge is significant conditional on fitted models, but not when refitting variability is included.Source: §5 · p. 12

6. AI usage

The authors used generative AI tools to assist with proof checking, evaluation-code implementation, manuscript editing, and literature identification.

6. AI usage

  • Generative AI tools checked algebraic steps after the authors had developed the propositions and proof strategies.Source: §6 · p. 12
  • The tools also supported implementation of the evaluation code.Source: §6 · p. 12
  • The tools assisted with readability edits and identifying relevant literature.Source: §6 · p. 12

Limitations

The evaluation is limited to a fixed pool of nine models and 1,560 mostly multiple-choice questions. Its gains and cost comparisons also reflect task-specific sampling and the complete system, not aggregation alone.

Limitations

  • The evaluation uses a fixed pool of nine models and 1,560 mostly multiple-choice questions, so its findings do not establish generalisation beyond that setting.Source: Limitations · p. 12
  • Testing every subset of three, five, and seven experts covers combinations within the pool, but not combinations of different underlying models.Source: Limitations · p. 12
  • Most of the gain over the best single expert comes from GSM8K.Source: Limitations · p. 12
  • The jury requires every expert and repeated sampling for generative tasks, increasing inference cost; its best-greedy-expert comparison includes that sampling cost.Source: Limitations · p. 12

A. Reproducibility and experimental details

The appendix details the benchmark and protocols, compares label-fitted and jury reliability estimates, and examines where answers are reachable and where the jury’s gains arise. The source breakdown shows its largest margin on GSM8K, with little change on ARC-Challenge and HellaSwag.

A. Reproducibility and experimental details

  • The supplement provides cached answers and a Makefile that rebuilds tables and figures in about ten minutes without rerunning models.Table 3 reports individual experts over all 1,560 questions, unlike Table 5’s held-out-half averages over ten splits; its best-expert-per-source baseline is not query-conditioned routing.Source: A. Reproducibility and experimental… · p. 19, Table 3 · p. 20
  • The nine-expert pool spans model sizes, domains, and families; the benchmark uses fixed-seed, binary-scored questions and drops multiple-choice items with three or five options.GSM8K uses eight real generations, whereas multiple-choice sampling quantises the distribution from one forward pass; candidate sets average 17.84 answers after sampling on GSM8K and equal four on all but four multiple-choice questions.Source: A. Reproducibility and experimental… · pp. 19–20

A.1. The supervised linear fusion

  • 72.3 ± 1.1% for the greedy jury versus 73.3 ± 1.1% for Optimal Weight refitted to the jury’s convention; the −0.8-point difference is not significant.The matched comparison differs only in how reliability is estimated; with task-defined K=4, Optimal Weight scores 73.0 ± 1.4%.Source: §A.1 · p. 21

A.2. Reachability and question strata

  • 94.2% of questions are reachable by some weighted greedy-answer vote; signed distributional fusion reaches 99.7%, while non-negative fusion reaches 95.8%.The distributional gain mainly changes how already-reachable candidates are ranked, rather than making new answers reachable.Source: §A.2 · p. 22
  • No expert proposes the gold answer on 5.8% of questions with greedy answers; sampling reduces the unreachable share to 0.3%.These are errors for every combiner restricted to the corresponding candidate set.Source: §A.2 · p. 22
  • On minority questions, the jury recovers 55.5%, versus 95.7% on plurality questions, and remains 17.4 points below the greedy-answer oracle overall.Minority questions have an average of 2.4 experts supporting the gold answer, compared with 4.3 for a plurality answer.Source: §A.2 · p. 22

B.1. Accuracy by source

  • +20.0 points on GSM8K and +2.2 on MMLU; the jury is nearly even with the best expert on ARC-Challenge (−0.1) and HellaSwag (+0.2).GSM8K contributes +1.54 of the +1.92-point overall margin; Table 5 reports accuracy by source.Source: §B.1 · p. 22, Table 5 · p. 23
  • After eight-sample self-consistency lifts qwen3b by +4.9 points on GSM8K, the jury remains +15.1 ahead there and +1.6 overall.The margin is concentrated where qwen3b is weakest and olmo2-1b is strongest; on the two sets where qwen3b leads, combining does not improve on it but incurs essentially no accuracy loss.Source: §B.1 · pp. 22–23

B.2. Refitting and resampling

Refitting uncertainty removes a significant supervised advantage, while the jury remains ahead of the supervised fusion when reliabilities vary by task. Comparisons with label-free alternatives show that answer distributions and evaluation protocol shape the results.

B.2. Refitting and resampling

  • After refitting variability is included, the jury’s +0.6 over supervised fusion has 95% interval [−0.6, +2.0], while its margins over equal-weight and majority voting survive.The nested bootstrap resamples questions, splits them, and refits every method; the supervised-fusion interval includes zero, while differences are +5.5 ([+2.9, +8.3]) over equal-weight and +14.1 ([+10.5, +17.8]) over majority voting.Source: Table 6 · p. 24, §B.2 · p. 24
  • The nested bootstrap widens intervals against fitted rules by about 140% for supervised fusion and about 180% for the answer scorer, reflecting refitting variability.The paired bootstrap holds the ten fitted models fixed; the nested procedure also resamples questions, splits, and refits methods.Source: §B.2 · pp. 23–24, Table 6 · p. 24

B.3. Per-task reliabilities

  • Allowing one reliability per expert and task yields 77.3±1.3% for the jury versus 76.4±1.3% for supervised fusion, a +0.9 difference ([+0.3, +1.5]).The authors leave open whether supervision wins with richer parameterisations, such as per-subject or query-conditioned reliabilities.Source: §B.3 · p. 25, Table 7 · p. 26

B.4. SkillAggregation, OW-L and OW-I on the multiple-choice questions

  • On multiple-choice questions, the jury with answer distributions reaches 76.6%, 5.3 and 5.2 points above OW-L and OW-I, which use greedy answers only.OW-L and OW-I reach 71.3% and 71.6% on greedy answers, respectively; the comparisons with the jury have intervals [+3.8, +6.9] and [+3.6, +6.7].Source: Table 8 · p. 26, §B.4 · p. 27
  • SkillAggregation reaches 73.2% without labels on answer distributions, while its test-selected upper references remain below the jury with repeated sampling.Its base and context-dependent test-oracle results are 73.5% and 74.0%; both are explicitly test-selected, and the 40-epoch protocol was tuned after inspecting gold-answer accuracy.Source: §B.4 · pp. 25, 27, Table 8 · p. 26

B.5. Replication across sub-pools

Across sub-pools, the jury generally beats equal weighting and matches or exceeds the best expert, while staying close to supervised fusion. Fractional evidence weighting can produce degenerate fits in small pools; the tested multinomial-weight fits were more stable, though not guaranteed to avoid that mode.

B.5. Replication across sub-pools

  • With α=8, the jury beats equal weighting on 79%, 98%, and 100% of the 3-, 5-, and 7-expert pools, by +2.1, +4.9, and +4.7 points on average.It is at or above the best expert on 82%, 96%, and 100% of those pools, by −0.7, +1.3, and +1.8 points on average; against supervised fusion, mean differences are −0.9, +0.2, and +0.8.Source: §B.5 · p. 27, Table 9 · p. 28
  • With fractional α=1, the jury replicates from five experts upward, beating equal-weight pools on 94% and 100% of those pools, but collapses on 58 of 84 three-expert pools to 15.8% average accuracy, far below chance.Source: §B.5 · p. 28
  • The α=1 collapse persists across tested initialisations, but the checks do not establish local or global optimality.The proposed mechanism is a fractional-likelihood shared-miss explanation; α=8 concentrates E-step posteriors and had no degenerate fits from the default start on tested pools, but random starts reached a degenerate mode in 3 of 200 runs.Source: §B.5 · pp. 28–29
  • On the full pool, α=8 and α=1 perform similarly at 76.9% and 76.7%, but α=8 avoids the observed three-expert failures from the default start.The multinomial weight is at least as good as α=1 on every five- and seven-expert pool; its measurable cost is a more concentrated posterior.Source: §B.5 · p. 29

B.6. Sensitivity to the EM start

  • For sampled distributions with α=8, 247 of 250 initialisations match the default solution; the other three reach a lower-likelihood degenerate solution.In the greedy α=1 regime, every split has a higher-likelihood degenerate fixed point than the default fit, so the reported default fit is not a global likelihood maximiser.Source: §B.6 · pp. 29–30, Table 10 · p. 30

C. Ablations

The ablations examine evidence weighting, correlated errors, selective overrides, and calibration. Accuracy is insensitive to α≥1 on the full pool, while the jury posterior is highly concentrated and requires recalibration before being interpreted as probability.

Evidence weight

  • On the full pool, sweeping α over {0.5, 1, 2, 4, 8} gives 75.9%, 76.7%, 76.8%, 76.9%, and 76.9%; α≥1 is within noise of α=8.On three-expert sub-pools, α=1 produced degenerate fits on most pools, unlike α=8 from the default initialisation on tested pools.Source: C. Ablations · p. 30, Figure 4 · p. 31

Correlated errors

  • With repeated sampling, modelling shared errors costs accuracy: −0.6 points for lineage grouping and −0.4 pool-wide, while lineage does not predict correlated error.The estimated excess shared-wrong-answer rates are +10.3 points for same-lineage pairs and +11.6 for different-lineage pairs.Source: C. Ablations · pp. 30–31

Selective override

  • Accuracy is highest at or near τ=0 for both jury and scorer, so requiring the verdict to beat plurality before overriding it costs more than it returns.Under α=8, thresholds up to 0.5 change the jury’s concentrated posterior almost nothing.Source: C. Ablations · p. 31

Calibration

  • The α=8 posterior has 0.991 mean top-1 confidence against 76.9% accuracy, with expected calibration error 0.223 and Brier score 0.448.The posterior ranks candidates well but needs recalibration before being read as probability; the calibration gap is smaller at α=1.Source: C. Ablations · p. 31

D. The support control

The support control compares the observed candidate set with the known four-option space on multiple-choice questions, re-estimating parameters under both conventions. Results change little on this benchmark, but the control does not test other model assumptions.

D. The support control

  • With repeated sampling, fixing the support to four options leaves fitted reliabilities equal to three decimals and every verdict identical for both jury and supervised fusion.The control compares observed support with all four task options and re-estimates parameters under each convention.Source: D. The support control · pp. 31–33, Table 13 · p. 33
  • On greedy answers, four-option support shifts jury accuracy by −0.4 points ([−1.2, +0.4]) and Optimal Weight by −0.2 ([−1.2, +0.9]), neither significant.The pool proposes 2.98 options on average; unanimous questions count as agreement under fixed support.Source: D. The support control · pp. 32–33, Table 13 · p. 33
  • The control indicates that candidate-set construction is not what the benchmark results rest on, but it does not test other assumptions, particularly conditional independence.GSM8K retains observed support because its answer space is unbounded.Source: D. The support control · pp. 31, 33

E. Proof of Proposition 1

The proof establishes that the reliability vector is identifiable from the observable distribution of unlabelled expert outputs, under the stated conditions on support size and expert reliabilities.

E. Proof of Proposition 1

  • Each expert’s empirical answer frequencies are unbiased for its emission probabilities under the sampling model.The proof then centres these frequencies and rescales reliabilities so chance-level reliability corresponds to zero.Source: E. Proof of Proposition 1 · pp. 33–34
  • The second- and third-order centred agreement moments are determined by observed empirical answer distributions, without using the latent true answer.The proof computes these moments by conditioning on the latent answer and questions with fixed support size, then averaging.Source: E. Proof of Proposition 1 · pp. 34–35
  • For support size K≥3 and at least three experts with nonzero centred reliability, a usable pair exists for every target expert.The nonzero centred reliabilities ensure the selected pair's denominator is nonzero.Source: E. Proof of Proposition 1 · p. 35
  • Equal observable output distributions imply equal reliability vectors, proving identifiability for every expert.The proof applies the same second- and third-order moments to each expert using a usable pair with nonzero second-order agreement.Source: E. Proof of Proposition 1 · p. 35

Related papers

Loading 2610.21085v1…