Source-linked AI summary
Candidate supply and answer selection shape the value of LLM judging in multi-agent systems
Jia-Hao Ji, Sijie Li, Jiabei Cheng, Zixi She, Jin-Tai Yu, Zhiyuan Yuan
TL;DR
Multi-agent systems can generate correct answers yet lose them during communication and final selection. This paper separates candidate generation, judge recognition, and terminal selection, finding that judge-guided selection improves accuracy mainly by rescuing correct answers outnumbered by popular errors.
Problem
The paper addresses why multi-agent systems sometimes fail to retain generated correct answers despite substantial communication overhead.
Method
The study analyzes judge ranking reliability across tasks and generators, then replays fixed candidate pools to isolate terminal answer selection.
Results
Judge-guided selection improves final accuracy primarily by rescuing correct answers that peer interaction would otherwise lose to popular errors.
Takeaways & Limitations
Judge reliability depends on the evaluated candidates, so generating more candidates helps only when selection can recognize and retain correct answers.
Takeaways & Limitations
The offline pool analyses evaluate terminal selection over fixed candidate traces, isolating it from dynamic feedback loops.
Abstract
from arXiv · showhide
Multi-agent systems (MAS) sometimes already have the potential to answer correctly, but still report a wrong answer. Explaining this outcome is difficult because generation, communication and final answer-selection rules usually change simultaneously. We conceptualize multi-agent reasoning as an evolutionary pipeline of candidate generation, peer communication and terminal selection, wherein consensus without quality control can exhibit patterns of memetic drift. We study two questions: (1) when an LLM judge provides effective selection pressure by supplying a signal of answer correctness for candidates generated in a multi-agent system, and (2) when using that signal improves the reported answer. To map judge reliability, we analysed 15,336 questions from MMLU-Pro, GPQA, MedXpertQA and MuSR, with Humanity's Last Exam analysed separately. To test these rules, we replayed 81,390 fixed candidate pools drawn from 16,278 questions across five benchmarks. We report three findings. (1) A correct answer is often already present among the generated candidates, but the system can still converge on and report a wrong answer. (2) Judge reliability is not a fixed trait of the model, but varies with the task, the generator and how rare the correct answer is. (3) Combining answer frequency with the judge's evaluation changed only the final answer-selection rule and raised accuracy from 63.82% to 70.82-70.95%, primarily by rescuing correct answers that were outnumbered by popular errors. In the systems studied here, the value of generating more candidates depends on whether those extra samples make correct answers present, frequent or recognisable. By isolating generation, recognition and selection, these findings establish a diagnostic basis for designing multi-agent architectures that protect generated correct answers from being lost.
Results
The results reveal that multi-agent systems often lose correct candidates during consensus, while judge reliability varies by candidate availability, task, and generator. Combining answer frequency with judge rankings improves final selection substantially on fixed candidate pools.
- Communication protocols: The open committee achieved 42.20% accuracy at 10.73× token cost, only 1.14 percentage points above majority voting.The evidence board achieved 41.89% at 6.07× cost, with a 95% CI of −0.05–1.74 that included zero.
- Candidate retention: 13.13–14.38 percentage points separated oracle performance from actual system accuracy because correct candidates were often not retained.When an outnumbered correct candidate coexisted with an incorrect majority, the open committee selected the popular error in 73.25% of cases and the evidence board in 59.61%.
- Judge reliability: Rank AUC fell below 0.5 when correct candidates were sparse, then rose with pgen, reaching an empirical half-rise midpoint of pgen = 14.7%.Reliability also varied with task and generator: Mimo no-think showed R2 = 0.022, whereas medium-thinking showed R2 = 0.928 with a 19.0% half-rise midpoint.
- Terminal selection: 63.82% accuracy came from majority voting on identical candidate pools, while judge-guided selection reached 70.82–70.95%.The improvement was achieved offline without generating new candidate traces.
- Terminal selection: 70.82–70.95% accuracy at t = 2–4 exceeded majority voting by 7.00–7.13 percentage points.First-ranked selection reached 69.04% (+5.22 pp; 95% CI, 4.85–5.59).
Discussion
The discussion frames multi-agent accuracy as three separable bottlenecks—candidate generation, judge recognition and terminal selection—and shows how memetic drift can discard correct answers. It further argues that judge reliability and the value of additional computation depend on candidate context, motivating diagnostic evaluation and dynamic-system analysis.
- Separable bottlenecks: Accuracy reflects whether generation produces a correct candidate, whether a judge recognises it, and whether terminal selection acts on that signal.Without selection pressure, peer interaction can produce memetic drift, allowing systems to generate correct answers but report wrong ones.
- Communication and computation: Discussions often preserved popular errors over outnumbered correct answers, while additional rounds increased token use without consistent accuracy gains.The resulting generation–retention bottleneck combines memetic drift with majority-biased cascades.
- Communication and computation: Extra candidate generation improves reported answers only when selection identifies the correct candidate or generation makes it the overwhelming majority.Judge-guided selection improved accuracy in small fixed pools when correct answers were present but outnumbered.
- Context-dependent judging: Judge reliability varies with the candidate generator, task and empirical correct fraction rather than constituting a static model trait.Matched questions can yield different availability–reliability relationships because generator settings produce different candidate traces.
- Context-dependent judging: Evaluation quality should be measured within the generator regime, task context and candidate-availability range in which the evaluator will operate.Static benchmark scores should not be treated as universal guarantees of evaluator quality.
- Diagnostic design: If candidate supply is zero, no downstream selection or communication rule can recover the answer; when correct candidates are minority-held, evaluate the exact generator before comparing selection rules.The proposed progression compares majority vote, judge selection and hybrid rules on held-out data according to accuracy and cost.
- Limitations and future work: Fixed-pool analyses isolate terminal selection, whereas live interactions can alter candidate distributions through receiver-mediated selection and transformation.Future work should track information evolution across dynamic multi-turn graphs, open-ended generation and real-time execution environments.
- Limitations and future work: Reliable scaling requires tracking generation, receiver-mediated local selection, transformation and terminal selection to protect correct answers from being lost.The discussion argues that scaling agent systems effectively requires more than simply adding model instances.
Methods
The study used controlled multi-agent protocols, candidate banks, and topology sweeps to separate answer generation, communication, and selection. Analyses preserved matched question-seed comparisons while defining candidate availability and uncertainty procedures explicitly.
- Protocol comparisons: Four protocols were evaluated on 2,450 MedXpertQA medical-reasoning questions across three runs, producing 7,350 evaluations per protocol.Seeds 42, 43, and 44 were used across runs.
- Protocol comparisons: Single-answer generation used one response, majority voting used five independent responses, and open committee agents revised answers after reviewing peer responses, confidence scores, and evidence.The evidence board instead pooled evidence from five independent Round 1 responses for an anonymous final-judge input.
- Evaluation and uncertainty: The oracle upper bound counted questions with at least one initial correct response, measuring recoverable answer content rather than an implementable selection rule.Protocol comparisons were paired by question and seed, with uncertainty estimated from 10,000 bootstrap samples and 95% confidence intervals.
- Communication topology: A topology sweep tested star, complete, ring, and hierarchical five-agent networks under early-stopping and forced-six-round conditions, with 3,150 trials per cell.Token consumption was recorded across API requests and normalized to the single-agent baseline.
- Candidate-bank construction: Parse failures were excluded from candidate-bank denominators and ranking pools, while quota-deficient banks were retained for availability summaries but excluded from analyses requiring missing candidates.The resulting pgen measure reflects collection under the adaptive protocol, not an unconstrained single-call success probability.
LLM judge ranking and reliability fitting
Judge reliability was measured by how well a temperature-zero ranking judge ordered correct candidates above wrong ones, then fitted as a sigmoid relating rank AUC to correct-answer availability. The analysis used sampled candidate pools across four primary benchmarks, with Humanity’s Last Exam evaluated separately.
- Pool construction: Two sampling designs were evaluated: variable-composition pools and fixed-composition pools containing exactly 2 correct and 6 wrong traces.The fixed design held displayed composition constant while characterising the association between full-bank availability (pgen) and rank AUC.
- Reliability measurement: Rank AUC measured the probability that a randomly selected correct candidate ranked above a wrong candidate within a pool.Candidate pools contained k = 8 candidates, and rankings were usable only when every anonymous candidate was included.
- Reliability fitting: A three-parameter sigmoid was fitted to pair-weighted rank AUC across 0.02-wide bins of full-bank correct-answer availability (pgen).The fit used nonlinear weighted least squares, with weights proportional to correct–wrong pair counts and a lower asymptote fixed at 0.5 by constraining A to be non-negative.
- Primary analysis: 15,336 questions across MMLU-Pro, GPQA, MedXpertQA and MuSR contributed at least one ranked pool containing both correct and incorrect candidates for the primary rank-AUC analysis.The candidate banks covered 15,784 questions in total.
- Humanity’s Last Exam: HLE was analysed separately, with 475 multiple-choice questions contributing to the fit and 494 to replay.For HLE QA, the equivalence judge’s labels agreed with human assessment in 120 of 124 non-ambiguous reviewed answers (96.8%) and correctly labelled 195 of 200 constructed controls (97.5%).
Offline replay over fixed candidate pools
Offline replay evaluated answer-selection rules on fixed eight-candidate pools while holding candidate content, rationales, frequencies and judge orderings constant, except for a shuffled-order control. A sensitivity analysis varied rank-weighting strength to compare majority voting with frequency-plus-ranking selection.
- Replay design: 81,390 fixed k = 8 candidate pools covered 16,278 benchmark questions across MMLU-Pro, GPQA, MedXpertQA, MuSR and HLE MC.The replay used five pools per question, with candidate text, rationales and answer frequencies fixed.
- Replay design: Stored judge orderings remained fixed, while a prespecified control randomly permuted the same eight candidate identifiers before rank-based selection.Single-candidate sampling, majority-vote tie resolution and shuffled-order controls used deterministic seeds 0–99.
- Replay design: Question-level comparisons classified whether first-ranked selection, majority voting or both tied after averaging accuracy across five stored pools.The comparison also separated questions where correct candidates were less common or more common in the full candidate bank.
- Rank-weight sensitivity: The judge-weighting sensitivity analysis found a flat t = 2–4 plateau in the complete t = 0–5 sequence.Rank i received weight w_i = (k - i)^t; t = 0 reproduces majority voting, while t = 1–4 combines frequency with rank ordering.
- Rank-weight sensitivity: At k = 8 and t = 5, the first-ranked candidate determines the answer because its weight exceeds the combined weights of all seven lower-ranked traces.Tied answer scores were resolved with deterministic seeds 0–99 and averaged across 100 results.
Targeted reranking and information perturbation
The study targeted reranking in 713 matched eight-trace pools where correct candidates were outnumbered, testing how visible answer and rationale information affected selection. It sampled three generator–judge cells and compared five information conditions using paired pool-level evaluation.
- Targeted reranking: 713 matched eight-trace pools contained correct candidates outnumbered by incorrect candidates.The realised correct-trace fraction in each pool was below 0.5.
- Targeted reranking: Three generator–judge cells supplied the pools: DeepSeek→DeepSeek on HLE MC, DeepSeek→DeepSeek on GPQA, and Mimo no-think→DeepSeek on GPQA.Each cell contributed three non-overlapping operational samples capped at 80 questions.
- Information perturbation: Five visible-information conditions varied whether reranking received answers, rationales, both, or reassigned answers or rationales.Reassignments used deterministic within-pool permutations that moved content across candidate labels.
- Information perturbation: 6,488 correct–wrong comparisons per condition were evaluated with pair-weighted rank AUC, and paired differences used 2,000 pool-level bootstrap resamples.The bootstrap procedure used seed 1729.
Pool size and reconstructed token-based cost
The analysis compared selection strategies across candidate-pool sizes using matched candidate traces and reconstructed token-based costs from stored character counts. It also quantified tie-handling sensitivity and documented bootstrap uncertainty and parse-validity exclusions.
- Pool size: The primary pool-size analysis evaluated k ∈ {2, 4, 6, 8, 10, 12, 14, 16}.At each k, questions were retained only when every compared strategy produced parse-valid outputs.
- Pool size: At each k, majority voting and first-ranked selection used exactly the same k candidate traces.The ranking judge ordered the pool, and first-ranked selection returned the answer attached to the first-ranked trace.
- Pool size: 0.36 percentage points was the maximum accuracy change across k under tie-neutral sensitivity.Tied plurality winners were selected uniformly, while the sensitivity analysis averaged correctness across tied winners.
- Reconstructed token-based cost: Estimated tokens equalled ceil(character count/4) + 12 for prompt messages, while structured completion tokens used ceil(serialized JSON character count/4).Generation prompts, generation completions, ranking prompts and ranking completions were reconstructed separately.
- Reconstructed token-based cost: Incremental judging cost per 1,000 additional correctly answered questions = 1,000 × (Crank - Cmajority) / (Nrank - Nmajority).The quantity was defined only when Nrank > Nmajority, with C and N denoting reconstructed USD cost and correctly answered questions.
- Reconstructed token-based cost: 99.63% of tasks produced parse-valid outputs in the cost-analysis audit across 48 primary model-and-pool-size combinations.The audit counted 308,456 parse-valid outputs out of 309,612 tasks.
Software, provenance and reproducibility
The released implementation targets Python 3.10 or later and organizes analysis into versioned modules controlled by configuration manifests. Reproduction materials are versioned on GitHub and archived in an access-controlled Hugging Face dataset repository; no human participant data were collected.
- Implementation: The analysis code targets Python 3.10 or later, with the manuscript build verified under Python 3.12 using NumPy, pandas, SciPy, Matplotlib and statsmodels.The released implementation separates candidate-bank construction, judge ranking, curve fitting, fixed-pool replay, candidate-information perturbation and pool-size cost analysis into versioned modules.
- Provenance: Analysis scripts and reproduction manifests are versioned at GitHub in release commit 7840cf3dbde77e61a1b4349ad3239669a7502431.Repository: https://github.com/YSTLab/mas-reasoning
- Reproducibility: The complete access-controlled research archive, containing raw model outputs, candidate banks, ranked pools and analysis summaries, is deposited in the Hugging Face dataset repository 60a16ff6026f96d6b9da78816d60eeeb14d43fdf.No human participant data were collected.
Definitions and notation
The framework distinguishes live communication processes that alter candidate information from fixed-pool replay, which isolates terminal selection using stored traces and fixed judge rankings.
- Agents may repeat, ignore, reject, or adopt peer information during communication, or transform it by revising and combining statements into a new rationale.
- Memetic drift describes stochastic variation in which statements persist, while majority-biased cascades arise from repeatedly adopting already popular statements.
- Together, these processes can create a generation–retention bottleneck in the information evolution framework.
- Fixed-pool replay evaluates judge ranking and terminal selection over stored candidate traces while holding traces and rankings fixed.This isolates terminal selection from live communication processes that repeat, reject, adopt, revise, or combine information.
Models and inference conditions
The study tested seven generator–judge combinations under controlled sampling temperatures and reasoning settings. Benchmark splits and correctness labels were specified for the evaluation and reranking analyses.
- Models and inference conditions: Seven generator–judge combinations were tested, with DeepSeek-V4-Flash (Preview) and Mimo v2.5 among the evaluated models.DeepSeek-V4-Flash (Preview) was accessed through the API in June and July 2026; Mimo v2.5 was evaluated with optional internal reasoning.
- Models and inference conditions: Benchmark analyses used stored test splits, official correctness keys and choice parsers, with HLE QA additionally using a separate DeepSeek reference-answer equivalence judge.MMLU-Pro, GPQA, MedXpertQA, MuSR and HLE benchmark roles differed between primary modelling, external comparison and targeted reranking analyses.
- Models and inference conditions: The fixed research archive included raw model outputs, candidate banks, ranked pools, analysis-ready summaries, manifests and source data for the reported figures.Reviewer access was provided through an access-controlled Hugging Face dataset repository, with a redistribution-compatible release planned upon publication.
Dataset Pool construction
The dataset-pool construction section presents generator–judge evaluation results organized by AUC, fit measures, quality, midpoint, pairs, pools, bins, and parse fields. Reported rows classify boundary fits as strong or usable across numerical result pairs.
- Dataset Pool construction: 0.728 0.700 usable boundary fit is a reported result pair.Other usable-fit rows include 0.656 0.686, 0.707 0.592, and 0.775 0.679.
- Dataset Pool construction: 0.815 0.855 strong boundary fit is one reported result pair.The passage labels this pair as a strong boundary fit.
- Dataset Pool construction: The evaluation table is organized by Generator, Judge, AUC, Fit R2, Fit quality, Midpoint, Pairs, Pools, Bins, and Parse.The supplied passage identifies the table’s column structure but does not provide the underlying construction procedure.
- Dataset Pool construction: 0.745 0.882 strong boundary fit is another reported result pair.Additional strong-fit rows include 0.710 0.830, 0.670 0.847, and 0.785 0.855.