PaperlayerView exact revision on arXiv

Source-linked AI summary

Candidate supply and answer selection shape the value of LLM judging in multi-agent systems

Jia-Hao Ji, Sijie Li, Jiabei Cheng, Zixi She, Jin-Tai Yu, Zhiyuan Yuan

arXiv:2608.25937v1cs.AIcs.MA

TL;DR

Multi-agent systems can generate correct answers yet lose them in the reported output, while candidate generation, communication, and answer selection are difficult to disentangle. This study isolates these stages and finds that judge-guided selection can rescue correct answers outnumbered by errors in fixed candidate pools.

  • Problem

    Multi-agent systems can incur substantial communication overhead yet fail to reliably retain generated correct answers in reported outputs.

    Source: Multi-agent review does not reliably… · p. 3

  • Method

    The paper frames multi-agent reasoning as an evolutionary pipeline separating candidate generation, judge recognition, peer communication, and terminal answer selection.

    Source: Discussion · p. 14

  • Results

    Groups can generate a correct answer yet fail to report it, while judge-guided selection improves small pools when correct answers are present but outnumbered.

    Source: Discussion · p. 14

  • Takeaways & Limitations

    In the studied systems, extra candidate generation improves reported answers only when selection identifies the correct candidate or generation makes it the overwhelming majority.

    Source: Discussion · p. 14

    Offline analyses use fixed candidate traces and isolate terminal selection from the dynamic feedback loops of live interaction.

    Source: Discussion · p. 15

Abstract

from arXiv · show

Multi-agent systems (MAS) sometimes already have the potential to answer correctly, but still report a wrong answer. Explaining this outcome is difficult because generation, communication and final answer-selection rules usually change simultaneously. We conceptualize multi-agent reasoning as an evolutionary pipeline of candidate generation, peer communication and terminal selection, wherein consensus without quality control can exhibit patterns of memetic drift. We study two questions: (1) when an LLM judge provides effective selection pressure by supplying a signal of answer correctness for candidates generated in a multi-agent system, and (2) when using that signal improves the reported answer. To map judge reliability, we analysed 15,336 questions from MMLU-Pro, GPQA, MedXpertQA and MuSR, with Humanity's Last Exam analysed separately. To test these rules, we replayed 81,390 fixed candidate pools drawn from 16,278 questions across five benchmarks. We report three findings. (1) A correct answer is often already present among the generated candidates, but the system can still converge on and report a wrong answer. (2) Judge reliability is not a fixed trait of the model, but varies with the task, the generator and how rare the correct answer is. (3) Combining answer frequency with the judge's evaluation changed only the final answer-selection rule and raised accuracy from 63.82% to 70.82-70.95%, primarily by rescuing correct answers that were outnumbered by popular errors. In the systems studied here, the value of generating more candidates depends on whether those extra samples make correct answers present, frequent or recognisable. By isolating generation, recognition and selection, these findings establish a diagnostic basis for designing multi-agent architectures that protect generated correct answers from being lost.

Results

The results examine how multi-agent communication and terminal answer selection affect whether generated correct answers are retained and reported. They assess judge reliability and judge-guided selection across tasks, generators, rationale–answer correspondence, and candidate-pool compositions and sizes.

Multi-agent review does not reliably retain generated correct answers

  • Communication protocols produced small accuracy gains at higher token costs, while longer discussion increased token use without consistent accuracy gains.The open committee reached 42.20% accuracy at 10.73× token cost versus majority voting’s 41.06% at 4.96×; forcing six rounds used 33×–49× tokens, and 90.48–95.37% of standard runs ended after one round.Source: Multi-agent review does not reliably… · p. 4
  • 13.13–14.38 percentage points separated the oracle ceiling from actual accuracy, as systems sometimes reported popular errors despite having generated correct candidates.When an outnumbered correct candidate was present, the open committee chose the most frequent incorrect answer in 73.25% of cases and the evidence board did so in 59.61%.Source: Multi-agent review does not reliably… · p. 4, Figure 1 · p. 5

LLM judge reliability varies with correct-answer availability, task and generator

  • On HLE, the fitted half-rise midpoint was 28.6% for multiple-choice questions scored with official keys and 36.9% in the open-answer QA audit, indicating an upward shift in availability associated with reliable ranking.Mimo no-think candidates showed an almost flat availability relationship (R2 = 0.022), whereas medium-thinking candidates showed a steep sigmoid transition (R2 = 0.928; half-rise midpoint, 19.0%).Source: LLM judge reliability varies with… · pp. 6–7
  • Rank AUC rose with correct-answer availability, with a 14.7% half-rise midpoint in the primary evaluation and 12.2% under fixed 2C/6W composition.At low availability, observed rank AUC fell below chance; the fixed-composition result indicates that availability also reflects question-level candidate separability, not just the displayed ratio.Source: LLM judge reliability varies with… · p. 6, Figure 2 · p. 8

Combining answer frequency and judge rankings recovers outnumbered correct candidates

  • 70.82–70.95% accuracy from rank-power weighting (t = 2–4) exceeded majority voting’s 63.82% on identical candidate pools.The plateau gained 7.00–7.13 percentage points; first-ranked selection reached 69.04% (+5.22 pp). Moderate rank weighting combines repeated answer support with a judge’s ability to elevate an outnumbered correct candidate.Source: Combining answer frequency and judge… · p. 9
  • Judge selection gained 6.02 pp when correct answers were rare and 7.87 pp at intermediate availability, but lost 3.22 pp when correct candidates dominated the full bank.Where correct candidates were present but less common, first-ranked selection won in 54.5% of questions and lost in 24.5%; in majority-correct questions, majority voting was superior in 17.6% versus 8.7%.Source: Combining answer frequency and judge… · p. 9, Figure 3 · p. 11
  • For pools with 0 < ppool < 0.5, preserving the original rationale–answer association rather than swapping rationales increased rank AUC by 0.173 (95% CI, 0.139–0.208); swapped rationales scored 0.482.Unperturbed candidates scored 0.684 rank AUC; answers-only and rationales-only candidates scored 0.654 and 0.655, respectively.Source: Combining answer frequency and judge… · pp. 9–10
  • Hybrid selection reached 70.82–70.95% accuracy versus 63.82% for majority voting (+7.13 pp), while first-ranked selection beat majority voting across all five benchmarks by 4.33–10.62 pp.The hybrid plateau occurred at judge influence t = 2–4; first-ranked selection alone achieved 69.04%.Source: Combining answer frequency and judge… · p. 9, Figure 3 · p. 11

Candidate pool size changes the relative value of voting and judging

  • Tie frequency and tie-breaking did not explain the small-pool voting deficit: odd-pool accuracy differed from adjacent-even midpoints by at most 0.39 pp, with all seven confidence intervals including zero.Odd pools reduced plurality tie rates by 1.64–5.96 pp, while tie-neutral scoring changed accuracy by at most 0.36 pp.Source: Candidate pool size changes the… · p. 12
  • As correct answers accumulated enough support for voting in larger pools, judging cost rose from US$0.10 per 1,000 additional correct answers at k = 2 to US$6.28 at k = 14, and was undefined at k = 16.For k ≤ 6, selection cost US$0.10–US$0.39 per 1,000 additional correct answers over majority voting.Source: Candidate pool size changes the… · pp. 12–13
  • At k = 16 on matched questions, selection raised Mimo no-think accuracy from 20.28% to 57.73% (+37.44 pp), but Mimo medium-thinking accuracy from 61.83% to 64.79% (+2.96 pp).Each setting generated its own candidate answers and rationales.Source: Candidate pool size changes the… · pp. 12–13
  • First-ranked selection’s advantage over majority voting fell from 13.52 pp at k = 2 to 0.75 pp at k = 14 and −0.34 pp at k = 16, with confidence intervals including zero at k = 14 and 16.The paired 95% confidence interval remained above zero through k = 12.Source: Candidate pool size changes the… · p. 12, Figure 4 · p. 13
  • In pools where correct answers were under 25%, judge selection beat voting by 40.1 pp at k = 6 and 20.8 pp at k = 16; with at least 25% correct, voting led by 4.0 and 8.2 pp.The respective pool-composition condition was 0 < ppool < 0.25 versus ppool ≥ 0.25; all-wrong pools were excluded.Source: Candidate pool size changes the… · p. 12, Figure 4 · p. 13

Discussion

The discussion frames multi-agent reasoning as a pipeline with separate bottlenecks in candidate generation, judge recognition and terminal selection. Correct answers can be generated yet lost through unfiltered interaction or answer selection.

Discussion

  • Accuracy reflects whether generation supplies a correct answer, whether a judge recognizes it and whether terminal selection acts on that signal; without selection pressure, groups can lose generated correct answers through memetic drift.In judge-guided systems, final accuracy combines correct-answer availability, judge reliability and answer selection.Source: Discussion · p. 14
  • Additional discussion rounds raised token use without consistent accuracy gains, while extra candidates help only if selection identifies the correct answer or generation makes it the overwhelming majority.In fixed pools, judge-guided selection improved accuracy when correct answers were present but outnumbered.Source: Discussion · p. 14
  • Judge reliability varies with the task, candidate generator and correct-answer fraction, so evaluator quality should be measured within the generator regime and availability range where it will be used.Even matched questions can yield different availability–reliability relationships because each generator setting produces its own candidate traces; static benchmark scores are not universal guarantees.Source: Discussion · p. 14
  • System designers should first check whether generation supplies a correct option, then test evaluator reliability on minority-held candidates from that generator and compare vote, judge and hybrid rules on held-out data.Choose among rules by accuracy and cost; frequency captures repeated support, while judging evaluates rationale and conclusion coherence.Source: Discussion · pp. 14–15
  • The fixed-pool analyses isolate terminal selection but omit live feedback in which agents may repeat, ignore, reject or adopt information and thereby change candidate availability.Agents may also revise or combine information, so future work should track information evolution across dynamic multi-turn graphs, open-ended generation and real-time execution.Source: Discussion · p. 15
  • Scaling reliable systems requires more than adding model instances: tracking generation, receiver-mediated local selection, transformation and terminal selection can help protect correct answers from being lost.Source: Discussion · p. 15

Methods

The methods combine controlled MedXpertQA comparisons of multi-agent protocols and communication rules with independently generated candidate banks for evaluating answer selection. Candidate collection used adaptive quotas, with availability measures defined over parse-valid traces.

Initial MAS architecture and discussion experiments

  • On 2,450 MedXpertQA questions, four protocols were each run with seeds 42, 43 and 44, yielding 7,350 evaluations per protocol.The protocols were single-answer, majority voting, open committee and anonymous evidence board.Source: Initial MAS architecture and… · p. 15
  • Majority voting selected the most frequent of five independent responses; open committee agents revised after peer input, while an anonymous evidence board pooled Round 1 evidence for a judge.The evidence-board judge received no agent identifiers, answer-frequency metadata or explicit links between evidence and answer choices.Source: Initial MAS architecture and… · p. 15
  • A sweep compared star, complete, ring and hierarchical networks under alternative stopping rules, with 3,150 trials per topology-by-condition cell and token use normalized to the single-agent baseline.Conditions included the original stopping rule, confidence-qualified consensus and forced six-round execution.Source: Initial MAS architecture and… · p. 16

Candidate generation and adaptive candidate banks

  • Candidate traces paired an answer with its rationale and were generated independently by DeepSeek-V4-Flash (Preview) and Mimo v2.5 across five multiple-choice benchmarks and HLE open-answer QA.Inference used either no-think or medium-thinking conditions; correctness labels measured answer-key match, not whether the rationale was logically sound.Source: Candidate generation and adaptive… · p. 16
  • Each question began with 16 generation attempts; collection stopped at two correct and six wrong traces, expanded up to 128 attempts if none were correct, and up to 64 for other quota deficiencies.Closed-choice prompts rotated a focus option to encourage rationale diversity while allowing a different answer.Source: Candidate generation and adaptive… · p. 17
  • The full-bank pgen measure was the fraction of parse-valid traces with a correct answer, reflecting this quota-driven collection protocol rather than unconstrained single-call success.Parse failures were excluded from denominators and pool construction; quota-deficient banks remained in availability summaries but were excluded when required candidates were missing. Initial-16 availability was also calculated from parse-valid traces among the first 16 attempts.Source: Candidate generation and adaptive… · p. 17

LLM judge ranking and reliability fitting

The section specifies a blinded candidate-ranking protocol and an availability-based fit for judge reliability, with separate analyses for HLE and matched generator banks.

LLM judge ranking and reliability fitting

  • At temperature 0, the ranking judge ordered eight shuffled rationales without seeing selected answers, correctness labels, or generation metadata.A ranking was usable only if it included every anonymous candidate identifier exactly once.Source: LLM judge ranking and reliability… · p. 17
  • Rank AUC measured how often a correct candidate ranked above a wrong one, and its relationship with correct-answer availability was fitted using a weighted three-parameter sigmoid.AUC was pair-weighted within 0.02-wide availability bins; half-rise midpoints were interpreted only for fits with weighted R2 ≥ 0.50.Source: LLM judge ranking and reliability… · p. 18
  • The primary rank-AUC analysis included 15,336 of 15,784 questions with candidate banks, each contributing a pool with both correct and incorrect candidates.The study also evaluated five sampled pools per question and a fixed-composition control with two correct and six wrong traces.Source: LLM judge ranking and reliability… · p. 18

Separate HLE comparison and open-answer QA audit

  • HLE was analysed separately: 475 multiple-choice questions contributed to the fit and 494 to replay, while 811 QA questions contributed eligible ranked pools.For QA, a separate equivalence judge compared candidate answers with reference answers; the ranking judge saw neither references nor equivalence labels.Source: Separate HLE comparison and… · pp. 18–19
  • 96.8% of 124 reviewed answers matched human labels, and the model correctly labelled 97.5% of 200 constructed controls.All 124 reviewed answers and 197 of 200 control outputs followed the required response format.Source: Separate HLE comparison and… · p. 19

Matched generator-bank comparison

  • Matched Mimo no-think and medium-thinking banks were compared under the same DeepSeek judge using 12,477 questions without fixed composition and 7,544 with fixed composition.Paired curve differences used 1,000 benchmark-stratified question bootstraps.Source: Matched generator-bank comparison · p. 19

Offline replay over fixed candidate pools

The section describes an offline replay over fixed candidate pools, varying answer-selection rules and judge-ranking influence while holding candidate content and, except in a shuffled-order control, judge orderings fixed.

Offline replay over fixed candidate pools

  • 81,390 fixed k = 8 candidate pools from 16,278 questions across five benchmarks were replayed with candidate text, rationales and answer frequencies held constant.Each question contributed five pools; stored judge orderings were fixed except in the shuffled-order control.Source: Offline replay over fixed candidate… · p. 19
  • Single-candidate sampling, majority-vote tie resolution and shuffled-order control used deterministic seeds 0–99; pool-based replay scores were averaged across seeds and retained pools per question.Global uncertainty was estimated with 1,000 question-level bootstrap resamples using seed 20260628.Source: Offline replay over fixed candidate… · p. 19
  • For Fig. 3e, first-ranked selection and majority voting were compared per question after averaging accuracy across the same five stored pools of eight candidates.Questions were grouped by whether correct candidates were less common (0 < pgen < 0.5) or more common (0.5 ≤ pgen < 1); pgen = 0 or 1 cases were included overall, with all 363 tied and omitted from separate groups.Source: Offline replay over fixed candidate… · p. 19

Ranking influence parameter (t)

  • The t = 2–4 plateau was the flat region in the tested t = 0–5 sequence, which weighted each trace by wi = (k - i)^t and summed weights by answer.At k = 8 and t = 2, ranks 0–7 received weights 64, 49, 36, 25, 16, 9, 4 and 1.Source: Ranking influence parameter (t) · p. 20
  • t = 0 reproduces majority voting, while t = 1–4 combines candidate frequency with judge rank ordering.Tied answer scores were resolved with deterministic seeds 0–99 and averaged across 100 results.Source: Ranking influence parameter (t) · p. 20

Targeted reranking and information perturbation

The section evaluates targeted reranking on matched eight-trace pools where correct candidates were outnumbered, varying what information the judge could see. It describes the pool selection and the paired evaluation procedure.

Targeted reranking and information perturbation

  • 713 matched eight-trace pools contained outnumbered correct candidates, with each pool’s realised correct-trace fraction ppool between 0 and 0.5.Pools came from DeepSeek→DeepSeek on HLE MC and GPQA, and Mimo no-think→DeepSeek on GPQA.Source: Targeted reranking and information… · p. 20
  • Within each generator–judge cell, three non-overlapping samples were capped at 80 questions with at least 16 traces in the full candidate bank.Samples targeted the lowest pgen values, questions nearest the fitted half-rise, and highest diversity-plus-signal scores, defined as pgen(1 - pgen) max(fitted AUC - 0.5, 0).Source: Targeted reranking and information… · p. 20
  • Five reranking conditions showed the judge answer plus rationale, answer only, rationale only, answers reassigned across labels, or rationales reassigned across labels.Reassignments used deterministic within-pool permutations that moved content across candidate labels.Source: Targeted reranking and information… · p. 20
  • Pair-weighted rank AUC used 6,488 correct–wrong comparisons per condition, and paired differences were assessed with 2,000 pool-level bootstrap resamples.The bootstrap used seed 1729.Source: Targeted reranking and information… · p. 20

Pool size and reconstructed token-based cost

This section specifies matched pool-size comparisons, an offline odd-versus-even sensitivity test, reconstructed token-cost accounting, and bootstrap and exclusion procedures.

Pool size and reconstructed token-based cost

  • The primary pool-size analysis tested k ∈ {2, 4, 6, 8, 10, 12, 14, 16}, retaining questions with parse-valid outputs for every compared strategy.DeepSeek cohorts contained 11,207–11,241 questions, while the separate Mimo k = 16 cohort contained 16,247.Source: Pool size and reconstructed… · p. 20
  • At each k, majority voting and first-ranked selection used the same candidate traces, with the judge ranking the pool and selection returning the top trace’s answer.The direct no-think versus medium-thinking comparison used the same 11,241 question identifiers, with Mimo generating candidates and DeepSeek ranking them.Source: Pool size and reconstructed… · pp. 20–21

Odd-versus-even pool-size sensitivity analysis

  • Tie-neutral sensitivity changed majority-vote accuracy by at most 0.36 percentage points across k (0.01 percentage points at k = 2).This offline analysis used truncated k = 16 pools and did not reuse the inherited judge ranking at smaller pool sizes.Source: Odd-versus-even pool-size sensitivity… · p. 21

Token cost reconstruction and blended pricing

  • Token costs were reconstructed separately for generation and ranking, using character-count estimates and a 90% cache-hit/10% cache-miss input blend.Incremental judging cost per 1,000 additional correct answers was defined only when ranking produced more correct answers than majority voting.Source: Token cost reconstruction and blended… · p. 21

Statistical analysis, exclusions and data-quality checks

  • Two-sided percentile 95% bootstrap intervals were used; 99.63% of tasks in the cost-analysis audit produced parse-valid outputs, and invalid outputs were excluded.Pools lacking required traces were excluded, and all-correct or all-wrong pools were excluded from rank AUC calculations.Source: Statistical analysis, exclusions and… · p. 22

Software, provenance and reproducibility

The study’s released, configuration-driven implementation and versioned analysis materials support reproducibility, with an access-controlled archive preserving research outputs.

Software, provenance and reproducibility

  • The released implementation separates candidate-bank construction, judge ranking, curve fitting, fixed-pool replay, information perturbation and pool-size cost analysis into versioned modules controlled by configuration manifests.The analysis code targets Python 3.10 or later; the manuscript build was verified under Python 3.12 using NumPy, pandas, SciPy, Matplotlib and statsmodels.Source: Software, provenance and reproducibility · p. 22
  • Analysis scripts and reproduction manifests are versioned on GitHub at release commit 7840cf3dbde77e61a1b4349ad3239669a7502431, and an access-controlled Hugging Face archive contains raw outputs, candidate banks, ranked pools and summaries.No human participant data were collected.Source: Software, provenance and reproducibility · p. 22

Definitions and notation

The section defines live communication as peer selection or transformation, with stochastic retention and popularity-driven cascades contributing to a generation–retention bottleneck; fixed-pool replay isolates terminal selection from live communication.

Definitions and notation

  • In live communication, receivers may repeat, ignore, reject, or adopt peer information, or transform statements by revising and combining them into a new rationale.Source: Definitions and notation · p. 22
  • Across rounds, stochastic persistence is memetic drift, while repeated adoption of popular statements can produce a majority-biased cascade; both can contribute to a generation–retention bottleneck.Source: Definitions and notation · p. 22
  • Fixed-pool replay evaluates judge ranking and terminal selection over stored candidate traces, holding traces and rankings fixed to isolate terminal selection from live communication processes.Source: Definitions and notation · p. 23

Models and inference conditions

The study tested seven generator–judge combinations, including DeepSeek-V4-Flash (Preview) and Mimo v2.5, with generation and ranking settings varied by experiment.

Models and inference conditions

  • DeepSeek-V4-Flash (Preview) was accessed through the API in June and July 2026 and used medium-thinking for reported generation and ranking conditions.Candidate-bank generation used temperature 1.0 and ranking used temperature 0.0.Source: Models and inference conditions · p. 23
  • Mimo v2.5 was evaluated with internal reasoning disabled (no-think) or enabled at the medium-thinking setting.The tested configurations also included Mimo v2.5 generation with DeepSeek-V4-Flash ranking, and Mimo v2.5 used as both generator and judge with reasoning disabled or enabled.Source: Models and inference conditions · p. 23
  • Seven generator–judge combinations were tested, with the combinations and reasoning controls listed in Extended Data Table 2.The table covers generator and ranking-judge models, sampling temperatures and reasoning controls.Source: Table 2 · p. 23, Models and inference conditions · p. 23

Benchmarks and correctness labels

The study used four primary benchmarks with official answer keys and treated HLE as a separate external comparison, with open-answer HLE serving only as diagnostic evidence. It also documents data and code access for reproduction.

Benchmarks and correctness labels

  • MMLU-Pro, GPQA, MedXpertQA and MuSR formed the primary modelling set, while HLE was analysed separately; HLE open-answer labels were diagnostic only.Extended Data Table 3 summarizes benchmark splits, correctness sources and study roles.Source: Benchmarks and correctness labels · p. 24, Table 3 · p. 24

Data availability

  • A redistribution-compatible data release is planned upon publication, with non-redistributable components remaining available through controlled access.The code repository includes analysis, replay and figure-generation code, configuration files, tests, provenance records, reproduction guides and a lock file identifying the data revision.Source: Data availability · p. 25, Code availability · p. 25

Dataset Pool construction

The Dataset Pool construction section presents generator–judge results with judge AUC, fit R², and boundary-fit quality. The reported rows include both usable and strong fits.

Dataset Pool construction

  • The reported results pair judge AUC and fit R² values with a fit-quality classification.The table headers also list midpoint, pairs, pools, bins, and parse fields, but the supplied passages do not explain those fields.Source: Dataset Pool construction · pp. 38–39
  • 0.728 AUC and 0.700 fit R² are reported with a usable boundary fit.The supplied passages do not make the associated generator–judge pairing clear.Source: Dataset Pool construction · p. 38
  • 0.815 AUC and 0.855 fit R² are reported with a strong boundary fit.The supplied passages do not make the associated generator–judge pairing clear.Source: Dataset Pool construction · p. 38

Related papers

Loading 2608.25937v1…