PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge

Wei Yang, Shixuan Li, Heng Ping, Peiyu Zhang, Paul Bogdan, Jesse Thomason

arXiv:2602.09341v2cs.AI

TL;DR

Multi-agent aggregation often relies on majority voting even though correlated agents can converge on an incorrect rationale, motivating evidence-sensitive adjudication. AgentAuditor builds a Reasoning Tree and audits branch-level evidence at critical divergences, with ACPO training the adjudicator against misleading majority cues. Across frameworks and benchmarks, it consistently improves aggregation over majority voting, while its benefits are concentrated in cases containing a distinguishable correct minority path.

  • Problem

    Majority voting discards reasoning evidence and can fail under correlated errors when agents converge on an incorrect rationale.

  • Method

    AgentAuditor organizes traces into a Reasoning Tree, audits localized branch evidence at Critical Divergence Points, and uses ACPO to reduce conformity to misleading majority cues.

  • Results

    AgentAuditor consistently improves aggregation over majority voting across MAS frameworks and reasoning benchmarks, with gains of up to 5% absolute accuracy.

  • Takeaways & Limitations

    Localized evidence-based aggregation can recover correct minority paths that majority voting may suppress, with lower token cost.

  • Takeaways & Limitations

    The method has limited evidence when traces are nearly identical, all agents share the same error, or the correct solution is absent from the candidate set.

Abstract

from arXiv · show

Multi-agent systems (MAS) can substantially extend the reasoning capacity of large language models (LLMs). Most MAS frameworks aggregate agent outputs via simple majority voting, discarding the evidential structure of reasoning traces. Majority voting is brittle under confabulation consensus, where agents share correlated biases and converge on the same incorrect rationale. We introduce AgentAuditor, which moves beyond frequency-based aggregation by organizing agent traces into a Reasoning Tree that explicitly represents agreements and divergences in their reasoning. AgentAuditor resolves conflicts by comparing branch-level evidence at critical divergence points, turning global adjudication into efficient, localized verification. We further propose Anti-Consensus Preference Optimization (ACPO), which trains the adjudicator with evidence-verified preference supervision to reduce conformity to misleading majority cues. Across four MAS frameworks and multiple reasoning benchmarks, AgentAuditor consistently improves aggregation performance over majority voting, with gains of up to 5% absolute accuracy while remaining token-efficient.

1 Introduction

Multi-agent systems often reduce rich reasoning traces to majority votes, which can fail when correlated agents converge on an incorrect rationale. AgentAuditor instead organizes traces into a Reasoning Tree and audits localized divergences, with ACPO reducing conformity to misleading majority cues.

  • Majority voting compresses diverse reasoning traces, intermediate evidence, and disagreements into context-free answer counts.
  • Shared pretraining distributions, alignment biases, and prompt anchors can produce confabulation consensus despite repeated agreement.
  • AgentAuditor organizes agreements and disagreements into a Reasoning Tree and compares evidence at Critical Divergence Points.
  • ACPO fine-tunes the Auditor with evidence-verified preference supervision to reduce conformity to popular-but-wrong branches.
  • AgentAuditor consistently improves over majority voting across multiple MAS frameworks and reasoning benchmarks, especially when correct answers occur only in minority branches.

2 Related Work

Related work commonly aggregates multi-agent reasoning with majority voting or evaluates complete traces, leaving a gap in localized adjudication of the disagreements that separate competing reasoning paths.

  • Multi-agent research includes debate protocols, fixed communication topologies, and self-organizing collaboration graphs for distributed reasoning.
  • Majority voting and simple ensembling remain common, but their reliability depends on sufficiently independent individual errors.
  • Correlated agent errors can create confabulation consensus, where repeated unsupported rationales yield plausible but incorrect answers.
  • Referee, debate-adjudication, and peer-review approaches move beyond answer counting but typically evaluate whole traces rather than localized disagreements.

3 Problem setup.

The paper formulates aggregation as selecting one prediction from multiple agent traces and targets cases where answer multiplicity conflicts with branch-level evidence.

  • A multi-agent system produces agent outputs containing reasoning traces and final answers, which an aggregator maps to one prediction.
  • Majority voting selects the most frequent answer but can fail when correlated agents support an erroneous hypothesis and a correct solution appears in a minority trace.
  • The stated goal is an aggregator robust to answer multiplicity while remaining sensitive to branch-level evidence.

4 Methodology

AgentAuditor converts free-form multi-agent traces into a compact Reasoning Tree, then audits localized branch divergences under shared context rather than judging complete traces in isolation. Its traversal can preserve alternatives when early pruning is risky and aggregates surviving candidates conservatively.

  • Reasoning Tree construction: The framework constructs a Reasoning Tree by clustering semantically equivalent steps and exposing substantive disagreements as branch points.
  • Trace atomization: Trace atomization decomposes each reasoning trace into shorter steps representing local reasoning operations, factual assertions, or calculations.
  • Reasoning Tree construction: Semantic alignment merges a step with its best-matching child when cosine similarity reaches threshold τ; otherwise, it creates a new branch.
  • Node attribution: Each tree node records the agents whose traces traverse it, while support frequency is not treated by the local Auditor as direct correctness evidence.
  • Localized auditing: At Critical Divergence Points, the Auditor compares shared context with compact branch-specific evidence and selects the most reliable outgoing branch.
  • Localized auditing: The Auditor evaluates branches for factual accuracy, logical soundness, constraint adherence, assumption validity, and trace hygiene.
  • Aggregation: Traversal can preserve multiple plausible branches to reduce irreversible early pruning; differing surviving answers receive final adjudication, with majority fallback otherwise.

5 Anti-Consensus Preference Optimization

ACPO addresses sycophancy in auditing by training the Auditor to compare branch evidence rather than rely on misleading support counts. It uses evidence-verified preference supervision across varied support configurations to reduce majority-driven shortcuts.

  • ACPO trains the Auditor to decouple branch validity from majority support when support counts are misleading.It complements structure-adaptive auditing, which localizes disagreements at critical divergence points.
  • Support imbalance can cause an instruction-tuned Auditor to prefer a majority branch even when its evidence is weaker.This is characterized as sycophancy bias, especially when the majority branch substantially outnumbers the minority branch.
  • ACPO constructs Dtrap from evidence-verifiable cases containing competing correct and incorrect reasoning branches.The dataset retains cases where the ground-truth outcome reliably distinguishes the branches.
  • Training includes varied support configurations so branch correctness cannot be inferred from support frequency alone.Balanced branch presentation also reduces positional and support-related shortcuts.
  • The Auditor is optimized with DPO on evidence-verified preferences, using correct branches as preferred responses and incorrect branches as rejected responses.The objective uses a reference model, and β controls the strength of implicit KL regularization to that model.

6 Experiments

Across multiple MAS architectures and reasoning benchmarks, AGENTAUDITOR improves aggregation over Majority Voting and LLM-as-Judge, especially when the majority is wrong. It also reduces token cost by auditing localized divergences and remains effective across backbones, while ablations identify beam search as especially important.

  • RQ1: Does AGENTAUDITOR consistently improve MAS aggregation?: AGENTAUDITOR consistently outperforms Majority Voting and LLM-as-Judge across four MAS architectures and four reasoning benchmarks.It achieves an average absolute improvement of approximately 3.2% over Majority Voting, with clear gains on AMC and GSM8K.
  • RQ2: Performance under Majority Failure (Minority Correct): 65.35% on GSM8K and 81.82% on AMC are achieved in MinC cases where the majority is wrong but a correct minority exists.Majority Voting reaches 0% in this regime by construction, whereas AGENTAUDITOR recovers a large fraction of majority-wrong cases.
  • RQ2: Performance under Majority Failure (Minority Correct): AGENTAUDITOR remains strong on MajC cases, reaching 97.28% on GSM8K and 91.67% on AMC while improving MinC robustness.The method selectively corrects hard majority-wrong failures without broadly sacrificing standard consensus performance.
  • RQ3: Is AGENTAUDITOR Token-Efficient?: AGENTAUDITOR uses 973 total tokens per sample, reducing cost by 44.8% versus LLM-as-Judge and 52.4% versus LLM-as-Solver.Its localized evidence inputs avoid regenerating or re-evaluating complete solutions.
  • RQ4: Does AGENTAUDITOR Generalize Across LLM Backbones?: AGENTAUDITOR consistently improves Majority Voting across tested LLM backbones and MAS protocols, with roughly +3.3% to +4.1% gains on LLaMA-3B.The persistent improvements indicate that the adjudication mechanism is not tied to a particular generator family.
  • RQ5: Do AGENTAUDITOR Modules All Contribute?: Removing beam search consistently degrades performance, whereas replacing the step splitter or embedding encoder has smaller effects.The ablation results indicate that alternative lineages matter more than particular segmentation or embedding variants.
  • RQ2: Performance under Majority Failure (Minority Correct): A case study shows AGENTAUDITOR rejecting branches with a unit mismatch or unsupported population assumption and recovering the correct minority path.The method audits local branch evidence rather than counting final answers or judging complete traces.

7 Conclusion

AGENTAUDITOR organizes multi-agent reasoning traces into a Reasoning Tree and audits Critical Divergence Points to recover correct minority paths. Experiments show consistent gains with lower token cost, supporting localized evidence-based aggregation over final-answer agreement alone.

  • Conclusion: AGENTAUDITOR organizes reasoning traces into a Reasoning Tree and audits Critical Divergence Points to recover correct minority paths.The framework models where reasoning paths diverge rather than relying only on final-answer agreement.
  • Conclusion: Experiments across MAS frameworks, reasoning benchmarks, and LLM backbones show consistent gains with lower token cost.The results highlight localized evidence-based aggregation as an alternative to relying only on final-answer agreement.
  • Conclusion: A case study shows Majority Voting selecting an incorrect consensus while AGENTAUDITOR detects invalid reasoning branches and recovers the correct minority path.The detected issues include a unit mismatch and an unsupported population assumption.

Limitations

AGENTAUDITOR is most effective when agents generate meaningfully different reasoning paths and a correct minority hypothesis exists, but it does not guarantee correct aggregation. Its effectiveness can be limited by similar traces, shared errors, absent correct hypotheses, saturated baselines, and over-trust in structured decisions.

  • AGENTAUDITOR is most effective when multiple agents produce different reasoning paths and a correct minority hypothesis exists.
  • The method may have limited headroom when the underlying multi-agent system is already strong and majority voting is near saturation.
  • AGENTAUDITOR does not guarantee correct aggregation because the Auditor can make incorrect branch-level judgments.
  • Highly similar traces, subtle shared errors, or missing correct hypotheses constrain the Auditor’s ability to recover the answer.
  • Structured auditing may create overconfidence, especially in open-ended or safety-critical domains where local plausibility can mask incorrect evidence.

C.2 Evaluation and Adjudication in Multi-Agent Reasoning

Multi-agent coordination improves reasoning diversity, but collective correctness remains difficult because correlated errors can make majority consensus unreliable. Existing validation methods largely emphasize single-agent reflection rather than distributed adjudication across agents.

  • Multi-agent systems commonly aggregate outputs through majority voting or rule-based ensembling, assuming consensus approximates truth.
  • Agents trained on similar distributions can exhibit correlated errors, causing plausible but incorrect answers to dominate collective decisions.
  • Automatic validation methods using reward modeling and reflective self-evaluation primarily address single-agent reasoning rather than distributed adjudication.

D.1 Experimental Setup

The evaluation uses multiple reasoning benchmarks, baselines, controlled trace generation, and preference data built from reasoning-tree divergences. The theoretical analysis explains correlated-error failure and motivates branch-level evidence comparison beyond answer frequency.

  • Experimental Setup: The evaluation covers GSM8K, MATH, AMC, and MMLU to assess mathematical, symbolic, and general reasoning abilities.
  • Experimental Setup: AGENTAUDITOR is compared with single-agent, debate, collaborative, and dynamic-workflow baselines while preserving identical upstream candidate generation.The framework replaces only the final voting stage with its adjudication process.
  • Experimental Setup: Training preferences retain reasoning-tree forks where the ground-truth answer appears in at least one branch, targeting branch selection rather than answer generation.
  • Experimental Setup: Fork-level augmentation balances preferences and randomizes branch order to discourage majority-following and positional shortcuts.
  • Theoretical Perspective: The theoretical analysis shows that positive pairwise correlation prevents majority accuracy from improving reliably merely by increasing the number of agents.
  • Theoretical Perspective: Reasoning trees group redundant traces into semantic branches, while auditing compares evidence at divergence points instead of relying solely on multiplicity.

F Method

The method formulates aggregation over agent reasoning traces and addresses correlated-error consensus through semantic deduplication and branch-level auditing. On open-domain QA, it consistently improves over majority voting, with gains concentrated on contested questions and selective overrides rather than indiscriminate switching.

  • Problem Formulation: The aggregation problem maps agent outputs containing reasoning traces and final answers to one prediction, while majority voting selects the most frequent answer.
  • Problem Formulation: Correlated errors can make many agents converge on an incorrect answer, allowing majority voting to defeat a correct minority trace.
  • Method: AGENTAUDITOR combines semantic deduplication with auditing at critical divergence points to remain sensitive to validity when correct reasoning is outnumbered.
  • Evaluation: Open-domain QA evaluation uses MuSiQue and TriviaQA with three-agent candidate sets and the same candidates supplied to every aggregation method.
  • Results: AGENTAUDITOR consistently improves over Majority Voting on both MuSiQue and TriviaQA under the same three-agent budget.
  • Results: +4.38 absolute points: trained AGENTAUDITOR reaches 34.17% EM versus 29.79% for Majority Voting on MuSiQue-ctx.
  • Results: On TriviaQA, trained AGENTAUDITOR reaches 63.37% EM, outperforming MV@3 by +3.50 points.
  • Results: Improvements come mainly from contested questions, where the Auditor exploits structural differences among competing branches while leaving unanimous predictions unchanged.

G.4 Recoverability Analysis: Generation Errors vs. Aggregation Errors

AGENTAUDITOR’s gains are bounded by whether correct evidence exists in the candidate traces: it can recover aggregation mistakes but cannot fix generation-limited errors. Its advantage over flat judging comes from structured, selective comparison of reasoning branches at semantic divergence points.

  • Recoverability: When no candidate contains the correct answer, better adjudication cannot correct the failure because the error originates in generation rather than aggregation.
  • Recoverability: 83.1% of Majority Voting errors on MuSiQue-ctx are generation-limited, while 16.9% are aggregation-recoverable.Within the recoverable subset, the trained Auditor corrects 64.8% of cases.
  • Recoverability: 76.2% of TriviaQA Majority Voting failures are generation-limited, while the trained Auditor recovers 47.7% of the remaining cases with zero leakage.AGENTAUDITOR selects among existing branches rather than generating unsupported answers.
  • Flat judging versus auditing: The flat LLM-as-Judge only marginally improves over Majority Voting on MuSiQue-ctx and performs worse on TriviaQA.The comparison uses the same candidate set and inference budget, differing in how evidence is organized and compared.
  • Flat judging versus auditing: AGENTAUDITOR’s improvement comes from structured comparison of competing branches, with fewer overrides but higher precision than the flat judge.On TriviaQA, the trained Auditor overrides 25.6% of contested questions with 38.8% precision, versus 45.7% and 29.8% for the flat judge.
  • Case study: In the case study, auditing critical divergence points exposes a unit mismatch and an unsupported population assumption that majority voting would otherwise preserve.The Auditor favors flavor-wise accounting at CDP-1 and rejects the extra “×7” factor introduced at CDP-2.
Loading 2602.09341v2…