PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets

Dat Tran, Douwe Kiela

arXiv:2604.02460v2cs.CLcs.MA

TL;DR

The paper asks whether reported MAS advantages remain when test-time reasoning computation is matched and how context utilization affects the comparison. It combines a Data Processing Inequality analysis with controlled experiments across model families and architectures, finding that SAS generally match or outperform MAS under fixed thinking-token budgets, while MAS can become competitive under degraded context utilization. Diagnostics further identify evaluation artifacts that affect apparent gains.

  • Problem

    Comparisons between MAS and SAS remain unclear because reported MAS gains are often confounded by unequal test-time computation, especially for fixed-budget multi-hop reasoning.

  • Method

    The paper combines an information-theoretic Data Processing Inequality argument with controlled comparisons of SAS and multiple MAS architectures under matched reasoning-token budgets.

  • Results

    SAS consistently match or outperform MAS on multi-hop reasoning across multiple model families when reasoning-token budgets are held constant.

  • Takeaways & Limitations

    Reported MAS advantages are better explained by compute and context effects than inherent architectural superiority, with multi-agent structure most relevant when single-agent context utilization degrades.

  • Takeaways & Limitations

    The main conclusion is bounded to settings with proper context utilization: sufficiently degraded single-agent context can make MAS competitive or occasionally better.

Abstract

from arXiv · show

Recent work reports strong performance from multi-agent LLM systems (MAS), but these gains are often confounded by increased test-time computation. When computation is normalized, single-agent systems (SAS) can match or outperform MAS, yet the theoretical basis and evaluation methodology behind this comparison remain unclear. We present an information-theoretic argument, grounded in the Data Processing Inequality, suggesting that under a fixed reasoning-token budget and with perfect context utilization, single-agent systems are more information-efficient. This perspective further predicts that multi-agent systems become competitive when a single agent's effective context utilization is degraded, or when more compute is expended. We test these predictions in a controlled empirical study across three model families (Qwen3, DeepSeek-R1-Distill-Llama, and Gemini 2.5), comparing SAS with multiple MAS architectures under matched budgets. We find that SAS consistently match or outperform MAS on multi-hop reasoning tasks when reasoning tokens are held constant. Beyond aggregate performance, we conduct a detailed diagnostic analysis of system behavior and evaluation methodology. We identify significant artifacts in API-based budget control (particularly in Gemini 2.5) and in standard benchmarks, both of which can inflate apparent gains from MAS. Overall, our results suggest that, for multi-hop reasoning tasks, many reported advantages of multi-agent systems are better explained by unaccounted computation and context effects rather than inherent architectural benefits, and highlight the importance of understanding and explicitly controlling the trade-offs between compute, context, and coordination in agentic systems.

1 Introduction

The paper examines whether multi-agent gains reflect architectural benefits or simply greater test-time computation, focusing on fixed thinking-token budgets for multi-hop reasoning. It combines an information-theoretic perspective with controlled comparisons and diagnostic analyses.

  • MAS comparisons are often confounded because multi-agent systems consume more reasoning tokens through longer traces or multiple interactions.This makes architectural gains difficult to distinguish from increased computation.
  • The study defines thinking-token budgets as intermediate reasoning tokens excluding prompts and final answers, then asks when SAS or MAS perform better.Its questions concern fixed-budget advantages, conditions favoring MAS, and reliable evaluation.
  • Under fixed budgets, the Data Processing Inequality suggests that multi-agent decomposition can introduce communication bottlenecks and information loss.The perspective also predicts MAS advantages when single-agent context utilization degrades or additional computation is available.
  • Across Qwen3, DeepSeek-R1-Distill-Llama, and Gemini 2.5, SAS consistently match or outperform MAS on multi-hop reasoning under matched reasoning-token budgets.The comparison covers multiple MAS architectures in a controlled empirical study.
  • Diagnostic analyses identify API-based budget-control artifacts, benchmark vulnerabilities, and systematic architecture-specific failure modes.These diagnostics extend beyond aggregate performance to evaluation methodology and system behavior.

2 Related Work

Related work shows that apparent MAS advantages depend on computation, model capability, task regime, and implementation. This paper positions its contribution as a matched-budget comparison across representative collaboration mechanisms while accounting for context degradation.

  • Budget-Controlled Evaluation: Prior budget-aware studies report that elaborate reasoning or search strategies often lose their advantage over simpler baselines when computation is matched.This paper controls thinking tokens specifically and compares single-agent with multi-agent systems.
  • When and why MAS helps: MAS gains appear regime-dependent, concentrating in weaker-model or harder settings and diminishing as base-model capability increases.Coordination overhead is identified as a first-order constraint in this related work.
  • SAS vs. MAS: Evidence across frontier models and education analytics suggests SAS can match or surpass MAS, while collaborative-agent benchmarks show task-specific rather than universal MAS benefits.These findings complement the paper’s stricter compute controls.
  • Multi-agent collaboration mechanisms: Prior collaboration mechanisms include debate, role specialization, ensemble or self-consistency, and reflection.The paper evaluates representative mechanisms under a common matched-budget framework to separate structure from extra computation.
  • Context length, degradation, and long-context utilization: Long-context reasoning can degrade through attention dilution, noise sensitivity, context confusion, positional bias, and lost-in-the-middle effects.These findings motivate treating context utilization as an architecture-independent factor.

3 Theoretical justification

The theory models MAS messages as a processed representation of the full context and uses the Data Processing Inequality to compare ideal information access with degraded single-agent context. It predicts SAS dominance under intact context, but possible MAS competitiveness when context utilization deteriorates.

  • The theoretical section formalizes SAS–MAS comparison through minimum achievable prediction error under a thinking-token budget.For an observable representation X, Pe(X) is the smallest error achievable by a predictor observing only X.
  • The model defines C as the full single-agent context and M = g(C) as messages or summaries passed between agents.The multi-agent representation is generated from the single-agent context.
  • The Markov chain Y ←→ C ←→ M yields H(Y | M) ≥ H(Y | C), so messages retain no more information about the answer than the full context.The result follows from the Data Processing Inequality.
  • Any estimator using M can be induced from C, reproducing the same joint prediction distribution and implying Pe(C) ≤ Pe(M).Thus, an ideal single-agent system with full access to C is guaranteed to perform at least as well as the corresponding MAS.
  • Context degradation: Context degradation transforms C into an effective context C̃_α through deletion, masking, noise, distractors, or other operations that make relevant information harder to recover.The framework assumes degradation is monotone and begins with the identity transformation at α = 0.
  • Context degradation: Heavier degradation cannot improve the best achievable prediction error, because it leaves more residual uncertainty about the answer.The formalization expresses this as increasing conditional entropy with degradation.
  • Context degradation: When single-agent context utilization deteriorates, MAS may recover task-relevant information more reliably than a degraded single pass, while remaining bounded by the ideal information in C.The theory therefore predicts shrinking SAS advantages and occasional MAS superiority in sufficiently degraded regimes.

4 Approach

The study evaluates a direct single-agent pipeline against several multi-agent architectures under a shared thinking-token budget, using multi-hop reasoning benchmarks and LLM-based judging. Sequential receives particular attention because it mirrors SAS’s serial reasoning process while differing in whether intermediate states remain latent or become explicit messages.

  • Evaluation: The evaluation uses FRAMES and 4-hop MuSiQue questions, with accuracy judged by whether the ground truth appears or is semantically present in the answer.The evaluation follows an LLM-as-a-judge rubric for semantic equivalence, paraphrases, and minor formatting differences.
  • Single-agent baseline: SAS uses one direct model call allocated the entire global thinking budget B before answer extraction.The pipeline prompts the model to think step by step, then extracts text after the </think> tag or Gemini API call.
  • Single-agent baseline: SAS-L preserves the single-call pipeline and requested budget B while adding a scaffold for identifying ambiguities, proposing interpretations, evaluating them, and answering.The variant addresses cases where Gemini’s visible thought text plateaus below the requested budget without changing other compute knobs.
  • Multi-agent architectures: The study compares SAS with Sequential, Subtask-parallel, Parallel-roles, Debate, and Ensemble under the same global thinking-token budget B.Planner and aggregator components are kept near budget-neutral whenever possible, while workers receive equal or divided budget allocations depending on the architecture.
  • Multi-agent architectures: Sequential is the closest SAS comparator because both solve the whole question through serial reasoning, but Sequential externalizes intermediate states as messages while SAS keeps them latent.Figure 1 presents this latent-versus-explicit intermediate-state distinction over the same question and global budget.

5 Results

Under matched thinking-token budgets, single-agent systems are generally the strongest default for multi-hop reasoning across models and datasets, while multi-agent advantages emerge under degraded context or particular model and budget conditions. Results also show diminishing returns from larger budgets and persistent SAS competitiveness across Gemini versions.

  • SAS vs. Multi-Agent Results Across Models and Datasets: SAS is the strongest default architecture under matched thinking-token budgets, except at the lowest budget where neither approach produces useful reasoning.SAS also uses fewer thinking tokens than MAS while achieving the same or better results.
  • SAS vs. Multi-Agent Results Across Models and Datasets: SAS-L mainly improves Gemini results, especially on Gemini-2.5-Flash and higher-budget Gemini-2.5-Pro MuSiQue settings.The longer-thinking variant is not a universal gain for Qwen3-30B-A3B, where standard SAS is usually stronger in middle and high-budget ranges.
  • SAS vs. Multi-Agent Results Across Models and Datasets: Debate is the most consistently strong MAS variant, while Ensemble becomes strongest on Gemini-2.5-Pro FRAMES at 5000 and 10000 thinking tokens.Parallel-roles is frequently the next strongest MAS configuration, whereas Ensemble is more budget-dependent.
  • SAS vs. Multi-Agent Results Across Models and Datasets: Performance generally improves with larger thinking budgets before flattening, suggesting diminishing returns and possible overthinking at high budgets.For many configurations, the 1000- and 2000-token averages are already close to the 5000- and 10000-token averages.
  • SAS vs. Multi-Agent Results Across Models and Datasets: Across Gemini model versions with unlimited thinking tokens, stronger models improve both SAS and Sequential MAS, but SAS remains at least as competitive.This persistence suggests the SAS-versus-Sequential pattern is not specific to one Gemini checkpoint or release window.
  • Context degradation experiments: Under context degradation, Sequential becomes better than SAS when masking or substitution is severe, whereas distractors leave SAS ahead throughout.The experiments use Qwen3-30B-A3B on MuSiQue 4-hop with a fixed 1000-token budget and vary deletion, masking, substitution, and distractor levels.

6 Conclusion

Across datasets, model families, and five MAS architectures, SAS match or outperform MAS when thinking-token computation is normalized, unless context utilization is sufficiently degraded. The results suggest reported MAS gains often reflect compute and context effects rather than inherent architectural superiority.

  • Conclusion: Across FRAMES and MuSiQue, three model families, and five MAS architectures, SAS consistently match or outperform MAS under normalized computation.The comparison covers Sequential, Debate, Ensemble, Parallel-roles, and Subtask-parallel MAS architectures.
  • Conclusion: MAS can become competitive when single-agent context utilization is degraded beyond a certain point.
  • Conclusion: Many reported MAS gains are better explained by compute and context effects than by inherent architectural superiority.

A Paraphrasing Ablation Study

The paraphrasing ablation evaluates SAS and Sequential MAS on paraphrased MuSiQue 4-hop questions under 1k and 2k thinking-token budgets. Light paraphrasing generally hurts performance, whereas deep semantic paraphrasing improves it, especially for Gemini.

  • Setup: The ablation evaluates SAS and Sequential MAS on MuSiQue 4-hop questions at 1k and 2k thinking-token budgets.It uses light rule-based paraphrases and deep LLM-based paraphrases designed to preserve meaning while reducing lexical overlap.
  • Light Paraphrase: Light paraphrasing decreased SAS performance for Gemini from .331 to .326 and Qwen3 from .260 to .249.Sequential MAS in Qwen3 also fell from .229 to .204.
  • Deep Paraphrase: Deep paraphrasing increased Gemini-2.5-Flash SAS performance from .331 to .358 at 1k tokens.Qwen3 also showed slight gains in SAS and Sequential MAS, though they were not significant.
  • Interpretation: The divergence between harmful light paraphrasing and beneficial deep paraphrasing suggests original questions may reflect memorization or pretraining overfitting.The authors present deep semantic rephrasing as a way to elicit more robust reasoning for agent benchmarks.

B Error Analysis

The error analysis compares Gemini-2.5-Flash and Qwen3-30B-A3B across four outcome buckets, linking success to whether reasoning surfaces the gold answer and preserves it through finalization. SAS favors tighter question anchoring, while Sequential MAS benefits from breadth only when paired with late constraint checking.

  • Analysis Setup: The analysis partitions MuSiQue 4-hop examples into MAS-right/SAS-wrong, SAS-right/MAS-wrong, both-right, and both-wrong buckets.It examines internal chains at 1k thinking tokens for Gemini-2.5-Flash and Qwen3-30B-A3B.
  • MR/SW: When Sequential MAS is right and SAS is wrong, MAS writes longer thoughts, explores twice as many entities, and surfaces the gold answer more often.Gold-in-thoughts rates are 41.7% vs. 12.5% for Gemini and 56.7% vs. 18.3% for Qwen3; SAS also has extraction failures.
  • SR/MW: When SAS is right and Sequential MAS is wrong, SAS maintains higher question overlap and surfaces the gold answer more often than MAS.Gold-in-thoughts rates are 42.7% vs. 18.6% for Gemini and 63.5% vs. 28.1% for Qwen3; MAS also shows extraction failures.
  • BR: When both systems are right, both frequently contain the gold answer in their thoughts and show backtracking consistent with late constraint re-checking.For Gemini, gold appears in both systems' thoughts in 50.9% of cases; for Qwen3, the corresponding rate is 69.9%.
  • BW: When both systems are wrong, the gold answer almost never appears and both systems pursue disjoint candidates without reconciliation.The authors characterize this as mutual drift with insufficient late binding to the question’s constraints.
  • Takeaway: Overall, SAS succeeds through question anchoring and reliable final copying, whereas Sequential MAS succeeds when breadth is paired with late constraint checking.

C Limitations

The paper’s limitations constrain its conclusions to text-only multi-hop reasoning and approximate Gemini budget accounting. Its experiments vary upper thinking-token caps without requiring models to consume the full budgets.

  • Scope: The study focuses on text-only multi-hop reasoning, leaving MAS advantages involving tools, vision, or safety constraints out of scope.
  • Measurement: Gemini thinking-token accounting is approximate, so the paper reports API and content-based proxies while emphasizing accuracy under matched requested budgets.
  • Budget Control: The experiments restrict only an upper thinking-budget cap and do not require models to use all available tokens.Thus, the study does not impose a lower thinking-token bound.

F.4 Multiple Gemini model version results

This section reports unlimited-thinking results for multiple Gemini model versions and full context-degradation results with confidence intervals.

  • Unlimited-thinking results are reported for multiple Gemini model versions with confidence intervals.
  • MuSiQue 4-hop full results for Gemini-2.5-Flash are presented in Table 7.
  • Full context-degradation results are reported across all four perturbation families with 95% bootstrap confidence intervals.

G Diagnostic: On Gemini Thought Token Accounting

The Gemini diagnostic finds discrepancies between reported and visible thought-token counts, visible-text plateaus, and greater visible output from Sequential MAS. These artifacts make observed-token comparisons intractable, so the analysis matches requested budgets instead.

  • Token accounting: Gemini experiments tracked API-reported counts, content-based word counts, and proxy token counts, revealing discrepancies between reported and visible thought content.The proxy token count was derived by multiplying word count by 10/7.
  • Token accounting: At a requested 10k budget, Gemini-2.5-Flash SAS reported 1,687 thinking tokens versus 359 proxy tokens, a 4.7x inflation factor.
  • Token accounting: For Gemini-2.5-Flash SAS, visible thought output plateaued near 350 tokens, reaching 354 at 1k and 359 at 10k.
  • Token accounting: At a matched 1k budget on Gemini-2.5-Pro, Sequential MAS produced 693 proxy tokens versus 390 for SAS.The paper attributes the larger MAS text volume to concatenated outputs from separate agent calls.
  • Evaluation implications: Because reported counts are unreliable and proxy tokens scale nonlinearly with requested budget, direct observed-token comparisons are intractable.
  • Evaluation implications: The primary analysis therefore matches requested budget B, the only directly controllable variable, while acknowledging opaque differences in budget utilization.

H Gemini Results across different version:

This section presents Gemini results across model versions, including FRAMES, paraphrase, MuSiQue 4-hop, and context-degradation evaluations.

  • Gemini model results across different versions are reported in detail.
  • FRAMES: Table 10 contains full FRAMES results for Gemini-2.5-Pro.
  • Paraphrase evaluation: Table 11 reports paraphrase results with 95% bootstrap confidence intervals.
  • MuSiQue 4-hop: Table 12 reports MuSiQue 4-hop accuracy across Gemini model versions with unlimited thinking and 95% bootstrap confidence intervals.
  • Context degradation: Table 13 reports context-degradation results on MuSiQue 4-hop using Qwen3-30B-A3B with a fixed 1000-token thinking budget.
Loading 2604.02460v2…