Source-linked AI summary
DarkForest: Less Talk, Higher Accuracy for Multi-Agent LLMs
Yi Li, Songtao Wei, Dongming Jiang, Zhichun Guo, Qiannan Li, Bingzhe Li
TL;DR
Interaction-heavy multi-agent LLM systems risk error propagation and communication overhead. DarkForest preserves independent answers, structures and calibrates candidate evidence, and exposes only policy-permitted information to a coordinator. Across six reasoning benchmarks, it reports strong or near-strong quality with substantially fewer tokens than communication-heavy baselines.
Problem
Uncontrolled communication can amplify errors, compromise independent evidence, and increase inference cost in multi-agent LLM reasoning.
Method
DarkForest independently generates candidates, parses and clusters them, builds a calibrated belief state, and exposes selected evidence to a coordinator.
Results
Across six benchmarks, DarkForest achieves strong or near-strong task quality while using substantially fewer tokens than communication-heavy baselines.
Takeaways & Limitations
The results suggest that multi-agent reasoning should prioritize preserving independent evidence and controlling communication rather than increasing agent interaction.
Takeaways & Limitations
Agents may share training data, architectures, or instruction-tuning procedures, so DarkForest does not assume full statistical independence.
Abstract
from arXiv · showhide
Multi-agent LLM systems improve reasoning by combining outputs from multiple agents, but interaction-heavy methods can introduce error propagation and high communication overhead. When agents exchange raw responses or reasoning traces, incorrect intermediate reasoning may be adopted and amplified, leading to confident but wrong consensus; multi-round communication also increases token consumption, latency, and inference cost. In this paper, we propose a controlled-communication coordination framework named DarkForest. DarkForest first keeps agents independent, so each agent produces an answer without seeing the others' outputs. It then parses the raw responses into structured candidate records, groups semantically equivalent candidates into clusters, and estimates a calibrated belief distribution over these clusters using agent reliability, confidence, parse quality, support-pattern reliability, and independence corrections. A coordinator receives only policy-permitted evidence from this belief state with controlled communication. Experiments on six reasoning benchmarks show that DarkForest achieves leading overall quality, improves the strongest baseline by up to 30.7\% on benchmark metrics, and reduces token consumption by up to $6.5\times$ compared with communication-heavy baselines.
1 Introduction
Multi-agent LLM systems can improve reasoning through interaction, but unrestricted communication risks propagating errors and increasing inference costs. DarkForest instead preserves independent evidence and coordinates through calibrated, policy-controlled summaries.
- Unrestricted information exchange can amplify incorrect outputs, undermine independent verification, and increase token consumption, latency, and inference cost.
- Figure 1 compares coordination accuracy with the availability of a correct independently generated candidate, exposing failures to select useful initial evidence.The candidate-availability upper bound is the share of examples where at least one independently queried agent is correct.
- DarkForest frames coordination as controlled exposure of compact, policy-permitted, and verifiable evidence rather than unrestricted text sharing.
- DarkForest independently generates candidate answers, constructs a calibrated belief state, and discloses selected evidence to a coordinator.Its belief construction combines parsing, clustering, reliability calibration, confidence weighting, support-pattern modeling, and independence correction.
- The paper evaluates DarkForest across six reasoning benchmarks spanning mathematics, code generation, general knowledge, scientific question answering, finance, and law.
2 Background
Multi-agent LLM collaboration queries multiple language-model agents and combines their outputs, using interaction patterns such as debate, discussion, role-based workflows, graph message passing, and answer aggregation.
- Multi-agent LLM systems combine outputs from multiple agents solving the same task or different subtasks.
- Existing collaboration methods include debate, round-table discussion, role-based workflows, graph-structured message passing, and aggregation over sampled answers.
3 DarkForest Design
DarkForest keeps candidate generation independent, converts outputs into structured and clustered hypotheses, calibrates belief over them, and exposes controlled evidence to a coordinator. Its design also includes offline calibration, uncertainty handling, and a narrow coordinator interface.
- 3.1 Overall Architecture: DarkForest uses independent candidate generation, calibrated aggregation, and controlled communication as its three core design principles.
- 3.1 Overall Architecture: The pipeline parses independent outputs into structured observations, clusters compatible candidates, estimates belief over clusters, and exposes a controlled summary to a coordinator.
- 3.1 Overall Architecture: Agents do not observe one another’s outputs before answering, preserving runtime independence without assuming statistical independence across agents.
- 3.2 Parsing and Canonicalization: Each parsed observation records a candidate, canonical representation, confidence, parse validity, and parse-quality metadata; invalid observations are excluded from clustering.
- 3.3 Candidate Clustering: Candidate clusters represent canonical answers together with their supporting-agent sets and support patterns, turning heterogeneous outputs into competing hypotheses.
- 3.4 Calibrated Belief: Cluster evidence combines agent reliability, support-pattern reliability, parse quality, confidence, and independence corrections before normalization into a belief distribution.The independence correction discounts correlated supporting agents, while confidence contributes through a bounded multiplier.
- 3.4 Calibrated Belief: Reliability terms are estimated offline from examples with known outcomes, and the resulting orchestrator parameters remain frozen during evaluation.
- 3.4 Calibrated Belief: Uncertainty based on posterior concentration and separation controls how the belief summary is presented, encouraging independent auditing when the state is diffuse.
4 Evaluation
Across six reasoning benchmarks, DarkForest delivers strong task quality while substantially reducing token use through independent generation, calibrated belief construction, controlled disclosure, and a guarded final decision.
- Experimental setup: DarkForest is evaluated across six benchmarks spanning mathematical, code, general, scientific, financial, and legal reasoning.Metrics include exact match, Pass@1, accuracy, execution accuracy, and program accuracy.
- Overall comparison: DarkForest achieves the strongest or near-strongest quality across the benchmark suite while using substantially fewer tokens than communication-heavy baselines.The comparison covers both task quality and average token consumption.
- Quality results: 2.52–5.00 absolute points, or 4.5%–30.7% relative improvement, separate DarkForest from the second strongest baseline across reported comparisons.On MATH, exact match reaches 76.80%, while MMLU-Pro reaches 58.38%.
- Quality results: HumanEval is the only benchmark where DarkForest is not among the top standalone methods, matching Mixture-of-Agent at 84.00% and trailing Graph-of-Agent (Max) by 2.00 points.It remains competitive with only 1.5k tokens per sample versus 8.5k and 7.2k for the two Graph-of-Agent variants.
- Efficiency results: 13.8k to 4.7k tokens on MATH and 24.6k to 5.9k on MMLU-Pro show major reductions versus Graph-of-Agent (Mean).Usage also falls from 13.0k to 3.2k on GPQA and from 8.7k to 1.9k on LegalBench.
- Decision ablation: Removing the guardrail lowers MATH accuracy from 76.80% to 75.40% and LegalBench accuracy from 68.00% to 65.60%, without changing token cost.The no-coordinator variant is cheaper, using 2.1k tokens on MATH and 1.2k on LegalBench.
5 Related Work
Related multi-agent methods differ in how agents exchange and combine information, with debate-based approaches exposing reasoning traces and aggregation-based approaches combining candidates.
- Coordination patterns: Debate-based methods exchange full reasoning traces across multiple rounds, whereas aggregation-based methods combine independently produced candidates.These interaction patterns define how information crosses agent boundaries.
6 Conclusion
DarkForest coordinates multi-agent reasoning by preserving independent signals, calibrating evidence over candidate clusters, and controlling what reaches the coordinator. Its guardrail protects strongly supported candidates from being discarded, while the framework treats coordination as belief-based decision-making under incomplete information.
- Framework overview: DarkForest keeps candidate generation independent, converts responses into structured observations, builds a calibrated belief state, and exposes only policy-permitted evidence.The observations include canonical candidates, validity, confidence, and parse-quality metadata; the belief state incorporates reliability, support patterns, confidence, parse quality, and dependence corrections.
- Incomplete information: Each structured observation represents a private signal about the unknown correct answer, while agent reliability and dependence remain latent to the coordinator.The system does not directly know whether a candidate is correct, whether an agent is reliable on the instance, or whether agreement reflects independent evidence.
- Calibrated aggregation: Candidate clusters represent hypotheses that the unknown answer equals a canonical candidate, rather than simple counts of supporting agents.DarkForest weights support using calibrated evidence terms, including agent reliability, confidence modulation, support-pattern reliability, and corrections for correlated contributions.
- Controlled communication: The disclosure operator sends compact evidence such as candidates, support patterns, posterior mass, margins, and uncertainty indicators instead of full reasoning traces unless explicitly allowed.This implements a conservative policy that preserves independent generation, aggregates signals into a calibrated belief state, and limits cross-agent information.
- Final decision: A deterministic guardrail overrides the coordinator only when the calibrated belief state provides strong conflicting evidence for a trusted candidate.The coordinator remains a decision rule under incomplete information rather than an oracle, and the belief state is treated as a prior over candidates rather than proof.
B.1 Backbones
The evaluation uses multiple open-source instruction-tuned backbones, including general-purpose, code-specialized, and mathematically oriented models. Models are served with fixed hardware and controlled decoding settings.
- Backbone selection: DarkForest evaluates multiple open-source instruction-tuned language models spanning general-purpose instruction following and domain-specialized tasks.The listed general-domain backbones are Qwen2.5-7B-Instruct, Qwen2.5-Coder-7B-Instruct, and Mathstral-7B-v0.1.
- Backbone roles: Qwen2.5-Coder-7B-Instruct supplies stronger code-generation ability, while Mathstral-7B-v0.1 provides a mathematically oriented agent.Qwen2.5-7B-Instruct serves as the general instruction-following backbone.
- Backbone inventory: Table 3 lists the backbone models and their Hugging-Face identifiers.The table provides the model inventory used in DarkForest.
- Serving: All backbones are served through vLLM with bf16 weights, tensor-parallel size 1, and one H100 GPU per model.Servers expose OpenAI-compatible local chat endpoints.
B.2 Benchmarks
DarkForest is evaluated across six reasoning benchmarks covering mathematical, coding, scientific, financial, and legal tasks. The experiments use benchmark-specific metrics, splits, agent compositions, and matched baseline settings.
- Benchmark scope: The six-benchmark suite spans mathematical reasoning, code generation, broad scientific knowledge, graduate-level reasoning, financial question answering, and legal reasoning.The suite is designed to test controlled communication across general-domain and professional-domain multi-agent reasoning.
- Benchmark rationale: GPQA targets high-uncertainty graduate-level science questions where naive majority voting may fail under shared misconceptions.This setting directly tests calibrated evidence rather than unweighted agreement.
- Experimental configuration: Table 4 specifies the per-benchmark three-agent composition, with all agents queried independently before DarkForest constructs its calibrated belief state.Table 5 summarizes splits, calibration sizes, evaluation sizes, and primary metrics.
- Baselines: Baseline comparisons use the same benchmark-specific backbone pool and model decoding settings as DarkForest whenever applicable.The compared methods include self-consistency, Graph-of-Agents, and Mixture-of-Agents.
C Detailed Related Work
DarkForest is positioned among multi-agent systems that extend language models through tools, planning, memory, and communication. Its distinct focus is calibrating and controlling evidence disclosure after independent candidate generation.
- Broader agent research: Prior agent research includes tool use, planning, memory, role specialization, and inter-agent communication for extending language models beyond single-turn generation.These approaches differ in how information is represented, routed, and exposed during decision-making.
- Research focus: The paper asks what evidence should reach a coordinator when several agents independently answer the same test-time question.It also asks how that evidence should be calibrated before disclosure.
- Aggregation methods: Aggregation-based methods combine independently produced candidates, but simple voting treats samples as exchangeable and weights every agreement equally.DarkForest instead weights agent support by calibrated reliability, support-pattern reliability, and independence corrections.
- Mixture-of-Agents: Mixture-of-Agents synthesizes raw candidate text in language space, whereas DarkForest exposes structured candidates, support patterns, posterior mass, and uncertainty indicators.The structured interface is intended to prevent the coordinator from inheriting hallucinated or confidently incorrect proposer content.
- Distinctive positioning: DarkForest differs from refinement methods by aggregating evidence from independent agents under a controlled disclosure policy rather than refining one output in place.The supplied ablation material identifies component-level isolation in the final decision procedure, while Figure 5 concerns token consumption for voting ablation.
D.1 Voting ablation
DarkForest outperforms majority and reliability-weighted voting by using structured belief evidence and a coordinator, with only a modest token increase.
- Voting comparison: 58.38% accuracy on MMLU-Pro and 68.00% on LegalBench make Full DarkForest stronger than both voting baselines.Majority Vote scores 53.00% and 65.00%, while Weighted Vote scores 56.38% and 66.60%, respectively.
- Voting comparison: Reliability calibration helps, but DarkForest gains further by modeling support patterns, uncertainty, and coordinator verification.
- Communication cost: Controlled disclosure, rather than unrestricted multi-round communication, produces the quality gains over voting.
- Communication cost: 5.1k to 5.9k tokens on MMLU-Pro and 1.2k to 1.9k on LegalBench quantify DarkForest’s modest additional cost.
D.2 Coordinator and Guardrail
A coordinator and deterministic guardrail complement calibrated belief aggregation, while compact evidence supports quality more efficiently than longer reasoning traces.
- Decision mechanism: Removing the coordinator lowers LegalBench accuracy from 68.00% to 67.20%, while leaving MATH unchanged.The coordinator is most useful when candidate labels are heterogeneous or evidence checking benefits the task.
- Decision mechanism: Removing the guardrail lowers MATH from 76.80% to 75.40% and LegalBench from 68.00% to 65.60%.The guardrail protects against coordinator outputs conflicting with strongly supported belief clusters.
- Decision mechanism: The guardrail improves quality without increasing token consumption because it is deterministic post-processing.Full DarkForest uses the same token budget as the no-guardrail variant.
- Disclosure policy: Belief summary and reasoning summary both reach 40.00% GPQA accuracy, but belief summary uses 3435.5 rather than 4136.0 tokens per sample.Full raw traces cost 5004.5 tokens per sample and reduce accuracy to 36.67%.
- Calibration: 76.8% full calibration on MATH exceeds uniform weighting’s 71.2%, while LegalBench rises from 65.2% to 67.2%.Agent-level calibration reaches 76.8% on MATH and 66.6% on LegalBench.
- Scalability: Adding independent backbones raises FinQA execution accuracy from 15.67% with three to 19.33% with five.Program accuracy rises from 11.33% to 13.67%, while token usage grows from 4.5k to 5.4k per sample.
D.7 Coordinator Robustness
Coordinator robustness is tested by replacing only the coordinator while fixing the initial traces, calibrated belief state, disclosure policy, and guardrail.
- Coordinator robustness: 17.67% execution accuracy and 12.33% program accuracy make Qwen-Coder stronger than the default coordinator on FinQA.
- Coordinator robustness: Mathstral remains competitive at 16.00% execution accuracy and 11.67% program accuracy, while Finance-Llama and Saul perform worse.
- Coordinator robustness: Several coordinators use the same calibrated belief state effectively, although weaker coordinators degrade quality.The controlled-disclosure procedure remains unchanged across coordinator choices.