PaperlayerView exact revision on arXiv

Source-linked AI summary

Consilience: Conformally Calibrated Communication Control for Hidden-Profile Multi-Agent Reasoning

Abhijith Babu, Ramneet Kaur, Vishal Pramanik, Olivera Kotevska, Nathaniel D. Bastian, Susmit Jha, Sunny Raj, Yanzhao Wu, Sumit Kumar Jha, Anirban Roy

arXiv:2608.20564v1cs.AI

TL;DR

Hidden-profile multi-agent systems need communication control because agents hold complementary evidence and existing protocols lack guarantees that selected actions are appropriate. Consilience summarizes discussion state, adaptively selects and routes interventions, and conformally screens them using calibrated one-step regret. Across two benchmarks and 12 language models, it improves over uncontrolled discussion and often matches or exceeds full-information baselines, while the evaluation remains focused on hidden-profile tasks.

  • Problem

    Hidden-profile reasoning requires agents to integrate distributed private evidence, but existing discussion protocols provide no statistical guarantee that each selected communication action is appropriate.

  • Method

    Consilience uses an observable discussion state to propose, conformally certify, and route communication interventions, with calibration based on round-specific one-step regret.

  • Results

    Across two benchmarks and 12 language models, Consilience consistently improves over uncontrolled discussion and often matches or exceeds full-information baselines.

  • Takeaways & Limitations

    The results indicate that structured communication can be as important as information access itself for hidden-profile multi-agent reasoning.

  • Takeaways & Limitations

    The evaluation focuses on hidden-profile tasks, and calibration uses only trajectories that reach each round.

Abstract

from arXiv · show

Multi-agent LLM systems can improve reasoning by pooling diverse perspectives, but their effectiveness depends on coordinating communication, particularly in hidden-profile settings where each agent holds only part of the evidence required for a correct decision. Existing protocols, including fixed schedules, round-robin exchange, and unstructured debate, provide no guarantee that a conversational action is appropriate. We propose Consilience, an inference-time orchestration framework that both steers and certifies multi-agent communication under distributed private information. At each turn, Consilience summarizes the discussion using a compact state capturing uncertainty, disagreement, evidence gain, redundancy, and premature consensus, then selects both a communication intervention (challenge, clarify, seek evidence, or route) and an appropriate speaker. Its central contribution is a round-wise conformal calibration procedure that provides a distribution-free, finite-sample guarantee: at each discussion round, conditional on reaching that round, the one-step regret of a controller's proposed action is bounded by a calibrated threshold with marginal probability at least 1 - alpha; an acceptance mechanism enforces the same guarantee for the executed action by replacing inadmissible proposals. On HiddenBench-style hidden-profile tasks spanning 12 open and closed weight language models, Consilience improves decision accuracy and communication efficiency over fixed and unstructured discussion protocols, sometimes surpassing a full-information baseline where every agent observes all evidence. These results demonstrate that certified adaptive communication control can be more valuable than increasing information availability, providing a practical mechanism for reliable multi-agent LLM coordination.

1 Introduction

Multi-agent reasoning can pool complementary perspectives, but hidden-profile tasks expose failures in evidence integration and unreliable communication control. Consilience addresses this gap with adaptive, conformally certified interventions and broad evaluation across models and benchmarks.

  • Multi-agent deliberation can duplicate single-agent capabilities, amplify correlated errors, or prematurely reinforce incorrect conclusions.
  • Hidden-profile decisions distribute task-relevant evidence across agents, requiring communication to integrate complementary private knowledge.
  • Common-information bias causes groups to discuss and reinforce shared information more than unique information, limiting correction of initial preferences.
  • Fixed round-robin protocols cannot adapt to evolving discussion states, while unconstrained orchestrators may make unreliable control decisions.
  • Consilience treats hidden-profile reasoning as sequential communication control, selecting, certifying, and routing interventions from the evolving discussion state.
  • The evaluation spans 12 open- and closed-weight language models and compares hidden-information, round-robin, and full-information baselines while analyzing efficiency and coordination.

2 Problem Formulation

Consilience formalizes hidden-profile tasks as collective decisions where agents hold asymmetric private clues and communicate through a shared transcript. A closed-loop controller selects interventions and a router selects speakers, after which agents vote collectively.

  • Hidden-profile tasks distribute the evidence needed for the correct answer across multiple agents.
  • Each agent observes shared task information and one private clue, but not the private clues held by other agents.
  • Agents communicate through a public transcript, making private evidence available to the group when an agent shares it.
  • Consilience uses closed-loop control: a controller chooses the intervention, an action-conditional router chooses the speaker, and messages update the transcript and beliefs.
  • The controller and router do not answer the task or generate evidence; the final answer comes from the agents’ quorum-based beliefs.

3 Consilience: Conformally Calibrated Communication for Multi-Agent Systems

Consilience converts discussion into an observable state, proposes and screens communication actions with one-step regret, then routes accepted actions to speakers. Round-wise conformal calibration provides finite-sample coverage for controller regret under round-specific calibration.

  • 3.1.1 Stage 1: Belief Elicitation and State Construction: The method represents discussion with an observable state and scalar loss, then uses that state to propose the next intervention.The loss combines uncertainty, disagreement, redundancy, evidence gain, and premature-consensus signals; lower values are preferred.
  • 3.1.2 Stage 2: Action Proposal: The four communication interventions are Challenge, Clarify, SeekEvidence, and Route; controllers that support adaptive termination additionally include Vote.
  • 3.1.2 Stage 2: Action Proposal: Challenge tests leading answers, Clarify elicits divergent interpretations, SeekEvidence requests unshared decision-relevant facts, and Route solicits concise facts from under-participating agents.
  • 3.1.5 Termination and Final Decision: Deliberation ends through Vote, early stopping, or the maximum round count, followed by independent agent answers and confidence reports.
  • 3.1.3 Stage 3: Counterfactual Evaluation and Conformal Acceptance: Controllers propose actions from the state, while conformal acceptance evaluates proposals against admissible alternatives and can replace inadmissible actions.
  • 3.1.3 Stage 3: Counterfactual Evaluation and Conformal Acceptance: One-step regret compares an action’s counterfactual next-state improvement with the best admissible action, using independently evaluated branches from the same pre-action state.
  • 3.2 End-to-End Procedure and Theoretical Guarantee: Round-wise calibration uses only trajectories reaching the relevant round, and Proposition 1 gives finite-sample marginal regret coverage for a fixed controller.
  • 3.1.4 Stage 4: Action-Conditioned Routing and Transcript Update: Routing selects disagreeing agents for evidence or interpretation and least-heard agents as proxies for unshared evidence, with fixed-order tie-breaking.

4 Experiments

The experiments test adaptive communication against no-discussion, fixed round-robin, and full-information references on hidden-profile benchmarks. They measure both individual voting consistency and collective task correctness across models.

  • The evaluation asks whether adaptive communication improves accuracy, matches full-information reasoning, and generalizes across controllers, actions, routing, termination, tasks, and agent groups.
  • Consilience is evaluated on 65 HiddenBench decision problems and a separately generated GroupTravelBench-structured benchmark.The generated benchmark is not a reproduction of the unavailable original data, so its results are not directly comparable.
  • Four Consilience controllers—Rules, MLP, LLM, and LLM+Vote—are compared with Hidden-Pre, Hidden-Post, and Full-Info references.Full-Info supplies all evidence without discussion and is an empirical reference rather than a theoretical upper bound.
  • Table 2 reports task accuracy and vote accuracy averaged across 12 models on the generated GroupTravelBench-structured benchmark.
  • Task accuracy measures whether the plurality-based collective decision is correct, while vote accuracy measures the proportion of individual votes matching the ground truth.Task accuracy is the primary metric; vote accuracy is complementary.

5 Results and Discussion

Consilience improves hidden-information deliberation by adapting communication actions and speaker selection to the evolving discussion state. Across model evaluations and ablations, adaptive coordination improves accuracy, evidence integration, and discussion quality.

  • Main results: 0.83 average accuracy versus 0.26 for fixed round-robin discussion, an absolute improvement of 0.57.The strongest controller’s gain supports adapting communication to the collective state rather than merely enabling communication.
  • Main results: The best Consilience policy reaches or exceeds the full-information condition on 11 of 12 models and improves average accuracy by 0.13.This comparison evaluates controlled hidden-information deliberation against an information-rich no-discussion baseline.
  • Generalization: The unseen-task MLP achieves 0.73 average accuracy versus 0.77 for the all-task model, indicating that most learned coordination behavior transfers to unseen discussion states.The held-out evaluation trains on five-task subsets and tests on remaining tasks.
  • Conformal action sets: Randomly sampled actions from conformal prediction sets maintain high task success across controller variants, indicating that the calibrated sets generally contain multiple effective actions.The ablation samples an action uniformly from each conformal prediction set rather than executing the controller’s preferred action.
  • Speaker routing: Action-conditional routing yields the largest mean objective improvement while maintaining low regret, whereas generic routing policies produce negative mean improvements.Matching disagreement-oriented actions to dissenting agents and evidence-seeking actions to less-participating agents produces more informative messages.
  • Communication actions: Removing Route causes the largest average accuracy drop, while removing SeekNewEvidence also substantially harms performance, especially for LLM controllers.Challenge has a modest effect, while ClarifyDisagreement has the smallest effect and can slightly reduce accuracy.
  • Controller state: The complete controller state achieves the highest task accuracy and lowest discussion objective, with information, belief, and cost features providing complementary signals.Removing communication cost reduces accuracy and produces the worst objective, while belief-only features shorten discussions but lower accuracy.

6 Conclusion

The paper introduces Consilience as a closed-loop framework for selecting, certifying, and routing communication in hidden-profile multi-agent reasoning. Across two benchmarks and 12 language models, it improves over uncontrolled discussion and often matches or exceeds full-information baselines.

  • Consilience is a closed-loop framework for adaptively selecting, certifying, and routing communication in hidden-profile multi-agent reasoning.
  • Across two benchmarks and 12 language models, Consilience consistently improves over uncontrolled discussion and often matches or exceeds full-information baselines.The evaluation focuses on hidden-profile tasks, with broader collaborative reasoning settings left for future work.

Appendix

The appendix supplements the paper with a worked hidden-profile example, formal guarantee materials, expanded experiments, ablations, and prompt templates. Figure 2 illustrates how adaptive interventions and speaker selection recover complementary private evidence that round-robin discussion misses.

  • The supplementary material includes a worked hidden-profile example, the interpretation and proof of Proposition 1, expanded experiments, ablations, and complete prompt templates.
  • In Figure 2, round-robin discussion converges on Apartment A despite private clues identifying Apartment B, whereas Consilience surfaces unshared evidence and recovers B.Shared evidence initially favors A, while complementary private clues collectively support B.

A Interpretation, Scope, and Proof of Proposition 1

Proposition 1 provides round-specific marginal coverage for a controller’s one-step regret, while the acceptance rule deterministically ensures the executed action meets the calibrated threshold. The guarantee remains conditional on round-specific exchangeability and does not establish global correctness or trajectory-level coverage.

  • With probability at least 1 − α, the controller’s proposal belongs to the round-specific conformal action set.
  • Coverage is marginal over calibration and test trajectories, conditional on reaching round t and on the realized number of calibration trajectories reaching that round.
  • The executed-action property requires evaluating all admissible counterfactual branches and committing to the evaluated speaker message and belief updates.Resampling after acceptance can make the realized next state differ from the evaluated counterfactual state.
  • The acceptance mechanism turns the proposal guarantee into a deterministic property of the executed action by accepting admissible proposals or selecting a conformal fallback.The fallback set is nonempty because at least one admissible action minimizes realized counterfactual next-state loss.
  • The proposition does not identify a unique ground-truth action, guarantee improved answer correctness, establish global policy optimality, or guarantee a correct final answer.It also does not provide simultaneous coverage over all rounds; trajectory-level guarantees require an additional calibration argument.
  • The guarantee requires round-specific exchangeability under a fixed policy-induced trajectory distribution and independently fixed score-generating components.Later-round exchangeability does not follow automatically from exchangeability at the initial round when earlier replacements alter later-state distributions.

B Experimental Details

The evaluation spans 13 instruction-tuned open-weight models across multiple families and scales, with supplementary closed-model experiments comparing full information and adaptive orchestration. Controller variants either use the evaluated language model or replace controller decisions with a lightweight MLP.

  • 13 instruction-tuned open-weight language models span multiple families and scales in the Consilience evaluation.The models include seven Qwen3 sizes, Qwen2.5-32B, Mistral-24B, Phi-4, Llama-3.1-8B, Llama-3.3-70B, and Gemma-3-27B.
  • Supplementary experiments include GPT-4.1-mini and Claude-3-Haiku to compare a full-information baseline with LLM-based orchestration and adaptive voting.These experiments assess whether observed trends generalize beyond open-weight models.
  • LLM-based controllers use the same model for communication-action selection, while learned-controller variants use a lightweight MLP and retain the evaluated LLM for other reasoning components.

C Non-Conformal Results

Consilience improves hidden-profile decision accuracy across benchmarks, but the strongest controller depends on benchmark and discussion dynamics. Adaptive control can outperform full-information reasoning and reduce cross-benchmark degradation relative to other controllers.

  • HiddenBench: 0.843 average task accuracy makes Rules the strongest Consilience controller on HiddenBench, exceeding full-information reasoning at 0.731.LLM, LLM+Vote, and MLP achieve 0.816, 0.815, and 0.785, respectively.
  • HiddenBench: 0.938 versus 0.877 on openai’s gpt models and 0.862 versus 0.646 on Claude-3-Haiku show LLM+Vote matching or exceeding full-information reasoning.These supplementary frontier-model results reproduce the main trend.
  • Generated benchmark: 0.668 average task accuracy makes LLM+Vote the strongest controlled method on the generated benchmark, improving 0.170 over uncontrolled discussion.Rules, LLM, and MLP reach 0.654, 0.615, and 0.507; adaptive termination is especially useful when continued discussion may reinforce incorrect answers.
  • Model dependence: Controller performance depends strongly on participating-model capability, with Qwen2.5-32B and Llama-3.3-70B exceeding 0.88 under the strongest generated-benchmark controllers.The MLP is less consistent than Rules and LLM-based controllers for some models.
  • Cross-benchmark comparison: 0.115 points is LLM+Vote’s cross-benchmark accuracy decrease, smaller than Rules’ 0.160, LLM’s 0.170, and MLP’s 0.261.Rules is particularly effective on HiddenBench, whereas transcript-aware LLM+Vote transfers more robustly to the generated benchmark.

C.1 Generalization to Held-Out Tasks

The held-out evaluation trains an MLP controller on five HiddenBench tasks and tests it on 60 unseen tasks using collective-state summaries. Performance transfers substantially to medium and large models, but transfer is uneven and the reported comparison has a task-set confound.

  • Evaluation protocol: The held-out protocol trains on five tasks and evaluates on a disjoint 60-task test set without fitting on test-task outcomes.An exhaustive action explorer supplies rewards for Challenge, ClarifyDisagreement, SeekNewEvidence, and Route during training.
  • Results: 0.73 average task accuracy for the held-out MLP versus 0.77 for the all-task MLP corresponds to an approximate 0.04 descriptive decrease.The held-out controller is trained only on the five-task subset.
  • Results: 8 of 13 evaluated models achieve at least 0.83 held-out task accuracy, including four models reaching 0.86.The strongest transfer occurs for medium and large language models.
  • Transfer boundary: Held-out transfer is less reliable for noisier or behaviorally distinct models, with task-accuracy decreases of 0.17 for Qwen3-1.7B and 0.16 for Phi-4.The collective-state-only representation can be difficult to interpret when belief distributions or discussion states differ from training trajectories.
  • Evaluation signal: Held-out and all-task task accuracies differ by approximately 0.02 mean absolute points in their task-versus-vote comparison.For Qwen3-8B, Qwen3-14B, and Llama-3.3-70B, the two values are identical.
  • Caveat: 0.03 and 0.04 point differences combine training effects with different evaluation task sets, so the reported gap is not a strictly controlled generalization estimate.Both checkpoints should be evaluated on the identical 60-task split to isolate the effect.

D Heterogeneous-Agent Analysis

Consilience can coordinate heterogeneous agent groups, but performance is sensitive to model composition, agent assignment, and controller choice. Private-evidence placement across positions materially changes outcomes.

  • Heterogeneous performance: 0.966 is the rule-based controller’s highest task accuracy on both Position Rot-4 and Ladder L2 in heterogeneous four-agent groups.Overall heterogeneous performance can be comparable to strong homogeneous groups, but sensitivity is substantial.
  • Position assignment: 0.879 to 0.966 is the rule-based accuracy range across documented and rotated position assignments, while LLM+Voting varies from 0.828 to 0.931.The same models can perform differently when assigned different private evidence.
  • Group composition: Groups composed primarily of capable models remain strong across several controllers, whereas groups containing several small models perform substantially worse.Reliable heterogeneous performance depends on both group capability distribution and communication routing.

E Ablations

The ablations show that Consilience’s performance depends on combining informative state features, complementary communication actions, and effective speaker routing. Controller variants also reveal a trade-off between discussion length and task success, while heterogeneous groups remain sensitive to composition and controller choice.

  • State-feature ablation: 82.22% task success and a 0.0239 final objective make the complete state the best overall feature trade-off.The information-only controller matches success but has higher communication cost and a substantially worse final objective.
  • State-feature ablation: 75.56% success makes belief-only control the weakest reduced representation, while belief-plus-information and belief-plus-cost reach 77.78%.Belief-only control has the lowest communication cost, but success falls 6.66 percentage points relative to the full model.
  • Action ablation: 4.6 percentage points is the largest average accuracy loss from removing routing, followed by 3.8 points when removing SeekNewEvidence.Challenge produces a small consistent degradation, whereas ClarifyDisagreement has no average effect and action effects vary by controller.
  • Speaker-routing ablation: ΔJ = −0.0079 gives action-conditional routing the best mean objective change, versus −0.0017 for round-robin and 0.0145 for random routing.Round-robin has the lowest normalized regret at 0.4839, narrowly below action-conditional routing at 0.4868.
  • Controller comparison: 82.75% is the highest mean success rate for both LLM and MLP controllers, while the rule-based controller terminates after 3.24 rounds on average.The early-terminating variants illustrate a trade-off between shorter discussions and lower task success; one variant falls to 64.50%.
Loading 2608.20564v1…