Source-linked AI summary
Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards
Keru Chen, Sen Lin, Yingbin Liang, Nathaniel D. Bastian, Shaofeng Zou
TL;DR
LLM agents can have enough team-level evidence yet fail to communicate or use it, and training signals often do not identify the value of individual messages. CoSIR uses verifiable information rewards in a rating-table task to train collaboration behaviors; across three backbones, it improves scores on seven unseen collaboration tasks and achieves the highest aggregate score among compared methods.
Problem
LLM teams may withhold useful findings, fail to request clarification, or ignore teammates’ input, while post-training objectives do not explicitly measure individual messages’ contributions to team decisions.
Source: §1 · p. 1Method
CoSIR trains agents in a verifiable rating-table environment, rewarding shared information by its reduction in team surprisal, requests by potential reduction after a truthful reply, and use of information in private answers.
Source: §1 · p. 2, §3.3 · p. 5Results
Takeaways & Limitations
Behavioral analyses show increased sharing of private information and better use of teammates’ input, while general capabilities remain close to starting checkpoints.
Source: §1 · p. 2, §5 · p. 9The rewards depend on a toy environment with known facts and a computable evaluator; open-ended tasks would require assessing information value without access to the complete state.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Effective collaboration among large language model (LLM) agents in multi-agent systems requires sharing relevant evidence, requesting missing information, and incorporating teammates' contributions into decisions. Training these behaviors is challenging because final task outcomes provide limited feedback on the value of individual messages. We introduce CoSIR (Collaboration via Surprisal-based Information Rewards), a post-training approach that uses a simple, verifiable environment to teach collaboration behaviors that generalize to more complex, unseen tasks. We propose a Bayesian evaluator that scores shared information by its reduction in team surprisal, defined as the sum of the negative log probabilities assigned to the correct option given each agent's available evidence. Requests are scored by the potential reduction in the same quantity if the requested information were truthfully provided. Separate rewards encourage agents to use the received information in their private choices and calculations. We post-train Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it through self-play and evaluate transfer to seven unseen natural-language collaboration tasks. CoSIR achieves the highest aggregate score, while its performance in general tasks remains competitive. Behavioral analyses show increased disclosure of private evidence and fewer judged failures to use teammates' input.
1 Introduction
LLM teams can possess enough evidence yet fail to share, request, or use it, while ordinary post-training objectives do not directly reward the value of individual messages. CoSIR trains these behaviors in a verifiable rating-table environment and evaluates transfer to unseen natural-language tasks.
1 Introduction
- Distributed evidence does not guarantee successful collaboration: teams can underperform a single agent with the same evidence and may withhold findings, skip clarification, or ignore teammates.Prior work also reports that agents may abandon correct answers in favor of an incorrect majority.Source: §1 · p. 1
- Post-training objectives for instruction following and multi-turn completion do not explicitly distinguish how individual messages contribute to a team decision.Open-ended tasks make message value difficult to verify because relevant facts, their distribution, and their contributions are not readily known.Source: §1 · p. 1
- CoSIR trains sharing, requesting, and integrating evidence in simplified, verifiable environments that preserve information dependencies from more complex tasks.The goal is to learn transferable behaviors and evaluate them on unseen natural-language tasks.Source: §1 · p. 2
- Agents jointly select the highest-total-rating option from a table whose entries are distributed across agents, making sharing known ratings and requesting missing ones useful.Specific table entries make exchanged information identifiable, reports verifiable, and each agent’s evidence trackable.Source: §1 · p. 2
- A Bayesian evaluator scores shared ratings by their reduction in team surprisal and requests by the potential reduction if a truthful answer were provided.Separate private-answer rewards encourage agents to use received information in option choices and calculated totals.Source: §1 · p. 2
- Across three backbones, CoSIR improves all seven unseen collaboration-task scores over the starting checkpoint and achieves the highest sum among compared methods.Behavioral analyses also report increased sharing of private information and better use of teammates’ input.Source: §1 · pp. 2–3
2 Related Work
Prior work addresses multi-agent interaction through inference-time workflows and collaboration-focused training, while synthetic environments have supported transfer beyond their training settings. CoSIR instead targets collaboration behaviors using explicit rewards for information value.
LLM Multi-Agent Collaboration
- Inference-time approaches guide interactions among existing models: Theory-of-Mind methods reason about teammates’ beliefs, while TalkHier organizes communication and answer refinement [Li et al., 2023; Wang et al., 2025a].The paper compares CoSIR with these approaches under matched agent counts and public speaking-turn limits.Source: §2 · p. 3
Post-Training for LLM Collaboration
- Collaboration training includes supervised planning with an environment verifier [Peng et al., 2025], rewards for correct answers and discussion [Park et al., 2025], cooperative multi-agent reinforcement learning [Liu et al., 2026b], and actor-critic methods [Liu et al., 2026a].CoSIR makes information contributed by a statement or potential reply an explicit reward signal, calculated in its rating-table environment.Source: §2 · p. 3
Transfer from Synthetic Multi-Agent Environments
- Synthetic self-play has transferred to unseen games and reasoning tasks [Liu et al., 2025; Yuan et al., 2025], whereas CoSIR targets transferable collaboration behaviors using ratings to calculate the value of shared or requested facts.Source: §2 · p. 3
3 Method
CoSIR trains agents to share, request, and use information through team discussions in a rating-table environment. It assigns verifiable rewards to information exchange and private answers, then updates a shared policy through self-play.
3 Method
- CoSIR uses a simple rating-table environment to generate tasks, distribute information, manage discussions, score messages and private answers, and update the model from new discussions.Source: §3 · p. 3
3.1 A Task with Distributed Information
- Agents choose the highest-total option using a table of ratings, with each agent receiving only a subset of entries that may overlap with teammates’ information.Across five settings, tasks vary agent count, evidence distribution, shared ratings, knowledge of who holds missing information, and discussion rounds; agents exchange messages and provide private choices and totals.Source: §3.1 · pp. 3–4
3.2 Measuring the Value of Information
- A Bayesian evaluator estimates each option’s probability of being best from the agent’s available ratings, using a uniform prior over complete tables with a unique best option.It computes the posterior exactly by dynamic programming without using hidden ratings from the true table; team surprisal sums agents’ negative log probabilities for the correct option, and information value is its reduction.Source: §3.2 · p. 4
3.3 Rewards for Communication and Answers
- Sharing rewards measure reductions in team surprisal, but positive information rewards require ratings to be correct and supported by the speaker’s briefing or received reports.Supported ratings avoid negative information rewards even if they temporarily increase surprisal, while each rating incurs a small communication cost.Source: §3.3 · pp. 4–5
- Requests are scored by the potential surprisal reduction if a teammate truthfully provided a missing rating, while private-answer rewards assess option choice and reported totals using the agent’s available information.The request value is computed when asked, and private-answer scoring encourages using received information in both choices and calculations.Source: §3.3 · p. 5
3.4 Learning from Team Discussions
- Self-play uses the same model for all agents, samples multiple candidate responses per turn, scores them under shared conditions, and trains on candidates while one continues the discussion.Token-level statement rewards and group-relative request and answer advantages guide updates through a PPO variant; useful-request demonstrations can be added early in training, and a reference-model penalty helps preserve general capabilities.Source: §3.4 · p. 6
4 Experiments
The experiments test collaboration training on seven unseen tasks, alongside changes in communication and general capabilities. CoSIR leads aggregate comparisons across three backbones, with behavioral gains concentrated in sharing and using information.
4.2 Generalization Across Tasks and Models
- SUM margins over the strongest alternative are 19.79 points for Qwen3-8B, 22.95 for Qwen3-4B, and 24.86 for Gemma-3-4B-it; CoSIR ranks highest for all three.Table 1 defines SUM as the sum of seven task scores.Source: Table 1 · p. 7, §4.2 · p. 7
- Each trained model has a higher mean score than Base on all seven tasks, so the gains are not confined to one benchmark.The seven tasks span private-fact decisions, distributed-data computation, cross-document question answering, mathematical interaction, peer-influenced answers, puzzles, and coordination.Source: §4.1 · p. 6, §4.2 · p. 7
- Inference-time methods lead on some individual tasks, but both lower SUM than Base for Qwen3-4B and Gemma-3-4B-it, while CoSIR raises SUM for all three models.ToM leads on SILO-BENCH with Qwen3-8B, and TalkHier leads on BenchForm with all three models.Source: §4.2 · p. 7
- On SplitQA, CoSIR improves over Base by 2.50 points for Qwen3-8B, 0.36 for Qwen3-4B, and 8.21 for Gemma-3-4B-it.SplitQA requires combining facts from separate supporting paragraphs into a free-form answer; the Qwen3-4B improvement is small.Source: §4.2 · p. 7
- Against Outcome-RL, CoSIR exceeds SUM by 20.72 points for Qwen3-8B, 22.95 for Qwen3-4B, and 24.86 for Gemma-3-4B-it.CoSIR scores higher on all seven tasks with Qwen3-8B and six tasks with each other model; Outcome-RL wins on HiddenBench for Gemma-3-4B-it and ties on MathChat for Qwen3-4B.Source: §4.2 · p. 7
4.3 How Collaboration Changes
- On Qwen3-8B, first-message disclosure rises from 61.1% to 69.0% on HiddenBench and from 41.1% to 51.1% on SplitQA, measuring information shared early enough for later teammate use.Source: §4.3 · p. 8
- Coordination failures fall from 67.8% to 62.0% and verification or termination failures from 58.8% to 53.5%, while specification and system-design failures remain nearly unchanged.Repeated steps and information withholding also do not improve, indicating that the behavioral gains do not cover every failure type.Source: §4.3 · p. 8, Table 3 · p. 8
- Across GSM8K, MMLU, and IFEval, performance changes from Base range from −0.6 to +0.8 percentage points, indicating that general capabilities remain largely preserved.The study also presents paired cases illustrating revised recommendations, cross-document fact combination, and corrected calculations using shared data.Source: §4.3 · pp. 8–9
5 Conclusion
CoSIR trains LLMs to share, request, and use information with computable rewards in a rating-table environment. Across three backbones, trained models score higher than Base on all seven unseen collaboration tasks and achieve the highest aggregate scores among compared methods.
5 Conclusion
- CoSIR uses computable rewards to train sharing, requesting, and use of information in a rating-table environment.The conclusion describes transfer from that environment to unseen collaboration tasks.Source: §5 · p. 9
- Across three backbones, trained models achieve higher mean scores than Base on all seven unseen collaboration tasks and the highest aggregate scores among compared methods.Behavioral analyses and case studies show more sharing and use of teammates’ information.Source: §5 · p. 9
- General capabilities remain close to the starting checkpoints, while behavioral analyses show more sharing and use of teammates’ information.Source: §5 · p. 9
Limitations
CoSIR’s evaluation and reward design have important scope boundaries: its evaluator relies on a toy setting with known facts, and some collaboration failures remain frequent.
Limitations
- Inference-time workflows and Outcome-RL remain stronger on some model-task combinations, while repeated steps and other MAST failure categories remain frequent.The reward also depends on a toy environment with known facts and a computable evaluator; open-ended use would require assessing information value without the complete state.Source: Limitations · p. 10
C.1 Prior and Observations
- The Bayesian prior excludes family-specific instance-generation conditions, so its posterior is exact for the evaluation model, not the full training-instance distribution.Source: §C.1 · p. 14
C.2 Exact posterior computation
- A reduction in team surprisal need not increase every agent’s probability of the correct option or the LLM’s actual decision accuracy.Communication rewards use probabilities floored at 10^-4.Source: §C.2 · p. 14
D.1 Statement reward computation
- The statement reward’s nominal 0.7 delivery weight does not match every delivery condition, including guaranteed delivery to a routed-request recipient.The scoring weight and the state-update rule serve different purposes.Source: §D.1 · p. 15
D.2 Request value and remaining turns
- Request rewards assume the holder answers truthfully and every other present agent receives the rating, without estimating willingness to answer or revising the score after delivery.The score is assigned when the request is made.Source: §D.2 · p. 16
E Private-Answer Evaluation
Private answers are scored using an evidence-conditioned choice score and a check of totals computed from the agent’s available ratings; these checks do not establish the true table totals.
E Private-Answer Evaluation
- A parseable private choice receives the evaluator’s posterior probability for that option, floored at 10^-4; an unparseable choice receives zero.Source: E Private-Answer Evaluation · p. 16
- Reported totals are checked against totals computed from one retained rating per cell, using briefing values first and otherwise the latest received rating.A missing total line counts as incorrect.Source: E Private-Answer Evaluation · p. 16
- Matching the known-rating totals does not imply recovery of the full table’s true totals, and an incorrect received report can affect the probability estimate and totals target until corrected.Source: E Private-Answer Evaluation · pp. 16–17
F Self-Play Training Details
Appendix F details self-play collection, component-level credit assignment, optimization, and measures used to retain the initial models’ general behavior. Training uses shared settings across three backbones, with three collection rounds.
F.1 Collection and candidate groups
- Each collection round contains 60 games, and candidate messages are scored against the same conversation history before one policy message is randomly selected to continue discussion.Groups with no scoring differences are discarded; private answers are sampled separately and do not continue the dialogue.Source: §F.1 · p. 17
F.2 Component advantages and token spans
- Centered advantages assign credit to actions, choices, and option totals, while statement spans receive their own reward without group normalization.Nearly tied policy actions are treated as equally good using a 0.03 tolerance; total-line advantages compare correctness separately for each option and are scaled by 1/k.Source: §F.2 · p. 17
- Parsed text positions and attribute-word anchors assign token credit, but approximate span boundaries can place a rating token in a neighboring span.Request sentences and closing sentences receive action credit with a fixed total span weight of 16; private-answer choice and total lines receive their corresponding advantages.Source: §F.2 · p. 17
F.3 Generated demonstrations
- When requests are judged the best action, generated request demonstrations supplement sampled responses, but their clipped imitation update is not an unbiased on-policy PPO estimate.Demonstrations are not selected to advance discussion; they are generated using rewrites or request templates, and action advantages are clipped to [−0.5, 0.5].Source: §F.3 · pp. 17–18
F.4 Policy objective and optimization
- The policy objective uses clipped, advantage-weighted updates and a fixed token-sum divisor of 64, so longer statement spans can receive more total gradient weight.Optimization uses AdamW8bit, a gradient-norm limit of 1.0, and 64 samples per step; policy-sampled tokens with zero assigned advantage receive a reference penalty against the initial backbone.Source: §F.4 · p. 18, §F.5 · p. 18
F.5 Retaining the initial model’s behavior
- A reference penalty and general-prompt imitation help retain initial-model behavior; all backbones share reward definitions and information-tracking rules.The general prompts come from a cleaned SmolTalk subset, while the frozen parser uses the initial Qwen3-8B model; the learning rate is fixed within each of the three rounds.Source: §F.5 · p. 18, §F.6 · pp. 18–19, Table 6 · p. 19
F.7 Learning rates and hardware
- Three collection rounds train each backbone; each round takes about five hours, or about 15 hours per complete run.The hardware listed is two NVIDIA H100 GPUs and four Intel Gaudi 2 accelerators; the time estimate excludes evaluation and other runs.Source: §F.7 · p. 19
G SplitQA Construction and Evaluation
SplitQA tests whether teams can combine facts distributed across agents while filtering distractors. Its construction and scoring protocol make evidence sharing and exact-match team answers central to evaluation.
G SplitQA Construction and Evaluation
- The 70-question SplitQA set uses HotpotQA distractor questions: 63 bridge questions and 7 comparison questions, each with two distinct supporting paragraphs.Items whose gold answer appears verbatim in a distractor paragraph are excluded.Source: G SplitQA Construction and Evaluation · p. 19
- Two agents each receive one supporting paragraph, while two receive only distractors, requiring the team to identify evidence holders and link or compare their facts.No agent receives both annotated supporting paragraphs.Source: G SplitQA Construction and Evaluation · p. 19
- Four agents speak in fixed order for five rounds and then answer privately; the most-voted exact-match answer scores one, while incorrect answers and top-vote ties score zero.Each question is run with four seeds.Source: G SplitQA Construction and Evaluation · p. 19
H Evaluation Settings and Score Aggregation
The evaluation uses distinct protocols and score definitions across unseen collaboration tasks, then aggregates task means equally for SUM. General-capability results remain close to the starting checkpoints.
H Evaluation Settings and Score Aggregation
- HiddenBench uses five discussion rounds and majority voting, SILO-BENCH uses three rounds and task success rate, and SplitQA uses five rounds with team accuracy by exact match.Tied votes count as incorrect for SplitQA.Source: H Evaluation Settings and Score… · p. 19
- MathChat pairs an LLM with a tool-using user proxy, BenchForm averages accuracy across five peer-influence settings, and Collaboration Tax reports collaborating-agent scores.BenchForm’s trust and doubt settings include five rounds of prior discussion history.Source: H Evaluation Settings and Score… · p. 20
- SUM adds the seven task means after within-task averaging, gives every task equal weight, and excludes general-capability benchmarks and behavioral measurements.Outcome-RL replaces CoSIR’s behavior-specific information reward with final team-outcome feedback; ToM and TalkHier instead add inference-time reasoning or revision steps.Source: H Evaluation Settings and Score… · p. 20
I General Capability Retention
- Changes on GSM8K, MMLU, and IFEval all lie between −0.6 and +0.8 percentage points, indicating performance remains close to the starting checkpoints.These are the paper’s mathematical reasoning, general knowledge, and instruction-following benchmarks, respectively.Source: I General Capability Retention · p. 20
J Behavioral Measures
Behavioral measures quantify disclosure, engagement, evidence listing, supporting-fact sharing, and correction attempts. The case-study analysis also reports increases in evidence disclosure at the first message.
J Behavioral Measures
- HiddenBench measures private facts disclosed by original holders and messages responding to teammates, with response measurement not requiring correctness.A disclosure match requires at least 60% of a fact’s content words in the holder’s message.Source: J Behavioral Measures · p. 20
- SILO-BENCH tracks agents listing all their own data and result-only messages that give an accumulated conclusion without the speaker’s own data.The first measure reflects exposure of checkable inputs; the second decreases when fewer messages omit those inputs.Source: J Behavioral Measures · pp. 20–21
- SplitQA measures whether either of the two supporting-fact holders states their assigned fact, excluding agents given only distractors..Source: J Behavioral Measures · p. 21
- MathChat’s correction-language measure counts attempts to identify or correct an error, without separately scoring whether each correction is valid.The behavioral measures are reported as percentages, with denominators that differ by measure.Source: J Behavioral Measures · pp. 20–21
- Qwen3-8B’s first-message disclosure rate rises from 61.1% to 69.0% on HiddenBench and from 41.1% to 51.1% on SplitQA.These first-message rates are distinct from disclosure at any point in the discussion; earlier evidence enters while teammates can still use it in later turns.Source: J Behavioral Measures · p. 21
K MAST Failure Analysis
MAST judgments show decreases in some coordination and verification failures but little change in other categories. Paired cases illustrate how sharing private evidence can change a decision or make a counting error checkable.
K MAST Failure Analysis
- The MAST evaluation used the original judge prompt and 14 failure-mode definitions, changing only the judge model to gpt-5-mini.Conversations could receive multiple labels, so category percentages need not sum to 100%.Source: K MAST Failure Analysis · p. 21
- Coordination failures fell from 67.8% to 62.0%, and verification or termination failures from 58.8% to 53.5%.Specification and system-design failures remained nearly unchanged (88.2% and 88.3%), while information withholding changed from 18.5% to 18.7%.Source: K MAST Failure Analysis · p. 21
- Disclosure counts and MAST withholding judgments measure different aspects: whether designated private facts were shared versus a conversation-level failure label.Source: K MAST Failure Analysis · p. 22
L.1 HiddenBench: Sharing Facts That Change a Decision
- In HiddenBench, trained agents shared missing facts in round one, and Agent 1 revised its Central Depot recommendation to Hilltop Storage by round five.In Base, Agents 3 and 4 withheld their facts and all agents ultimately chose Bayview Warehouse.Source: §L.1 · p. 22, L Case Studies · p. 22
L.2 SILO-BENCH: Making a Wrong Count Checkable
- In Base SILO-BENCH, agents repeated counts without exposing their word lists, ending with an incorrect total of 18 instead of five.The four agents held two, one, two, and zero occurrences, respectively.Source: §L.2 · p. 22
- After a trained agent exposed its word list despite a wrong count, a teammate corrected its contribution and all agents reached five.The corrected calculation was 2 + 1 + 2 + 0 = 5.Source: §L.2 · p. 22
L.3 SplitQA: Connecting Facts Held by Different Agents
SplitQA tests whether agents can connect facts held in separate paragraphs to answer a question. In the trained example, agents supplied and combined the missing evidence, reaching the correct answer.
L.3 SplitQA: Connecting Facts Held by Different Agents
- The question required identifying the Eminem album featuring vocals from Rihanna, whose album Unapologetic was named in another agent’s paragraph.Source: §L.3 · p. 22
- After training, Agent 2 supplied the missing Rihanna link, and Agent 1 combined both facts; all agents answered The Marshall Mathers LP 2.Source: §L.3 · p. 22, Figure 4 · p. 23
- In Base, Agent 2 asserted an unsupported connection to Recovery, which became the team’s final answer.Source: §L.3 · p. 22