Don't lie to your friends: Learning what you know from collaborative self-play

• Language models often lack collaboration skills, including clarification, faithful uncertainty, and correct tool use; supervised fine-tuning lacks a social, goal-oriented signal.The cited work reports fewer grounding actions than humans (Shaikh et al., 2024), unfaithful uncertainty (Zhou et al., 2024), and incorrect tool use (Yoran et al., 2024; Wu et al., 2024).Source: §1 · p. 1
• CSP is a framework for teaching language-model agents to collaborate, using small societies with different retrieval tools to answer factoid questions.Success requires judging when parametric knowledge is reliable, expressing uncertainty, and using retrieval appropriately.Source: §1 · pp. 1–2
• CSP-trained agents improve selective searching and calibrated uncertainty expression on BioASQ and PopQA, with the advantage transferring to two unseen benchmarks.Training uses multi-agent rollouts selected for high reward, but evaluation is in a single-agent setup.Source: §1 · p. 2
• Training rewards task completion and effort, encouraging high accuracy without unnecessary tool calls, while leaving individual rollout actions unprescribed.Agents share parameters and differ by prompts and tools; the model is evaluated as a single agent at test time.Source: §1 · p. 2
• Game-theoretic analysis identifies conditions under which collaborative self-play can induce efficient tool use and calibrated question answering.The analysis provides guidance for designing collaborative self-play learning scenarios.Source: §1 · p. 2

Multi-agent LLM systems

• Unlike inference-time deliberation and dialogue-distilled reasoning work (Du et al., 2023; Chen et al., 2024), it uses collaboration during training to teach conversational skills rather than end-task reasoning.The authors focus on skills intended to improve collaboration with humans.Source: §2 · pp. 2–3

Social learning

• Unlike social-learning approaches that assign teachers, judges, or distillation targets (Mohtashami et al., 2023; Wang et al., 2024), its reward mechanism makes pro-social behavior necessary for high reward.No agent is designated as teacher or judge.Source: §2 · p. 3
• Its cooperative natural-language games train broadly useful single-agent conversational policies, unlike work aimed at effective multi-agent systems or game success (Foerster et al., 2016; Bard et al., 2020; Liao et al., 2024).The paper likens its cooperative setting to competitive games underlying GANs and adversarial domain adaptation (Ganin et al., 2016).Source: §2 · p. 3

Calibration and confidence

• It studies epistemic uncertainty about model parameters and tool outputs in retrieval-augmented models, learning answer-or-abstain pragmatics through self-play rather than stipulating the correct action.The choice depends on both the likelihood of answering correctly and the consequences of abstaining versus being wrong.Source: §2 · p. 3

Game-theoretic dialogue

• Its game-theoretic analysis characterizes when collaborative self-play induces rationally calibrated confidence, rather than explaining features of human conversation.The analysis focuses on equilibrium conditions for normatively desirable behavior.Source: §2 · p. 3

Social supervision

• Social supervision comes from interaction efficacy, with all agents sharing model parameters and a joint objective rather than receiving separate rewards and models.The shared parameters yield a single trained model for deployment after collaborative self-play.Source: §3 · pp. 3, 4, 6

Multi-agent orchestration

• A rollout proceeds through orchestrator-selected agents and prompts until a terminal state, turn limit, or effort budget is reached.The orchestrator can add tool results as evidence, route questions to other agents, and choose patterns such as broadcast communication or a hub.Source: §3 · p. 4

Actions and arguments

• Agents emit a constrained action and argument: #ANSWER, #ASK, #HEDGE, or an available tool action; #SEARCH adds near-neighbor retrievals to the prompt.The action-and-argument format is a deliberately constrained instantiation of tool use.Source: §3 · p. 4

Inter-agent communication and orchestration

• #ASK passes control to other agents, whose answers or hedges return to the asker; the rollout ends when the initial agent relinquishes control with an empty ask stack.The implemented broadcast setting scores task performance minus a penalty proportional to the number of search calls.Source: §3 · p. 5

Training

• ReST generates and scores rollouts, then trains iteratively on the best rollout compatible with a high-average-reward action sequence, avoiding reliance on lucky guesses.Action sequences are scored by the average reward of compatible rollouts and selected only when that average exceeds a threshold.Source: §3 · pp. 5–6

4 Analysis: An information-provision game

• The game rewards both players for a correct answer, charges δ > 0 for search, and has the asker choose the more confident answer for a searcher-guesser pair.Each player chooses S (search-then-answer) or G (guess-then-hedge); otherwise, the asker selects an answer at random. Search accuracy is αi and guess accuracy is β.Source: §4 · p. 6
• A unique pure-strategy equilibrium exists iff α1 > α2 + 2δ or α2 < β + δ; it is (S, G) when α1 > β + δ and (G, G) otherwise.If neither condition holds, (S, G) and (G, S) are both pure-strategy equilibria. The theorem assumes α1 > α > α2 and excludes {α1, α2} = β + δ and α1 = α2 + 2δ.Source: §4 · p. 6
• The (S, G) equilibrium is most relevant when a search tool is much more accurate than guessing on some questions and complementary enough to make the best tool clear.When α2(x) ≈ β(x), agents can learn to search precisely when it outperforms guessing; the analysis motivates designing a game that meets these criteria.Source: §4 · p. 6

5 Experimental setup

• Tool-equipped helpers search Wikipedia with BM25 or PubMed abstracts with Gecko embeddings, while the tool-less asker requests help and receives helper answers or hedges.The reward is effort-penalized F1, with a low per-tool-call penalty serving as a tie-breaker between rollouts with the same F1.Source: §5 · p. 7
• Training and in-distribution evaluation use BioASQ biomedical questions and long-tail PopQA questions; out-of-distribution tests use Wikipedia-based Natural Questions and EntityQuestions.NQ emphasizes common entities likely answerable parametrically, while EntQ emphasizes tail knowledge likely to require retrieval; PopQA excludes questions above median annotated popularity.Source: §5 · p. 7
• On BioASQ and PopQA, CSP achieves higher or similar task-level F1 than ICL with 2–5x fewer search calls; Action Supervision searches too little on EntQ, reducing performance.Action Supervision does not significantly improve task-level F1 on the in-distribution benchmarks.Source: Table 2 · p. 8, §5 · p. 8
• CSP ReST trains on three-agent rollouts, whereas CSP-DeAnon reveals helper identities so askers can learn source reliability, reducing helpers’ incentive to hedge.CSP uses τ = 0.1, 2,000 samples, and 32 rollouts; CSP-DeAnon otherwise follows the same setup.Source: §5 · p. 8
• Action Supervision labels answer rollouts with F1 > 0.5 and hedge rollouts with F1 < 0.5, while CSP-ReST relies on task completion rather than prescribed actions.The setup predicts calibration incentives from complementary helper knowledge; in 80% of rollouts receiving both an answer and a hedge, Gemma2-9b passes along the more confident answer.Source: §5 · pp. 7–9
• Single-agent evaluation disables #ASK and tests whether agents choose #ANSWER, #SEARCH, or #HEDGE appropriately, transferring multi-agent supervision to isolated deployment.The goal is to answer from parametric knowledge when possible, search when retrieval is likely to help, and hedge when unsure.Source: §5 · pp. 7–8

Task performance

• On held-out tasks, ICL searches nearly always and loses more than 10 F1 points to supervised methods with mismatched tools, while CSP leads Action Supervision by 6.1 F1 on Wiki/EntQ.Among supervised methods, F1 is broadly similar; Action Supervision searches least, while ICL exceeds 99% search on EntQ. The test-time oracle indicates 10 points of F1 headroom for in-domain agents at slightly higher search rates.Source: §6 · p. 9

Search calibration

• In-distribution, all trained models are better search-calibrated than ICL, which barely outperforms randomly shuffled questions.The evaluation sorts queries by P(#SEARCH), applies retrieval above a threshold, and uses base-model answers to isolate search calibration.Source: §6 · pp. 9–10
• Out of distribution, CSP generalizes better than Action Supervision; CSP-DeAnon slightly outperforms CSP at high search rates but has much lower P(#ANSWER) calibration.Source: §6 · p. 10

Answer calibration

• Action Supervision yields the best P(#ANSWER) calibration; CSP is slightly behind but substantially outperforms ICL and CSP-DeAnon.Deanonymization significantly reduces calibration: revealing the helper’s identity breaks the link between calibration and collaborative task performance.Source: §6 · p. 10

Learning dynamics

• Across three ReST epochs, search rates decrease consistently while task performance remains nearly constant after the first epoch.The policy maintains task performance while incurring lower search cost.Source: §6 · p. 10

7 Discussion

• Calibration and efficient tool use can emerge from weak supervision in a collaborative game, as agents learn when retrieval helps and provide binary confidence signals.The authors frame this as an alternative post-training approach based on mechanism design and multi-agent interaction.Source: §7 · pp. 10–11
• Mechanism design and multi-agent interaction offer an alternative route to post-training language models for stronger interactional capabilities.Source: §7 · p. 11
• The study is limited to retrieval tools, factoid QA, and one language model; proposed extensions include other tools, richer interactions, tasks without ground-truth validators, and clarification-based grounding.Source: §7 · p. 11

A Prompts

• Corpus-assisted agents choose SEARCH when the corpus may help, HEDGE when uncertain without more information, and ANSWER only when very confident from available evidence.The prompt illustrates searching, answering from retrieved evidence, and hedging a guess.Source: A Prompts · p. 16
• The prompts require short, single-line action responses, using HEDGE rather than verbal uncertainty and answering yes/no questions only with “yes” or “no.”They specify the format ACTION: <the action>: <the argument>.Source: A Prompts · pp. 16, 18
• The collaborative prompt adds ASK when a friend may know what the agent lacks, and demonstrates answering after receiving a friend's response.It also retains ANSWER and HEDGE as available actions.Source: A Prompts · p. 18

• A tool pairs a keyword with a function that maps an argument string to a list of output strings, as in #SEARCH and #CALCULATE (Parisi et al., 2022).Source: B Terminology · p. 19
• An agent has a unique identifier and may have no accessible tools; although mappings can be many-to-many, the paper focuses on assigning each tool to exactly one agent.Source: B Terminology · p. 19
• A society is a set of directed agent pairs, where each pair permits one agent to pass control and information to another; societies can vary in connectivity and symmetry.A dedicated orchestrator governs how control and information pass through the society.Source: B Terminology · p. 19
• An orchestrator selects the next agent and prompt from the society and interaction history; it may be deterministic or random, while agent policies are learned and interaction patterns can change during training.Source: B Terminology · p. 19
• A reward assigns a scalar score to a terminal interaction history, measuring society effectiveness through possible final-answer accuracy, rollout-length or tool-call penalties, a learned preference model, or a prompted auto-evaluator.Source: B Terminology · p. 19

C Additional theory

• In the two-player game, a unique pure equilibrium exists iff α1 > α2 + 2δ or α2 < β + δ; it is (S, G) when α1 > β + δ, otherwise (G, G).The theorem assumes α1 > α > α2, {α1, α2} ≠ β + δ, and α1 ≠ α2 + 2δ; when neither condition holds, both (S, G) and (G, S) are equilibria.Source: C Additional theory · p. 20
• In the n-player game, if some αi exceeds every other αj by more than nδ and αi > β + δ, the unique equilibrium has i play S and everyone else G; if all αi < β + δ, everyone plays G.Source: C Additional theory · p. 20
• In the n-player proof, the smallest-α S-bidder can switch to G to increase α and avoid the effort penalty, leaving the other players at G under the stated conditions.When maxj≠i αj < β, this process leaves only i potentially playing S; i plays S if αi > β + δ, while all-G is uniquely stable if every αi < β + δ.Source: C Additional theory · p. 20

D Equilibria of Table 1

• The display includes α2 settings of 1, β + δ, and β + 3δ.Source: D Equilibria of Table 1 · p. 21
• Multiple pure-strategy equilibria occur when α1 and α2 are sufficiently high and sufficiently similar.Figure 6 displays the equilibria as α1 and α2 vary.Source: Figure 6 · p. 21, D Equilibria of Table 1 · p. 21

E ReST learning dynamics

• Search rate fell monotonically during the first two ReST epochs, while task performance rose in mismatched settings and dipped slightly in matched settings, trading small matched regressions for larger mismatched gains and lower effort penalties.The passage attributes the pattern to retrieval results slightly impairing performance in mismatched settings, so searching less often helps those settings.Source: E ReST learning dynamics · p. 22, Figure 7 · p. 23, Figure 8 · p. 24
• Search calibration improved across methods during the first two ReST epochs, then plateaued in the third.Calibration is measured by average F1 when searching only a fraction of questions with the highest P(#SEARCH).Source: E ReST learning dynamics · p. 22, Figure 9 · p. 24
• Answer calibration improved for action supervision and collaborative self-play, but not CSP-DeAnon, whose asker can achieve high task accuracy without attending to helpers’ confidence.For CSP-DeAnon, the authors hypothesize that attending to helper identity in epoch one makes the asker’s #ANSWER-versus-#GUESS choice less correlated with reward in epoch-two training rollouts.Source: E ReST learning dynamics · p. 22, Figure 10 · p. 25