PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Calibrated Confidence for Multi-Agent Collaboration

Chenxiao Fan, Chongming Gao, Gangyi Zhang, Yaxin Gong, Leyang Shen, Jiakai Wang, Dong Wang, Yang Liu, Haoyan Liu, Xinting Hu, Fuli Feng, Xiangnan He

arXiv:2610.39927v1cs.LGcs.AIcs.MA

TL;DR

Multi-agent systems pass intermediate answers between agents, but downstream agents may lack a quantitative signal of their reliability. MASCAL trains agents to communicate calibrated confidence and use it during collaboration; across several architectures and tasks, it improves accuracy and calibration quality.

  • Problem

    Multi-agent systems pass intermediate conclusions to downstream agents, but those agents may lack a quantitative reliability signal for decisions that depend on upstream confidence.

    Source: §1 · p. 1, §3.1 · p. 3, §5.2 · p. 6

  • Method

    MASCAL jointly trains task performance and confidence calibration in reinforcement learning, then uses confidence-aware weighting to teach downstream agents to respond to upstream confidence.

    Source: §1 · p. 2

  • Results

    Across three collaboration architectures, MASCAL improves task accuracy and calibration quality, and its confidence communication transfers to deeper Web-retrieval chains without additional training.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Learned final confidence supports coverage-accuracy control through selective answering and targeted rechecking.

    Source: §1 · p. 2

    The study uses fixed role assignments, identical base models, and binary correctness feedback; heterogeneous agents and open-ended tasks remain future directions.

    Source: §6 · p. 8

Abstract

from arXiv · show

Complex information seeking on the Web often requires collecting evidence from multiple documents and reasoning across them, as in multi-hop question answering. Multi-agent systems address such tasks by dividing work among specialized agents, where upstream agents produce intermediate answers and supporting evidence that downstream agents build upon. However, upstream answers vary in correctness, and most systems provide no quantitative signal about it, leaving the downstream agent to judge whether to trust or verify what it receives based solely on the textual evidence. In this paper, we propose MASCAL. Each agent learns to produce a correctness probability for its answer by jointly optimizing task performance and confidence calibration in reinforcement learning; the upstream agent passes this probability at handoff, giving the downstream agent a quantitative basis for its decisions, and the final answer also carries a calibrated probability, quantifying the reliability of the system output. To guide the downstream agent in using this signal during training, we introduce confidence-aware weighting, which increases the learning weight at positions responsive to the upstream probability, so that the agent learns more from decisions where it actively uses the signal. Across three collaboration architectures spanning Web retrieval and reasoning tasks, MASCAL improves task accuracy and calibration quality for both agents, and generalizes to deeper collaboration chains and different model scales. Intervention analysis confirms that upstream confidence affects downstream decision behavior, and the learned final confidence supports selective answering and targeted rechecking.

1 Introduction

Multi-agent systems pass upstream answers and evidence downstream, but usually provide no quantitative indication of answer correctness. MASCAL trains agents to communicate calibrated confidence, and improves task accuracy and calibration across collaboration architectures.

1 Introduction

  • Multi-source Web tasks require evidence gathering and cross-document reasoning, while single agents can be limited in coverage and reliability on tasks with many sources or extended reasoning chains.Multi-agent systems address these demands through role specialization and handoffs of intermediate answers and evidence.Source: §1 · p. 1
  • Upstream answers vary in quality, yet most multi-agent systems provide only answers and textual evidence, leaving downstream agents to decide whether to trust or verify without a quantitative correctness signal.This motivates learning calibrated confidence while completing each agent’s task and learning to use that signal downstream.Source: §1 · p. 1
  • MASCAL jointly trains agents for task performance and confidence calibration, passes upstream correctness probabilities at handoff, and weights downstream positions that respond to the signal.The final agent also outputs calibrated confidence to support withholding uncertain answers or directing rechecks.Source: §1 · p. 2
  • Across three collaboration architectures and Web-retrieval and reasoning tasks, MASCAL improves task accuracy and calibration, and generalizes to deeper chains and smaller model scales.Intervention analysis finds that upstream confidence affects downstream decisions; final confidence supports selective answering and targeted rechecking.Source: §1 · p. 2

2 Related Work

Related work covers role-level collaboration learning and confidence or uncertainty in multi-agent systems. MASCAL connects calibrated confidence production, transmission, and downstream use in one learning process.

2 Related Work

  • The related-work review organizes prior research around how agents learn to collaborate and how confidence or uncertainty signals enter multi-agent settings.Source: §2 · p. 2

2.1 Multi-Agent Collaboration and Learning

  • Earlier collaboration methods train role policies, optimize system-level performance, or control Web search, but leave the reliability of upstream handoff answers outside their learning objectives.MASCAL incorporates calibrated answer confidence into role-level training and exchanged information.Source: §2.1 · p. 2

2.2 Confidence and Uncertainty in MAS

  • Prior confidence methods address expression, utilization, uncertainty propagation, or final-answer probability estimation, but no existing framework unifies production, transmission, and utilization across the collaboration pipeline.MASCAL trains roles on upstream-generated reports and uses confidence-aware weighting to strengthen downstream use of the signal.Source: §2.2 · p. 2

3 Preliminary

This section formalizes sequential collaboration and calibrated answer confidence, then introduces GRPO as the policy-optimization framework. It establishes why calibration matters when confidence directly informs another agent’s decisions.

3.1 Task Formulation

  • Each agent receives a subtask, context, and upstream reports, produces reasoning, an answer, and rationale, then passes a report onward; the final agent answers the original task using its subtask and upstream reports.An agent’s answer confidence is its estimated probability of correctness, with calibration relating that probability to actual correctness.Source: §3.1 · p. 3
  • Because another agent automatically consumes confidence and uses it in decisions, miscalibration can systematically bias downstream behavior, making calibration a prerequisite for effective confidence communication.Correctness is determined by a binary verification function.Source: §3.1 · p. 3

3.2 Group Relative Policy Optimization

  • GRPO estimates each trajectory’s advantage from its reward’s deviation from the group mean, enabling separate reward signals to produce advantages from the same rollouts.The policy is updated through an importance-weighted objective.Source: §3.2 · p. 3

4 Method: MASCAL

MASCAL trains agents to produce calibrated correctness probabilities alongside task answers, then explicitly trains downstream agents to use upstream confidence. Its method combines task and confidence learning with position-specific weighting for decisions responsive to that signal.

4.1 Joint Task and Confidence Learning

  • Each agent’s trajectory separates task behavior from confidence analysis, with distinct policy gradients for reasoning and analysis and a direct probability loss for calibration.The direct BCE loss calibrates the numerical readout independently of the generated text, while the confidence policy gradient shapes the analysis text.Source: §4.1 · pp. 3–4
  • The confidence readout uses the logits of two designated tokens and a sigmoid, allowing probability-loss gradients to provide finer-grained calibration supervision than reward signals alone.The tokens are named H and L.Source: §4.1 · p. 3
  • The task policy gradient applies its task advantage only to the task segment, while the confidence segment supplies the reliability analysis and readout.The joint objective combines task learning, confidence policy-gradient learning, and direct probability supervision.Source: §4.1 · pp. 3–4

4.2 Confidence-Aware Downstream Training

  • Confidence-aware weighting strengthens task learning at positions whose token probabilities change when upstream confidence is present rather than masked.The method re-scores the trajectory with confidence masked, normalizes and centers the absolute per-position log-probability differences, then weights the task advantage; λ controls adjustment magnitude.Source: §4.2 · p. 4
  • MASCAL trains downstream agents on reports from a trained, frozen upstream policy so their confidence-bearing inputs match the deployment distribution.The downstream task policy gradient receives additional confidence-aware weighting; its confidence policy gradient and BCE loss remain unchanged.Source: §4.2 · p. 4

5 Experiments

Experiments test MASCAL across three collaboration architectures and Web, logical, and mathematical reasoning tasks. MASCAL improves task accuracy and confidence quality, with confidence signals also shaping downstream decisions and supporting selective answering and rechecking.

5.2 Main Results (RQ1)

  • Across four scenarios, MASCAL achieves the highest accuracy; gains are strongest in search collaboration, where confidence at handoff raises accuracy above the single-agent level.Other multi-agent approaches do not match MASCAL’s improvements.Source: Table 2 · p. 5, §5.2 · pp. 5–6

5 Experiments

  • Experiments compare sequential handoff, report fusion, and answer review across multi-hop QA, logical and mathematical reasoning, with task-accuracy and confidence-quality metrics.The study uses Qwen3-8B and compares single-agent, no-confidence multi-agent, and confidence-based multi-agent baselines.Source: §5.1.1 · p. 5, §5.1.2 · p. 5, §5.1.3 · p. 5

5.3 Confidence Quality and Its Effect on Downstream Agents (RQ2)

  • MASCAL achieves the best upstream Brier and AUROC across settings, while interventions and confidence-bin behavior show that confidence shapes retrieval and revision decisions.In answer review, hiding confidence causes only minor accuracy drops, whereas random values cause a clear degradation; both interventions also worsen downstream Brier scores.Source: Table 3 · p. 6, §5.3 · p. 6, Table 4 · p. 6

5.4 Final Confidence Quality and Utility (RQ3)

  • MASCAL’s final confidence is best or near-best in every scenario, and its selective-answering curve exceeds other methods at every coverage level.At 20% coverage, accuracy reaches 86–97% depending on the task.Source: §5.4 · pp. 6–7, Table 5 · p. 7, Figure 5 · p. 7
  • Rechecking the least-confident answers improves accuracy monotonically with budget on handoff and fusion, while random rechecking stays near zero or turns negative on handoff.The gap widens with budget, indicating that final confidence directs extra search toward cases that need it most.Source: Table 6 · p. 7, §5.4 · p. 7

5.5 Ablation Study (RQ4)

  • Removing calibration loss causes the largest confidence-quality degradation; confidence-aware weighting chiefly benefits search, where removal reduces accuracy by 3–4pp.On fusion, removing calibration loss drops accuracy below Task-RL MAS, which transmits no confidence.Source: Table 7 · p. 7, §5.5 · pp. 7–8

5.6 Generalization (RQ5)

  • MASCAL’s accuracy and calibration advantages persist on unseen 3-hop and 4-hop chains and at 1.7B, 4B, and 8B model scales.Its Brier score remains stable near the single-handoff level in deeper chains; at 1.7B, MASCAL has substantially better Brier and AUROC than Base MAS and Task-RL MAS.Source: Table 8 · p. 8, §5.6 · p. 8, Table 9 · p. 8

6 Conclusion and Future Work

MASCAL trains agents to communicate calibrated answer confidence and teaches downstream agents to use it; the study also identifies limits tied to fixed roles, shared model initialization, and binary correctness feedback.

6 Conclusion and Future Work

  • MASCAL trains each agent to express calibrated answer confidence and uses confidence-aware weighting to help downstream agents act on upstream signals.The approach improves task accuracy and calibration quality across three collaboration architectures.Source: §6 · p. 8
  • Intervention and ablation analyses indicate that confidence must be reliable through dedicated calibration training and incorporated into downstream decisions.Producing a signal or exposing it to the downstream agent alone is insufficient.Source: §6 · p. 8
  • The study fixes role assignments, initializes agents from the same base model, and relies on binary correctness feedback; future work targets heterogeneous models, open-ended tasks, and dynamic multi-round collaboration.The authors also identify role selection and computational allocation as potential uses for confidence signals.Source: §6 · p. 8

A Experimental Details

The appendix outlines experimental setup, including its retrieval infrastructure and the allocation of retrieval calls across collaboration roles.

A Experimental Details

  • The appendix describes retrieval setup, training and evaluation protocols, and experimental prompts.Its retrieval infrastructure uses the Wiki-18 English Wikipedia dump and indexes candidate paragraphs from MuSiQue, HotpotQA, and 2Wiki.Source: A Experimental Details · p. 10, §A.1 · p. 10

A.1 Retrieval Infrastructure

  • Retrieval uses E5-base-v2 over a flat inner-product index, returning the top three passages truncated to 200 words each.The indexed corpus combines Wiki-18 with candidate paragraphs from MuSiQue, HotpotQA, and 2Wiki.Source: §A.1 · p. 10
  • Each trajectory allows at most three retrieval calls, allocated across upstream and downstream roles according to the collaboration architecture.A zero-shot Qwen3-8B planner produces frozen sub-questions for training and evaluation.Source: §A.1 · p. 10

A.2 Training and Evaluation

Training separates upstream and downstream agent phases, while evaluation and prompts specify the protocols used across retrieval and answer-review scenarios.

A.2 Training and Evaluation

  • The upstream agent is trained on one half of a disjoint training split and frozen before generating confidence-bearing reports for downstream-agent training on the other half.Both halves are equal in size, and the agents share scenario hyperparameters listed in Table 10.Source: §A.2 · p. 10, Table 10 · p. 10
  • The calibration-loss weight is 0.03 on MuSiQue and 0.01 on LogiQA2 and DeepScaleR; report fusion uses upstream agents with seeds 43 and 44.Source: §A.2 · p. 10
  • Retrieval scenarios sample at T=0.7, top-p=0.8, and top-k=20 and average three seeds; review scenarios use T=0.7, top-p=1.0, and top-k off.Task correctness uses sub-EM for retrieval, math-verify for math, and exact choice matching for logic; confidence uses Brier score and AUROC.Source: §A.2 · p. 10

A.3 Prompt Templates

  • Retrieval prompts have upstream agents answer a sub-question from retrieved documents and send a report for downstream agents to complete the full question.Report fusion supplies two upstream reports instead of one; upstream reports request a short answer, copied evidence, uncertainty analysis, and a confidence token.Source: §A.3 · pp. 10–11
  • The retrieval downstream prompt assigns the full-question answer, permits searches when knowledge is lacking, and asks the agent to assess uncertainty and provide a confidence token.The report is evidence to assess, and the downstream agent remains responsible for the final answer.Source: §A.3 · p. 11
  • Answer-review prompts have upstream agents solve tasks and downstream agents inspect proposed solutions before producing their own answers.The math and logic variants share this structure, with different answer carriers.Source: §A.3 · p. 11

B Additional Ablations

The appendix reports two additional ablations: a calibrated single-agent model and an incremental analysis of MASCAL components.

B Additional Ablations

  • Two additional ablations examine a single agent trained with calibration objectives and an incremental analysis of MASCAL’s components.Source: B Additional Ablations · p. 11

B.1 Calibration RL for a Single Agent

Calibration training improved single-agent confidence quality but did not account for MASCAL’s larger accuracy gains. The comparison isolates calibration training from inter-agent confidence communication.

B.1 Calibration RL for a Single Agent

  • Single-agent baselines used task-correctness training, while the calibrated single agent used MASCAL’s role-specific joint objective under the same data and search budget.This comparison separates calibration training from confidence communication between agents.Source: §B.1 · p. 11
  • Calibration training brought single-agent Brier scores close to MASCAL without reducing accuracy, but accuracy gains remained small and below MASCAL in every scenario.Table 11 reports accuracy (%) and Brier↓ averaged over test sets.Source: §B.1 · p. 11, Table 11 · p. 11

B.2 Component Analysis of MASCAL

The component analysis tests calibration, transmission of upstream confidence, and downstream confidence-aware weighting. Calibration improves confidence quality, while passing and using confidence contribute further gains.

B.2 Component Analysis of MASCAL

  • The analysis adds MASCAL components one at a time to Task-RL MAS: calibration training, transmission of upstream confidence, and confidence-aware weighting.Calibration training comprises a confidence policy gradient and BCE loss; transmission includes upstream confidence in the report, and weighting modifies downstream training.Source: §B.2 · pp. 11–12
  • Passing upstream confidence closes most of the calibration gap and raises accuracy further; weighting adds much of the accuracy gain in report fusion and logic.Table 12 reports accuracy (%) and Brier↓ averaged over test sets.Source: Table 12 · p. 12, §B.2 · p. 12
  • Calibration training lowers Brier scores in every scenario and raises accuracy in sequential handoff and math, but changes accuracy little in report fusion and logic.The downstream agent remains less calibrated than the calibrated single agent, especially in search-based collaboration, until it receives upstream confidence.Source: §B.2 · p. 12

C Training Overhead

Confidence-aware weighting adds a modest training-time cost while improving accuracy and calibration across scenarios. Its scoring pass is used during training, not deployment.

C Training Overhead

  • The overhead measurement covers complete upstream and downstream training iterations with and without weighting, using eight 80-GB GPUs.Both variants use the training configuration in Appendix A.Source: C Training Overhead · p. 12
  • 3–14%: confidence-aware weighting increases total training-iteration time across the four scenarios.The extra pass scores existing trajectories rather than generating new ones; Table 13 compares iteration time with and without weighting.Source: Table 13 · p. 12, C Training Overhead · p. 12
  • Confidence-aware weighting improves accuracy and calibration in every scenario, with the largest accuracy gains in search-based collaboration.At deployment, the downstream agent reads upstream confidence and answers without an extra scoring step.Source: C Training Overhead · p. 12

Related papers

Loading 2610.39927v1…