PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning

Zhiyuan Hu, Yunhai Hu, Juncheng Liu, Shuyue Stella Li, Yucheng Wang, Zhen Xu, See-Kiong Ng, Anh Tuan Luu, Xinxing Xu, Bryan Hooi, Cynthia Breazeal, Hae Won Park

arXiv:2601.09667v2cs.AIcs.CL

TL;DR

Multi-agent RL is costly and unstable because teammate adaptation causes non-stationarity and rewards are sparse and high-variance. MATTRL addresses this by injecting structured test-time experience into specialist deliberation and studying credit assignment for reusable dialogue experiences. Across medicine, math, and education, it reports improved performance over single- and multi-agent baselines, while retaining inference-time compute and experience-pool management as practical limitations.

  • Problem

    MARL training is resource-intensive and unstable because co-adapting teammates induce non-stationarity and rewards are sparse and high-variance.

  • Method

    MATTRL forms specialist teams, retrieves structured textual experiences during multi-turn deliberation, and uses credit assignment to construct reusable experience pools.

  • Results

    MATTRL improves results across medicine, math, and education benchmarks, outperforming single-agent and multi-agent baselines.

  • Takeaways & Limitations

    MATTRL provides a test-time experience-conditioned collaboration framework for distribution-shift-robust multi-agent reasoning without weight updates.

  • Takeaways & Limitations

    Inference-time compute and latency grow with multi-agent rollouts and exploration budget, while experience pools can accumulate stale, duplicated, or spurious heuristics.

Abstract

from arXiv · show

Multi-agent systems have evolved into practical LLM-driven collaborators for many applications, gaining robustness from diversity and cross-checking. However, multi-agent RL (MARL) training is resource-intensive and unstable: co-adapting teammates induce non-stationarity, and rewards are often sparse and high-variance. Therefore, we introduce \textbf{Multi-Agent Test-Time Reinforcement Learning (MATTRL)}, a framework that injects structured textual experience into multi-agent deliberation at inference time. MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making. We also study credit assignment for constructing a turn-level experience pool, then reinjecting it into the dialogue. Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67\% over a multi-agent baseline, and by 8.67\% over comparable single-agent baselines. Ablation studies examine different credit-assignment schemes and provide a detailed comparison of how they affect training outcomes. MATTRL offers a stable, effective and efficient path to distribution-shift-robust multi-agent reasoning without tuning.

1 Introduction

MATTRL addresses the instability and resource demands of multi-agent RL by injecting structured textual experience into multi-agent deliberation at test time. It combines specialist collaboration, experience construction, and credit assignment to improve reasoning across domains.

  • Motivation: MARL is resource-intensive and unstable because evolving teammates create non-stationarity while rewards remain sparse and high-variance.MATTRL avoids weight updates by conditioning behavior with structured experience during inference.
  • Framework: MATTRL injects test-time textual experience into collaborative multi-agent reasoning without updating model weights.The framework is intended to support rapid adaptation to new tasks and domains while preserving generality.
  • Framework: MATTRL constructs experiences using group-to-agent credit assignment and reintegrates selected dialogue turns into subsequent collaboration.The approach includes experience selection, pool construction from test-time examples, and experience integration.
  • Framework: The framework forms a specialized team that deliberates across multiple turns, retrieves relevant experience, and aggregates evidence toward agreement.A coordinator summarizes the discussion and produces the final decision after consensus or a turn limit.
  • Contributions: MATTRL validates its framework across medical, math, and education benchmarks and studies how credit-assignment choices affect experience construction and final decisions.The paper presents MATTRL as a new multi-agent test-time reinforcement learning framework using textual experience.

2 Related Work

Related work spans LLM-based multi-agent collaboration, reinforcement learning for reasoning, test-time adaptation, and credit assignment under cooperation. MATTRL connects these lines through test-time experience-guided multi-agent reasoning.

  • LLM-based multi-agent collaboration: LLM-based multi-agent research develops scalable coordination mechanisms and collaborative agent systems for complex task solving.Examples include surveys of coordination strategies and systems that add agents or organize specialized collaboration.
  • LLM-based multi-agent collaboration: Multi-agent LLM frameworks have also been applied to medical decision-making through adaptive collaboration and interactive diagnostic dialogues.MDAgents and MAC are cited as examples targeting clinical reasoning and diagnostic accuracy.
  • Reinforcement learning for LLM reasoning: Reinforcement learning research applies automatically verifiable tasks and targeted training to improve deductive, inductive, abductive, and general LLM reasoning abilities.The cited works include DeepSeek-R1, SimpleRL-Zoo, and methods targeting reasoning meta-abilities.
  • Test-time adaptation and structured experience: Test-time adaptation methods adjust LLM behavior during inference using unlabeled data, while test-time reinforcement learning converts scaling signals into pseudorewards.These approaches address adaptation to domain shifts without relying on conventional additional training data.
  • Credit assignment under collaboration: Collaborative credit-assignment research allocates contributions in cooperative multi-agent settings using value-based, LLM-based, and preference-data approaches.The cited methods include Shapley-Coop, LLM-MCA, and CollabUIAgents.

3 Methodology

MATTRL uses a staged, retrieval-augmented consultation in which selected specialists update opinions, synchronize salient changes, and produce a coordinator-synthesized decision. Test-time experiences are scored, selected, and converted into reusable structured records.

  • Multi-Expert Team Collaboration: The decision-making setting includes a task record, coordinator, specialist catalog, and callable test-time experience pool, with a preset maximum discussion length.The hospital consultation instantiates the task record as a patient record and the specialist catalog as clinical departments.
  • Multi-Expert Team Collaboration: MATTRL selects a task-conditioned team from a predefined specialist catalog rather than allowing unrestricted role invention.Each specialist maintains round-indexed opinions and a convergence flag.
  • Multi-Expert Team Collaboration: In synchronized rounds, non-converged specialists retrieve task-relevant experiences and revise their opinions before sharing salient updates with the team.The process stops when specialists converge or the maximum number of rounds is reached.
  • Multi-Expert Team Collaboration: A coordinator synthesizes cumulative team evidence into a discussion report and outputs the final decision, separating evidence aggregation from decision making.The design is described as improving controllability and auditability.
  • Test-Time Experience Construction: An LLM judge scores utterances on correctness, information gain, relevance, and clarity to support turn-level experience construction.Scores combine individual signals with a terminal team-level outcome distributed across turns and agents.
  • Test-Time Experience Construction: High-value utterances are selected and transformed into retrievable textual entries containing task context, the actionable step, and a rationale for assigned credit.The entries are generated from the utterance context and quantitative reward signals using a summarizer template.

4 Experiments

Experiments evaluate MATTRL across medicine, math, and education using multi-agent and single-agent baselines, reporting retrieval quality, exact-match accuracy, and learning gains. MATTRL achieves the strongest reported results across these settings, with test-time experience improving beyond collaboration alone.

  • Datasets and Domain Settings: The evaluation covers rare-disease diagnosis, expert-level HLE math problems, and SuperGPQA education tasks.Medicine uses 2,185 cases spanning 421 diseases; math uses 856 text-only problems.
  • Baselines: Medicine compares MATTRL with MDAgents, RareAgents, and RareAgents-Refined, while math and education compare multi-agent systems with single-agent solvers or teachers.RareAgents-Refined adds role-focused peer review and discourages fabricated tests and results.
  • Metrics: Hit@k measures whether the correct disease appears within the top-k predictions, MRR averages reciprocal rank, math uses exact-match accuracy, and education reports learning gain ∆Acc.For education, ∆Acc = Accpost −Accpre, with higher values indicating stronger instructional improvement.
  • Results: Medicine: averaged across k = 1, 3, 5, and 10, MATTRL reaches Hit@k 0.565 versus MDAgent 0.515 and RareAgents-Refined 0.528, with MRR 0.51.The largest advantages appear at Hit@1 and Hit@10, corresponding to top-rank precision and shortlist coverage.
  • Results: Math: exact-match accuracy rises from 0.27 for the single-agent baseline to 0.33 with multi-agent deliberation and 0.36 with MATTRL.The comparison attributes the additional improvement beyond deliberation to test-time experience.
  • Results: Education: with all methods starting at Accpre = 0.44, MATTRL achieves Accpost = 0.77 and ∆Acc = 0.33, compared with ∆Acc = 0.16 for the single-agent teacher.The multi-agent teacher’s proposal-and-critique process produces larger gains, while MATTRL provides additional benefits beyond collaboration alone.

5 Analysis

The analysis compares credit-assignment strategies, adaptive routing, team-size scaling, and structured experience types in MATTRL. Difference Rewards offers a strong precision–efficiency trade-off, while routing and team composition affect when collaboration is most beneficial.

  • 5.1 Group-to-Agent Credit Assignment: Difference Rewards yields the best strict-precision performance, with Hit@1/3 of 0.40/0.53 versus Naive at 0.39/0.51.
  • 5.1 Group-to-Agent Credit Assignment: Difference Rewards sharpens credit by comparing the full team with a counterfactual neutralizing one agent, whereas Shapley spreads credit across coalitions and can dilute decisive turns.Difference uses one counterfactual per agent; Shapley requires many marginal evaluations and has higher estimator variance unless heavily sampled.
  • 5.2 Adaptive collaboration between single agent and multi-agent framework: The Adaptive router improves performance by 10% over the single-agent baseline and 5.5% over MATTRL by selecting collaboration based on case characteristics.Its features include symptom complexity, multidisciplinary consultation needs, specialty count, cross-specialty divergence, and single-expert misguidance risk.
  • 5.2 Adaptive collaboration between single agent and multi-agent framework: Single-agent reasoning suits standardized, single-specialty cases, whereas multi-agent deliberation is stronger for cross-specialty evidence, risk assessment, and competing hypotheses.The router sent 282 cases to the single-agent solver and 840 to MATTRL; only the single agent solved around 300 cases, while only MATTRL solved 400+.
  • 5.3 Scaling with Team Size: Team-size scaling is task-dependent: Hit@1 peaks at three agents and declines afterward, while Hit@3, Hit@5, and especially Hit@10 gain modestly with larger teams.A three-agent team outperforms a single agent by about 14% on Hit@10; smaller teams favor precision, whereas larger teams favor recall.
  • 5.4 General and disease-specific experience: MATTRL retrieves general cross-disease rules and disease-specific checks, selecting higher-reward utterances and distilling their rationales into textual experiences.General experiences emphasize mechanism-grounded justification, high-yield discriminators, and explicit uncertainty; disease-specific experiences guide close-candidate ordering.
  • 5.5 Few-shot vs. Test-time Experience: Few-shot exemplars provide only a minor Hit@1 improvement while reducing Hit@3/5/10, supporting structured experience integration rather than undifferentiated added context.

Limitations

MATTRL has two practical limitations: inference-time cost increases with multi-agent exploration, and its experience pool can accumulate degraded or redundant guidance.

  • Inference-time compute and latency grow with multi-agent rollouts and the exploration budget.
  • A continually growing test-time experience pool may accumulate stale, duplicated, or spurious heuristics.
  • The authors propose dynamic budget control, confidence-based early stopping, and experience lifecycle management as future mitigations.Planned lifecycle measures include recency weighting, de-duplication, and anomaly screening.

A.1 Detailed Setup

The detailed setup instantiates MATTRL as a specialist multi-agent diagnostic workflow that scores, summarizes, indexes, and reinjects textual experiences during deliberation.

  • RareBench Task 4 provides patient records, and MATTRL outputs a ranked top-10 differential list evaluated with Hit@k and MRR.
  • A coordinator recruits a small multidisciplinary team from a predefined specialist catalog, grounding role selection in clinical departments.
  • Specialists produce role-specific top-10 lists across synchronized discussion rounds, after which the chair synthesizes the discussion into a final ranked list.
  • An LLM judge assigns per-utterance scores, which are combined with terminal case outcomes through decay-weighted allocation.
  • High-scoring utterances are distilled into ACTION/EXPERIENCE entries for experience-augmented prompting.
  • At test time, specialists retrieve relevant experiences by case and round context, then append them to prompts while keeping model weights fixed.

A.2 Description of Specialist Pool

The specialist-pool appendix specifies structured MDT roles, strict output formats, peer review, chair aggregation, and judge rules for producing and evaluating collaborative diagnoses.

  • A.2 Description of Specialist Pool: The specialist pool is designed to balance breadth and depth across core inpatient and outpatient specialties for targeted MDT assembly.
  • A.2 Description of Specialist Pool: The diagnostic workflow requires a reflection followed by exactly 10 numbered diagnoses with concise rationales.
  • A.2 Description of Specialist Pool: Specialists are instructed to reason only from the goal, patient information, and optional experience context, writing “insufficient evidence” when information is missing.
  • A.2 Description of Specialist Pool: Peer reviewers return structured JSON with an appraisal, verdict, validated parts, and concrete issues, while the chair consolidates agreements and disagreements.
  • A.2 Description of Specialist Pool: The chair summarizes findings, red flags, differential logic, and evidence weighting before outputting exactly 10 diagnoses without extra text.
  • A.2 Description of Specialist Pool: The judge accepts synonym and unspecified-parent matches but rejects distinct subtypes, enzyme defects, genes, unrelated conditions, and partial overlaps.

A.7 Retrieval Implementation Details

MATTRL retrieves experience hints with normalized dense embeddings, cosine-equivalent inner products, and top-K search before appending them to the reasoning prompt.

  • The retrieval module uses Qwen/Qwen3-Embedding-4B to encode stored experience key-value pairs and current instructions.
  • L2 normalization makes inner-product search equivalent to cosine similarity for indexed experience and query vectors.
  • The system retrieves the top-K experience entries by maximizing query-index similarity, with default K = 8.
  • Retrieved entries are appended through a strict “EXPERIENCE HINTS” template to guide model reasoning.

B.1 Detailed Setup

The math setup forms problem-specific specialist teams, runs bounded multi-round proposal and critique, and uses coordinator synthesis plus judged utterance credit to construct experience signals.

  • Team formation and collaboration: MATTRL’s math pipeline forms a small, problem-specific team of specialists and runs bounded multi-round collaboration.The coordinator recruits specialties tailored to each problem rather than selecting from a fixed catalog.
  • Team formation and collaboration: Each non-converged specialist proposes a solution, receives structured peer critiques, and revises until convergence or the round limit.Acceptance requires unanimous accept verdicts and no issues at any severity.
  • Coordinator synthesis: The coordinator synthesizes accepted specialist updates into a discussion report and final solution, with a rewrite pass enforcing the target format when needed.The target output includes analysis, a final answer, and a formal proof.
  • Credit assignment: An LLM judge scores terminal correctness and each utterance’s helpful influence, producing a 0–5 score optionally normalized to [0, 1].The terminal signal is binary, while utterance scores measure contribution toward the correct final solution.
  • Credit assignment: The framework combines direct utterance scores with a decayed terminal signal and selects high-value utterances for experience extraction.Credit is distributed across turns using decay and among same-turn agents proportionally to utterance scores.

B.9 Test-Time Experience Retrieval

At test time, non-converged specialists retrieve relevant textual experiences and receive them as consultative prompt hints without model-weight updates.

  • Experience retrieval: Each non-converged specialist retrieves relevant experiences from a shared pool using dense embeddings and a FAISS index.Retrieval conditions on the current problem and round context.
  • Experience integration: Retrieved experiences are appended to prompts as structured hints that agents may consult for choices such as invariants, missing cases, and counterexample checks.Agents are instructed not to quote the hints verbatim in final solutions.

C.1 Detailed Setup

The education setup evaluates a pedagogy-specialist team that diagnoses student errors, plans scaffolded instruction, and uses scored teaching interactions to build role- and context-specific experiences.

  • Experimental setup: The education experiment models a learning scenario in which pedagogy specialists guide students through complex problem-solving tasks.Pre-test questions are stratified across 13 subject matters and three difficulty levels.
  • Team formation: Teams include a diagnostician, pedagogy strategist, and subject matter expert selected from a predefined specialist pool.The roles respectively analyze errors, propose instructional strategies, and provide discipline-specific explanations.
  • Instructional dialogue: Teaching uses a structured three-round question–answer dialogue that diagnoses misconceptions and scaffolds reasoning without directly revealing the answer.The teacher receives the question, correct answer, pre-test response, reasoning, and correctness status.
  • Experience construction: 28 successful teaching cases are scored with binary post-test correctness and 0–5 step-level influence ratings to construct the pedagogy experience pool.The scoring combines global instructional success with utterance-level contribution and rubric-based quality evaluation.
  • Experience construction: The top-ranked 25% of teaching utterances are converted into general or subject-specific experiences indexed by teacher role, subject, and difficulty.At test time, role-specific experiences are retrieved and appended as consultative guidance.
Loading 2601.09667v2…