PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent

Yinyi Luo, Yiqiao Jin, Weichen Yu, Mengqi Zhang, Srijan Kumar, Xiaoxiao Li, Weijie Xu, Xin Chen, Jindong Wang

arXiv:2602.03955v3cs.AIcs.MA

TL;DR

Multi-agent systems offer strong reasoning but face high computational costs and error propagation. AgentArk distills their reasoning dynamics into a single model through reasoning-enhanced supervision, trajectory augmentation, and process-aware training, producing stronger reasoning behavior, generalization, and robustness while retaining single-agent efficiency.

  • Problem

    Multi-agent systems achieve strong reasoning through interaction, but their computational cost, latency, and vulnerability to propagated errors motivate internalizing these benefits in a single model.

  • Method

    AgentArk generates debate trajectories and corrective traces, then applies reasoning-enhanced supervision, trajectory-based augmentation, and process-aware distillation using process rewards and GRPO.

  • Results

    Across experiments, all three distillation methods improve single-agent performance; process-aware models also show better step decomposition, self-checking, error correction, generalization, and robustness.

  • Takeaways & Limitations

    The findings support shifting multi-agent reasoning computation from inference into training while preserving a single agent's deployment efficiency.

  • Takeaways & Limitations

    Experiments cover only a subset of reasoning benchmarks and multimodal models, and the study examines a specific set of distillation pipelines.

Abstract

from arXiv · show

While large language model (LLM) multi-agent systems achieve superior reasoning performance through iterative debate, practical deployment is limited by their high computational cost and error propagation. This paper proposes AgentArk, a novel framework to distill multi-agent dynamics into the weights of a single model, effectively transforming explicit test-time interactions into implicit model capabilities. This equips a single agent with the intelligence of multi-agent systems while remaining computationally efficient. Specifically, we investigate three hierarchical distillation strategies across various models, tasks, scaling, and scenarios: reasoning-enhanced fine-tuning; trajectory-based augmentation; and process-aware distillation. By shifting the burden of computation from inference to training, the distilled models preserve the efficiency of one agent while exhibiting strong reasoning and self-correction performance of multiple agents. They further demonstrate enhanced robustness and generalization across diverse reasoning tasks. We hope this work can shed light on future research on efficient and robust multi-agent development. Our code is at https://github.com/AIFrontierLab/AgentArk.

1 Introduction

AgentArk addresses whether a single model can internalize multi-agent reasoning benefits without high inference-time cost and collaborative vulnerabilities. It evaluates hierarchical distillation strategies that transfer reasoning dynamics into one agent and reports gains in performance, process behavior, scaling, generalization, and robustness.

  • Motivation: Multi-agent systems improve complex reasoning through debate, critique, and consensus but incur rapidly growing computational overhead and can amplify errors.Dense interactions may make computation quadratic in the number of agents, while biases or hallucinations can propagate across the group.
  • Research question: AgentArk asks whether a single model can internalize multi-agent reasoning benefits without their inference-time cost and collaborative vulnerabilities.
  • Approach: AgentArk uses reasoning-enhanced supervision, diverse reasoning-trajectory augmentation, and process-aware distillation to transfer multi-agent reasoning dynamics into one model.Process-aware distillation uses process reward modeling to train critique-and-revision behavior.
  • Findings: All three reasoning-centric distillation methods improve single-agent performance, and combining approaches can yield further improvement.
  • Findings: PRM capacity matters more than student model size, while student capacity bounds multi-agent gains and teacher scaling mainly benefits larger students.Weak PRMs limit gains, and teacher scaling shows diminishing returns for smaller students.
  • Findings: Process-aware distillation improves step decomposition, self-checking, and error correction beyond accuracy, while AgentArk transfers reliably to unseen and robustness benchmarks.

6. AgentArk extends to multimodal and heterogeneous LLMs and other MAS algorithms. (§4.6)

The paper presents AgentArk as a comprehensive framework for studying multi-agent-system distillation and evaluates it from multiple perspectives.

  • Contribution: AgentArk is presented as the first comprehensive framework to explore various strategies for multi-agent-system distillation.
  • Contribution: The framework includes a scalable, MAS-strategy-agnostic data-generation pipeline and extensive evaluation intended to provide insights for future reasoning-distillation research.

2 Related Work

Prior work establishes multi-agent systems as a strong reasoning paradigm but identifies efficiency, applicability, and task-specific design constraints. AgentArk is positioned within efforts to distill richer multi-agent reasoning into single models.

  • Multi-Agent Systems: Multi-agent systems use explicit multi-turn interactions to explore solution paths, detect errors, and refine predictions on complex reasoning tasks.
  • Scope: The paper focuses on reasoning tasks, primarily homogeneous agents sharing one LLM backbone, while tool use and memory management remain future work.Heterogeneous results are addressed separately in §4.6.
  • Limitations of MAS: Multi-agent coordination incurs substantial computational cost and latency, while task-specific roles, protocols, and evaluation criteria limit use in resource-constrained or real-time settings.
  • Distillation of Multi-Agent Reasoning: Prior distillation methods transfer signals ranging from final outputs and simplified traces to interaction graphs, skill selection, debate preferences, and agentic reinforcement learning.

3 Method

AgentArk generates diverse, corrective multi-agent reasoning traces and distills them into a single student through supervised, augmented, and process-aware training. Its process-aware stage uses step-level rewards and group-relative optimization to internalize critique and revision dynamics.

  • Data Generation and Knowledge Extraction: AgentArk generates debate trajectories through iterative reflection, error correction, and convergence among multiple agents.Each agent conditions its reasoning on peers’ previous traces, producing a debate log with final traces and answers.
  • Data Generation and Knowledge Extraction: Correctness-first extraction selects ground-truth-verified consensus answers and intermediate traces that successfully reach them.The method also extracts answer-consistent trajectories exhibiting explicit self-correction and structural diversity.
  • Hierarchical Distillation: Reasoning-enhanced SFT trains students on multi-agent reasoning traces alongside final answers to preserve coherent intermediate rationales.Its objective combines reasoning loss for intermediate sequences with answer loss grounded in the input and preceding reasoning path.
  • Hierarchical Distillation: Diverse trajectory augmentation exposes students to multiple valid solution paths, including distinct identities, heuristics, and starting assumptions.The augmented training set is intended to support robustness and generalization.
  • Hierarchical Distillation: Process-aware distillation uses a Process Reward Model to provide granular supervision for intermediate logical transitions from multi-agent debates.The PRM is trained with contrastive rewards that favor steps consistent with debate consensus rather than relying only on absolute labels.
  • Hierarchical Distillation: GRPO updates the student by comparing rewards among groups of sampled outputs, using a reference-policy KL penalty and normalized PRM scores.The group-relative update removes the need for a separate value function.

4 Experiments

Experiments evaluate AgentArk across model families, reasoning benchmarks, distillation settings, and agent scales. The reported results show a 4.8% improvement for single agents, with PAD providing the most consistent gains across datasets and distribution settings.

  • Experimental Setup: Experiments span Qwen 3, Gemma 3, and Llama 3 models, including distillation from larger teachers to smaller students.The study also evaluates multimodal models, heterogeneous agents, and other multi-agent algorithms.
  • Experimental Setup: Benchmarks cover mathematical, medical, augmented-math, multi-hop, and long-form reasoning tasks.The datasets include MATH, GSM8K, MedMCQA, MetaMathQA, QASPER, HotpotQA, and QMSum.
  • Experimental Setup: 120 experiments compare RSFT, DA, and PAD with single-agent and vanilla multi-agent debate baselines, using test accuracy as the primary metric.The primary study uses five agents, with additional scaling experiments.
  • Main Results: 4.8% overall improvement is reported for a single agent, which performs only slightly below vanilla MAS after distillation.Figure 2 compares Qwen3-32B distillation across student models and datasets.
  • Main Results: 30% versus 7% maximum improvement is observed for ID versus OOD settings, while average gains are 4-6% versus 1-3%, respectively.Distillation improves both settings, but the reported gain is larger in-distribution.
  • Main Results: PAD consistently improves performance across datasets, whereas RSFT and DA produce gains that fluctuate by dataset and task.The three strategies can also be composed for consistent gains.
  • Main Results: Cross-family distillation yields larger and more consistent gains than same-family distillation, especially for Gemma-7B and LLaMA-3-8B.Within Qwen-3, smaller 1.7B and 0.6B students benefit more than the 8B student; PRM-based gains persist on ID and OOD tasks.

Appendix G.2). We further conduct detailed PRM ablation studies to analyze its effects (Appendix C).

AgentArk’s distillation improves reasoning transfer across datasets, student models, robustness settings, and modalities, while revealing that reasoning quality and student capacity constrain scaling benefits. PAD is generally the most stable and effective strategy.

  • Performance across datasets: Distillation consistently improves performance across benchmark datasets, with the largest gains on MetaMathQA and GSM8K and the smallest on MedMCQA.The pattern is strongest on reasoning-intensive tasks and weaker where specialized factual knowledge dominates.
  • Scaling the Number of Agents: Scaling beyond 5 teacher agents can degrade Qwen3-0.6B but modestly benefits Qwen3-8B, showing that teacher diversity must match student capacity.The smaller student cannot faithfully absorb increasingly complex or lengthy trajectories.
  • Data Quantity vs. Quality: PAD remains stable as training data grows, whereas RSFT and DA show high variance, stagnation, or degradation on GSM8K and MedMCQA.The results support reasoning quality rather than raw data volume as the primary bottleneck for capacity-limited students.
  • Comparison with On-Policy Self-Distillation: +1.62% versus +0.99% for on-policy self-distillation on GSM8K, with PAD outperforming self-distillation.AgentArk uses teacher-induced multi-agent trajectories, while self-distillation is constrained by the student’s own exploration.
  • Robustness and Reasoning Quality: PAD achieves the highest scores across all reported reasoning-quality and TruthfulQA robustness metrics.The evaluation covers step decomposition, intermediate verification, error localization, reasoning coherence, and factual correctness retention.
  • Generalization and Extensions: AgentArk improves transfer across open-ended, multimodal, MAS-protocol, and heterogeneous-teacher settings, though multimodal gains are modest.The results indicate transferable interaction-level reasoning patterns rather than dependence on one MAS algorithm or homogeneous teacher composition.

A.1 Dataset Overview

AgentArk constructs a multi-agent distillation dataset from verified and diverse reasoning trajectories across mathematical and domain-specific benchmarks. Its selection pipeline preserves answer correctness while exposing students to multiple valid reasoning behaviors and contrastive process supervision.

  • Dataset composition: The training corpus covers GSM8K, MATH, MetaMathQA, and MedMCQA, while HotpotQA, QASPER, and QMSum are reserved for zero-shot generalization.The training benchmarks span mathematical reasoning, augmented multi-step reasoning, and medical factual knowledge.
  • Dataset scale: The final dataset contains approximately 342k unique questions and 2M reasoning trajectories after multi-agent generation, correctness filtering, and diversity-based selection.Table 3 reports per-dataset counts of questions, retained debates, and augmented trajectories.
  • Multi-agent generation: Five shared-backbone teacher agents debate for up to 3 rounds, producing diverse trajectories that support self-correction, hypothesis revision, and cross-verification.An additional agent summarizes the final answer after the debate.
  • Correctness filtering: Only agents with verifier-validated correct final answers contribute trajectories, and instances with fewer than two correct agents are excluded from augmentation.This filtering makes answer correctness independent of reasoning style before diversity selection.
  • Diversity selection: Correctness-First Diverse Extraction retains structurally different reasoning traces that agree on the correct answer.The judge selects differences in decomposition order, intermediate representations, or solution paths without ranking reasoning quality.
  • Process supervision: The resulting supervision combines answer consistency, reasoning diversity, and a contrastive PRM that rewards steps aligned with multi-agent consensus.The PRM uses a temperature-controlled contrastive objective to score consistent steps higher than contradictory ones.

B.2 Ablation: PPO Comparison

The ablation compares PPO and GRPO for PRM-guided distillation under otherwise identical training conditions. GRPO offers comparable reasoning performance with lower computational overhead, while PPO can provide more stable optimization through a learned value baseline.

  • Optimization trade-offs: GRPO achieves performance comparable to PPO across all benchmarks while substantially reducing computational overhead.Its group-relative updates are somewhat noisier but make PRM-guided distillation more scalable.
  • Ablation result: The ablation reports that GRPO delivers more stable learning and higher final reasoning accuracy than PPO in the reported comparison.This result is attributed to the benefit of group-relative updates.

C Ablation Studies on PAD

The PAD ablations examine whether process-reward modeling and policy optimization choices affect distillation. They find that PRM capacity matters, while GRPO remains a computationally efficient alternative to PPO with comparable benchmark performance.

  • PRM role separation: Separating PRM training from target-policy training allows the study to vary PRM capacity while distilling from the same Qwen3-32B teacher.The PRM is trained independently, then fixed while supervising the target policy.
  • PRM role separation: Smaller PRMs consistently provide limited improvement, indicating that process-reward capacity affects PAD outcomes.The comparison uses different PRM and policy parameter scales under a decoupled training scheme.
  • PPO versus GRPO: PPO achieves marginally higher performance than GRPO in the reported comparison, consistent with its learned value baseline.Both methods use identically trained PRMs and differ only in second-stage policy optimization.
  • PPO versus GRPO: GRPO achieves comparable performance across benchmarks while reducing computational overhead by removing PPO’s value function.It relies on group-relative comparisons, which can produce noisier updates but improve scalability.

D.1 Generalization on Out-of-Domain Datasets

AgentArk transfers multi-agent reasoning to out-of-domain tasks, with the strongest gains appearing in larger models. The framework also shifts computation toward training: offline costs rise, while single-agent inference reduces latency and GPU usage.

  • OOD generalization: AgentArk is evaluated on HotpotQA, QASPER, and QMSum to test transfer across multi-hop reasoning, long-context understanding, and summarization.The reported experiments identify a scaling law for reasoning transferability across these domains.
  • OOD generalization: The largest gains occur for 8B models, which show consistent performance lifts across all three out-of-domain tasks.Qwen3-8B improves on QMSum from 14.94 to 17.82 F1 and from 15.72 to 17.41 ROUGE-L.
  • Model scaling: Qwen3-1.7B improves from 24.38 to 25.16 F1 on QASPER and from 14.79 to 15.60 F1 on QMSum.The passage describes these mid-sized models as following a positive scaling trajectory.
  • Computation cost: AgentArk’s training pipeline, especially PAD, introduces additional computational overhead despite reducing inference-time cost.The study reports training costs for different distillation strategies to assess practicality.
  • Computation cost: At inference, AgentArk uses one autoregressive generation instead of repeated multi-agent invocations and coordination.The paper reports reduced inference latency and GPU usage for real-time and resource-constrained deployment.
  • Reasoning quality analysis: Reasoning perplexity evaluates likelihood only on intermediate reasoning tokens, excluding prompts, questions, and final answers.Lower reasoning perplexity is interpreted as more coherent and predictable reasoning steps.

F.1 Two-Step Training Protocol

The two-step protocol applies reasoning data augmentation after RSFT or PAD, using a second-stage fine-tuning pass to increase reasoning diversity. Across benchmarks, stacking DA yields modest gains without degrading performance, and heterogeneous teacher ensembles remain effective.

  • Two-step protocol: The protocol first trains a student with RSFT or PAD, then fine-tunes it with reasoning data augmentation.The second stage is intended to enhance reasoning diversity.
  • Two-step protocol: The experiments distill from Qwen3-32B to Qwen3-1.7B using GSM8K- and MATH-derived reasoning data in the second stage.The base setting aligns augmentation data with the first-stage task.
  • Combined-strategy results: Stacking DA on RSFT or PAD produces consistent but modest gains across benchmarks.The results characterize the improvements as incremental rather than large.
  • Answer-only baseline: Answer-only SFT produces inconsistent GSM8K improvements and can degrade performance, although it yields small repeatable MedMCQA gains in similar domains.The reported MedMCQA gains include +6.8 for Gemma-7B and +3.4 for Qwen3-8B.
  • Heterogeneous teachers: AgentArk improves both Qwen3-1.7B and Gemma-7B across RSFT, DA, and PAD with heterogeneous teacher ensembles.For Gemma-7B, the gains reach +1.85, +2.67, and +2.11 points respectively.
  • Heterogeneous teachers: The heterogeneous-teacher improvements remain stable across architectures and scales, suggesting reliance on diverse reasoning behaviors rather than homogeneous consensus.The reported trends are consistent with those in homogeneous multi-agent settings.

I Broader Impact

AgentArk is presented as reducing the deployment burden of multi-agent reasoning while broadening access to its capabilities. The paper also emphasizes inherited model risks and the need for careful verification, auditing, and responsible deployment.

  • Efficient distillation can reduce latency and deployment costs for reasoning-intensive applications, including on-device and resource-constrained settings.The paper frames this as broadening access to advanced agentic reasoning capabilities.
  • Distilled students may inherit biased, misleading, or logically unsound reasoning patterns from their teacher models.The paper warns that persuasive but incorrect outputs could encourage overreliance in high-stakes or decision-support scenarios.
  • The paper recommends rigorous correctness verification for process reward models and reinforcement-learning fine-tuned models before deployment.It also recommends systematic audits for hallucinations, bias, and harmful content.
  • Extending AgentArk to interactive tool use and safety-critical decision support requires careful attention to reliability, transparency, and alignment.These conditions are presented as essential for responsible and trustworthy deployment.
Loading 2602.03955v3…