Source-linked AI summary
From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search
Junlin Liu, Jiangwang Chen, Zixin Song, Shuaiyu Zhou, Chunji Lv, Hank Wu, Kailin Jiang, Jinyang Wu, Bohan Yu, Chenxi Zhou
TL;DR
Agentic search lacks a suitable way to distill black-box proprietary reasoning into open-source models without token-level logits or teacher-specific stylistic artifacts. MAPD uses a style-normalized JSON protocol for joint distillation and reinforcement learning, consistently outperforming baselines across seven QA benchmarks with average success rates of 39.4% and 44.4% on Qwen3-1.7B and Qwen3-4B.
Problem
Proprietary-to-open-source distillation lacks an intermediate representation that transfers reasoning strategies without requiring teacher logits or imitating teacher-specific surface styles.
Method
MAPD converts offline multi-agent exploration traces into a style-normalized JSON protocol used as privileged information for joint self-distillation and outcome-based reinforcement learning.
Results
MAPD consistently outperforms baselines across seven QA benchmarks, achieving average success rates of 39.4% on Qwen3-1.7B and 44.4% on Qwen3-4B.
Takeaways & Limitations
The protocol decouples transferable reasoning strategies from proprietary linguistic patterns while preserving the student’s native token distribution.
Takeaways & Limitations
The framework still relies partly on outcome-based GRPO, which lacks the granular supervision needed to optimize complex intermediate reasoning steps.
Abstract
from arXiv · showhide
Agentic search enables large language models to solve knowledge-intensive tasks by interleaving multi-step reasoning with retrieval, yet optimizing this with outcome-based reinforcement learning (RL) provides only sparse supervision. Knowledge distillation can supply denser guidance, and advanced proprietary models with their strong reasoning capabilities are promising teachers. While distilling from proprietary models can densify this supervisory signal, conventional logit-matching is precluded by hidden logits and mismatched tokenizers, whereas raw natural language trajectory imitation transfers superficial stylistic artifacts rather than core reasoning competence. To address the heterogeneous distillation problem and bridge the distribution gap, we propose Multi-Agent Protocol Distillation (MAPD), a joint distillation and RL framework uses a structured, style-normalized protocol as an intermediate representation. An offline multi-agent system (MAS) decomposes each query, retrieves supporting evidence, repairs failed searches, and converts the resulting exploration trace into a JSON protocol containing the task type, reasoning plan, and extractive grounding facts. During training, the protocol is provided only to a privileged branch of the student policy, whose token distributions furnish a dense distillation signal alongside the sparse RL objective. Extensive evaluations across seven QA benchmarks demonstrate that MAPD consistently outperforms competitive distillation and RL, achieving average success rates of 39.4\% on Qwen3-1.7B and 44.4\% on Qwen3-4B. Crucially, the framework generalizes robustly across diverse proprietary teachers while effectively mitigating the student policy from style drift and verbosity degeneration.
1. Introduction
Agentic search systems face sparse supervision and a distribution gap when transferring reasoning from proprietary teachers to open-source students. MAPD addresses these bottlenecks through a structured, style-normalized protocol that supports joint distillation and outcome-based reinforcement learning without proprietary logits or cross-tokenizer alignment.
- Introduction: Agentic search has emerged for knowledge-intensive multi-turn tasks, but outcome-based reinforcement learning provides limited supervision through final-answer correctness.The supplied passage identifies RLVR as the prevailing optimization approach and frames its supervision around verifiable final outcomes.
- Introduction: Proprietary teachers are desirable for their capabilities in dynamic task decomposition, multi-hop reasoning, and information synthesis, yet their inaccessible logits and heterogeneous tokenizers preclude direct alignment.These constraints make conventional logit-based distillation unavailable for proprietary-to-open-source transfer.
- Introduction: Free-form natural-language trajectory imitation can cause style drift and hallucinations by transferring verbose, idiosyncratic reasoning styles instead of underlying cognitive strategies.The passage describes this failure as exacerbating the distribution gap and degrading performance in agentic tasks.
- Introduction: MAPD uses a structured, style-normalized JSON protocol as an intermediate representation and combines protocol-conditioned self-distillation with outcome-based reinforcement learning.An offline proprietary-model-powered multi-agent system decomposes queries, retrieves evidence, repairs failed searches, and converts exploration traces into protocols for online policy alignment.
2. Problem Formulation & Preliminaries
Agentic search is modeled as a bounded multi-turn retrieval interaction with sparse exact-match rewards, while GRPO optimizes sampled rollouts but lacks supervision for intermediate reasoning. OPSD adds token-level self-distillation through training-only privileged information, yet its supervision remains limited by the student’s own reasoning quality.
- 2.1. Agentic Search: Agentic search models a question x as up to K turns of policy interaction with a retrieval-augmented environment.Each turn produces reasoning steps and may issue a retrieval query; the resulting passages become the next observation, and the trajectory is flattened into one token sequence.
- 2.1. Agentic Search: Episodes terminate when the agent emits a designated final answer or reaches K turns.The reward R(x, y) ∈ {0, 1} is assigned at the trajectory level using strict exact-match evaluation of the parsed final answer.
- 2.2. Group Relative Policy Optimization (GRPO): GRPO samples a group of G rollouts for each prompt and derives advantages from their terminal rewards.The policy update uses a clipped surrogate objective together with a per-token KL penalty against a reference model.
- 2.2. Group Relative Policy Optimization (GRPO): GRPO drives exploration but evaluates only outcome success, leaving complex intermediate reasoning steps without granular supervision.This limitation motivates denser token-level guidance beyond terminal rewards.
- 2.3. Online Policy Self-Distillation (OPSD): OPSD obtains student and teacher next-token distributions from identical current policy parameters under different context conditions.The student sees s_t = (x, y_<t), while the teacher additionally receives privileged information p through s_t^+ = (x, p, y_<t).
- 2.3. Online Policy Self-Distillation (OPSD): OPSD restricts gradient flow to the student path and minimizes per-token reverse KL divergence.The privileged information is available exclusively during training, providing denser supervision without changing the student’s inference context.
- 2.3. Online Policy Self-Distillation (OPSD): The homologous OPSD formulation avoids vocabulary mismatch, but standard OPSD derives privileged information from the student’s correct rollouts.Consequently, its supervision quality is fundamentally bounded by the student’s existing reasoning limits.
3. Methodology
MAPD uses a style-normalized Structured JSON Protocol to transfer proprietary reasoning into open-source students while preserving native token distributions. An offline three-stage multi-agent pipeline generates and verifies protocols, which then support joint dense distillation and sparse reinforcement learning.
- Structured JSON Protocol: Each protocol specifies a task type, ordered reasoning plan, and extractive grounding facts to structure search, inference, and faithful reasoning.Task types include single_hop, multi_hop, comparison, and others; grounding facts mitigate parametric hallucinations.
- Structured JSON Protocol: The Structured JSON Protocol decouples core cognitive strategies from proprietary linguistic patterns, reducing style drift and preserving the student’s native token distribution.The protocol is formatted through a descriptive template, p = f(z), and limits direct exposure to teacher-specific language.
- Multi-Agent System Generation Pipeline: A three-stage MAS pipeline performs collaborative search, protocol generation, and quality gating while keeping ground-truth answers confined to offline synthesis and privileged teacher context.The quality gate checks JSON schema validity, answer consistency, extractive grounding, and oracle-leak prevention.
- Joint Training Objective: The training objective combines token-level protocol distillation for stepwise credit assignment with clipped policy-gradient reinforcement learning toward the final environment reward.λ_OPSD controls the relative strength of the distillation signal.
4. Experiments and Analysis
MAPD is evaluated across seven knowledge-intensive QA benchmarks and consistently outperforms baselines, reaching 39.4% and 44.4% average success rates on Qwen3-1.7B and Qwen3-4B. Ablations show that structured protocols, multi-agent generation, and balanced distillation pressure jointly address style drift and supervision quality.
- Main Result: 39.4% and 44.4% average success rates are achieved by MAPD on Qwen3-1.7B and Qwen3-4B, respectively, across seven QA benchmarks.Relative average improvements over SDAR are 4.8% and 3.3%, respectively; gains are more pronounced on multi-hop tasks.
- Ablation Study: 7.0 and 5.6 points are gained by replacing raw proprietary-model trajectories with structured protocols under a single-teacher setting.Raw natural-language distillation performs worse than GRPO+OPSD, indicating that representation format is essential for cross-model distillation.
- Ablation Study: 29.2%/36.1% results show that adding MAS without structured protocols does not improve performance, whereas structured protocols alone reach 37.1%/42.9%.The full MAS pipeline complements structured representation by improving retrieval depth, error recovery, factual accuracy, reasoning completeness, and traceability.
- Distillation-Weight Analysis: λ=0.05 strikes a practical balance between insufficient guidance at λ=0.01 and behavioral degeneration under excessive pressure at λ=0.1.At λ=0.1, KL divergence reaches 1.06 on 4B while mean response length collapses from 135 tokens to roughly 42 tokens and tool calls saturate at 3.0.
- Cost Analysis: 99.94% yield is obtained from 25,600 training instances, producing 25,584 valid protocols at an amortized cost of $0.057 per instance.Using Gemini-3.1-Pro, the MAS pipeline invokes the teacher approximately 6.3 times per instance and consumes approximately 12.5K tokens per instance.
5. Related Work
Related work frames agentic search as multi-turn reasoning with retrieval, while proprietary-model distillation is constrained by opaque APIs, tokenizer mismatch, and style drift. Multi-agent systems improve retrieval for complex questions through collaboration, motivating structured approaches such as MAPD.
- LLM Reasoning and Agentic Search: Agentic search extends LLM reasoning to multi-turn interaction by interleaving multi-step logical deduction with environment-augmented retrieval.
- Knowledge Distillation from Proprietary Models: Classical probability-distribution matching is precluded when distilling proprietary models because black-box APIs hide logits and tokenizers are heterogeneous.
- Knowledge Distillation from Proprietary Models: Unconstrained teacher-trajectory imitation can cause verbose style overfitting, style drift, and exacerbated hallucinations under capacity mismatch.
- Knowledge Distillation from Proprietary Models: MAPD extracts semantic strategies into a style-normalized structured JSON protocol, separating core reasoning logic from proprietary linguistic expression.
- Multi-Agent System for Complex Reasoning: Multi-agent collaboration improves recall on multi-step retrieval questions through parallel sub-query orchestration and iterative debate.
6. Conclusion
MAPD is a joint distillation-and-RL framework that bridges the distribution gap between proprietary teachers and open-source students in agentic search. It uses a style-normalized structured JSON protocol to separate core cognitive strategies from linguistic variation and address style drift and verbosity collapse.
- Framework: MAPD bridges the heterogeneous distribution gap between proprietary teachers and open-source students through joint distillation and reinforcement learning.The framework is designed for agentic search.
- Protocol: A style-normalized, structured JSON protocol decouples core cognitive strategies from teachers’ idiosyncratic linguistic shells.This intermediate representation is intended to avoid transferring superficial language patterns as reasoning competence.
- Motivation: MAPD addresses the style drift and verbosity collapse associated with raw trajectory imitation.The protocol is presented as a way to separate reasoning strategies from linguistic form.
A. Structured Protocol Example
The MAS pipeline produces complete structured protocols across different proprietary teacher backbones, combining search plans, grounding facts, and grounded answers. The examples show protocols from Claude-Opus-4.6 and GPT-5.5 that correctly derive answers through multi-step entity identification and verification.
- Structured Protocol Example: Complete structured protocols are produced with Claude-Opus-4.6, GPT-5.5, and Gemini-3.1-Pro teacher backbones.Examples 2, 3, and 4 correspond to these three teacher backbones, respectively.
- Structured Protocol Example: The Claude-Opus-4.6 protocol searches both films, cross-references their casts, and identifies Alan Rickman as the grounded answer.The protocol links The Winter Guest to Rickman’s directorial role and Die Hard to his acting role.
- Structured Protocol Example: The GPT-5.5 protocol identifies Henry III as the last Valois monarch, finds Ambroise Paré as his barber surgeon, and verifies 1510 as Paré’s birth year.The final protocol marks the answer as grounded.
B. Training Dynamics
Figures 5 and 6 characterize MAPD’s training dynamics by tracking learning trajectories across model scales and examining how the distillation weight λ_OPSD affects convergence and performance evolution. They report training success rates for Qwen3-1.7B and Qwen3-4B under different proprietary teacher models and distillation weights.
- Training dynamics: Figures 5 and 6 track MAPD learning trajectories across different model scales during optimization.The visualizations provide a comprehensive overview of the optimization process.
- Training dynamics: Varying the distillation weight λ_OPSD is used to examine its influence on convergence and performance evolution.The training dynamics systematically highlight changes associated with different distillation weights.
- Training dynamics: Training success rate is evaluated for Qwen3-1.7B under varying λ_OPSD values and different proprietary teacher models.Figure 5 specifically presents this training success-rate analysis.
- Training dynamics: Training success rate is evaluated for Qwen3-4B under varying λ_OPSD values and different proprietary teacher models.Figure 6 extends the analysis to the larger Qwen3-4B model.