Co-Evolving Policy Distillation

Naibin Gu; Chenxu Yang; Qingyi Si; Chuanyu Qin; Dingyu Yao; Peng Fu; Zheng Lin; Weiping Wang; Nan Duan; Jiaqi Wang

Source-linked AI summary

Co-Evolving Policy Distillation

Naibin Gu, Chenxu Yang, Qingyi Si, Chuanyu Qin, Dingyu Yao, Peng Fu, Zheng Lin, Weiping Wang, Nan Duan, Jiaqi Wang

arXiv:2604.27083v1Apr 29, 2026cs.LG

TL;DR

Consolidating multiple expert capabilities into one model remains difficult because mixed RLVR incurs capability divergence and static OPD struggles with behavioral gaps between teachers and students. CoPD co-evolves experts through interleaved RLVR and mutual OPD, outperforming mixed RLVR, OPD, and MOPD across text, image, and video benchmarks while surpassing domain-specific experts.

Problem
Mixed RLVR and static RLVR-then-OPD pipelines lose expert capabilities when consolidating multiple experts into one model.
Method
CoPD interleaves branch-specific RLVR with cross-branch mutual OPD throughout parallel expert training, avoiding a separate distillation stage.
Results
CoPD consistently outperforms mixed RLVR, original OPD, and MOPD across text, image, and video benchmarks, surpassing domain-specific experts.
Takeaways & Limitations
CoPD turns cross-domain trade-offs into mutual gains and supports all-in-one consolidation of text, image, and video reasoning capabilities.
Takeaways & Limitations
The approach relies on the behavioral consistency hypothesis that OPD supervision is more easily absorbed when teacher and student behave similarly.

Abstract

from arXiv · show

RLVR and OPD have become standard paradigms for post-training. We provide a unified analysis of these two paradigms in consolidating multiple expert capabilities into a single model, identifying capability loss in different ways: mixed RLVR suffers from inter-capability divergence cost, while the pipeline of first training experts and then performing OPD, though avoiding divergence, fails to fully absorb teacher capabilities due to large behavioral pattern gaps between teacher and student. We propose Co-Evolving Policy Distillation (CoPD), which encourages parallel training of experts and introduces OPD during each expert's ongoing RLVR training rather than after complete expert training, with experts serving as mutual teachers (making OPD bidirectional) to co-evolve. This enables more consistent behavioral patterns among experts while maintaining sufficient complementary knowledge throughout. Experiments validate that CoPD achieves all-in-one integration of text, image, and video reasoning capabilities, significantly outperforming strong baselines such as mixed RLVR and MOPD, and even surpassing domain-specific experts. The model parallel training pattern offered by CoPD may inspire a novel training scaling paradigm.

1 Introduction

The introduction identifies capability trade-offs in mixed-capability RLVR and ineffective absorption in the conventional train-then-distill OPD pipeline. It proposes CoPD, which co-evolves parallel experts through interleaved branch-specific RLVR and mutual cross-branch OPD, outperforming major baselines across text, image, and video benchmarks.

Motivation: Mixed-capability RLVR can trade off capabilities, with gains in one capability coming at the expense of another.The paper names this phenomenon capability trade-off.
Motivation: In the prevailing OPD pipeline, experts are trained to convergence before distillation, creating teacher–student behavioral distance that limits effective supervision absorption.The paper’s pilot study reports that OPD gain is inversely related to teacher–student behavioral distance.
Method: CoPD interleaves branch-specific RLVR with cross-branch mutual OPD throughout parallel training, allowing branches to deepen expertise while continuously exchanging absorbable knowledge.RLVR drives branch-specific exploration, while mutual OPD transfers cross-domain knowledge without a separate distillation stage.
Method: CoPD extends to multi-expert parallel training by jointly consolidating text reasoning, image-text reasoning, and video understanding in a three-branch setting.The introduction also describes a two-branch setting unifying text reasoning and multimodal reasoning.
Results: CoPD consistently outperforms single-domain experts, mixed-data RLVR, and static (M)OPD pipelines across the evaluated two-branch and three-branch settings.The contribution summary likewise reports gains over mixed RLVR, original OPD, and MOPD baselines across text, image, and video benchmarks.

2 Unleash the Potential of RLVR and OPD

The section shows that mixed-data RLVR loses utility through capability divergence, whereas static OPD avoids conflict but poorly absorbs teacher knowledge because behavioral overlap is low. CoPD targets both costs by separating optimization while maintaining moderate teacher–student overlap, supported by experiments linking overlap to distillation gain.

Optimization costs: Mixed-data RLVR suffers capability divergence: conflicting gradients cause interference, so gains on one capability can cancel gains on another.This cost remains regardless of the data-mixing ratio.
Optimization costs: Static OPD removes gradient conflict by training experts separately, but converged experts diverge behaviorally from the student, making teacher supervision difficult to absorb.Its absorption efficiency is modeled as η(Olow), which is positive but small.
CoPD objective: CoPD aims to preserve capability-specific optimization separation while maintaining moderate teacher–student behavioral overlap, thereby avoiding divergence cost and improving signal absorption.Its alternating structure keeps experts mutually involved during ongoing RLVR rather than consolidating only after isolated convergence.
Behavioral consistency: Top-k token overlap measures behavioral consistency because higher overlap indicates closer agreement on likely tokens along the student’s on-policy states.The hypothesis predicts that absorption efficiency η increases with overlap until teacher and student become behaviorally indistinguishable.
Empirical validation: r = 0.89 and R2 = 0.79: post-OPD gain increases monotonically with teacher–student top-k overlap, supporting the behavioral consistency hypothesis.The experiment fixes an image-domain teacher and varies student overlap through different sampling temperatures before identical OPD training.
Empirical validation: During RLVR, top-k overlap with the shared base drops monotonically while symmetric KL rises by an order of magnitude across both expert branches.This indicates that RLVR training progressively increases behavioral separation from the shared base.

3 Co-Evolving Policy Distillation

CoPD co-evolves multiple expert branches by alternating branch-specific RLVR with mutual on-policy distillation, allowing branches to develop specialized capabilities while continuously exchanging knowledge. The framework generalizes to multiple branches and merges the co-evolved branches into a unified model.

Framework overview: CoPD initializes parallel branches from the same base model, each exploring a capability through RLVR and absorbing knowledge from other branches through mutual on-policy distillation.Each branch uses its own capability dataset, while mutual distillation transfers newly acquired capability-specific knowledge across branches.
Alternating training: Training alternates branch-specific GRPO on each capability dataset with mutual OPD, allowing differentiated expertise to create signals for cross-capability transfer.The RLVR phase widens the knowledge gap between branches, which subsequent mutual distillation exploits for transfer.
Mutual OPD: Branch-specific RLVR continues during mutual OPD, so both branches alternately serve as teacher and student while continuously exchanging newly acquired knowledge.The interleaved objectives keep mutual knowledge exchange active rather than pausing native capability learning during distillation.
Training dynamics: The balance between SRL and SOPD controls CoPD’s training dynamics by trading off capability-specific discovery against the thoroughness of knowledge transfer.Larger SRL yields richer differentiated discoveries, whereas larger SOPD enables more thorough transfer.
Multi-branch extension: CoPD scales to K > 2 branches through a hub-and-spoke topology and returns a unified model by merging the co-evolved branches.In the three-branch setting, the text reasoning branch serves as the hub for image and video branches.

4 Experiments

Experiments show that CoPD consolidates text, image, and video reasoning more effectively than mixed RLVR, static OPD, and MOPD, while ablations and training-dynamics analyses support mutual distillation during parallel training.

Training Data and Evaluation Benchmarks: CoPD is evaluated in dual-branch text-and-image and scalable three-branch text, image, and video settings using dedicated reasoning benchmarks.Image reasoning uses seven benchmarks, while text reasoning uses five benchmarks; the three-branch setting additionally evaluates video reasoning.
Co-Evolution on Text and Image Reasoning: CoPD achieves the best overall performance in text-and-image reasoning, improving both capabilities simultaneously and surpassing the corresponding domain-specific experts.Mixed RLVR weakens text reasoning through cross-domain interference, while static OPD avoids interference but struggles to absorb teachers after expert training diverges.
Scaling to Co-Evolution on Text, Image, and Video Reasoning: CoPD also achieves the best overall performance across text, image, and video reasoning, outperforming MOPD across major capability groups and generalizing beyond pairwise transfer.MOPD underperforms the Video-Expert on video reasoning, whereas CoPD jointly optimizes and consolidates all three capabilities.
Ablation Study: Removing I-OPD drops text reasoning from 58.76 to 57.41, while removing T-OPD drops image reasoning from 56.97 to 56.48, confirming both directions are necessary.Each branch benefits from receiving the other branch’s distillation signal.
Behavioral Pattern Consistency During Training: CoPD maintains behavioral consistency during training: top-k token overlap stays above 0.90, whereas static OPD overlap declines and symmetric KL rises by an order of magnitude.Static OPD distills after expert training, when the experts are furthest apart; CoPD alternates RLVR with mutual OPD phases.
Effect of the SRL/SOPD Ratio: SRL:SOPD = 1.5:1 achieves the best overall performance, and CoPD consistently outperforms static OPD across tested exploration-to-distillation ratios.The analysis indicates that sufficient branch-specific exploration is needed to create useful complementary knowledge before consolidation.

5 Related Work

RLVR has emerged as a scalable paradigm for eliciting reasoning with verifiable rewards, while OPD provides dense on-policy supervision that improves target models under teacher guidance. Recent work also adopts multi-teacher OPD for base-model post-training.

RLVR: RLVR uses rule-based verifiable rewards to elicit reasoning, with GRPO enabling scalable training without a separate value network.Subsequent work improves RLVR training stability and data curation.
On-Policy Distillation: OPD supervises student-generated trajectories with dense token-level signals, mitigating train-inference distribution mismatch and rapidly improving target capabilities.Multi-teacher OPD has been widely adopted in base-model post-training.

6 Conclusion

The paper asks how to absorb multiple experts’ capabilities into one model, arguing that mixed RLVR and static OPD both lose expert capabilities. It proposes Co-evolving Policy Distillation as a central approach in which distillation occurs among experts during their evolution.

Core findings: Mixed RLVR and the traditional static OPD pipeline both suffer non-negligible loss of expert capabilities.Mixed RLVR directly mixes expert-relevant data, whereas static OPD trains experts first and distills them afterward.
Core findings: Co-evolving Policy Distillation proposes distillation among experts as a central strategy for absorbing multiple expert capabilities into one model.The passage introduces this approach as an alternative to conventional mixed RLVR and static OPD.
Research program: This work is the third installment of a Self-Taught RLVR series spanning informed, temporal, and parallel self-evolution.RLSD studies the informed self, NPO studies the temporal self, and this paper studies the parallel self.

Appendix · A Preliminaries · A.1 Group Relative Policy Optimization

GRPO is a PPO variant for large language models that estimates advantages from group-level reward statistics without a separate value network. It samples multiple responses, computes group-relative advantages from verifiable rewards, and updates the policy with clipping and KL regularization.

A.1 Group Relative Policy Optimization: GRPO is a variant of Proximal Policy Optimization tailored for large language models.
A.1 Group Relative Policy Optimization: GRPO eliminates the need for a separate value network by estimating advantages directly from group-level reward statistics.
A.1 Group Relative Policy Optimization: Given a prompt x, GRPO samples a group of G responses from the current policy πθ.
A.1 Group Relative Policy Optimization: Each sampled response yi receives a reward ri from a verifiable reward function.
A.1 Group Relative Policy Optimization: The group-level advantage is computed by normalizing rewards within the sampled group.
A.1 Group Relative Policy Optimization: GRPO updates the policy by maximizing a clipped surrogate objective with KL regularization against a reference policy πref.
A.1 Group Relative Policy Optimization: The objective uses an importance sampling ratio ρi,t and clipping threshold ϵ to constrain policy updates.
A.1 Group Relative Policy Optimization: The coefficient β controls the strength of the KL penalty in the policy objective.

A.2 On-Policy Distillation

On-policy distillation (OPD) transfers teacher knowledge through token-level supervision on the student’s own generated trajectories, minimizing token-level KL divergence between teacher and student distributions. Unlike off-policy distillation, OPD avoids training–inference distribution mismatch by learning exclusively from student trajectories.

Mechanism: OPD trains a student model πθ by having a teacher model πT evaluate responses generated by the student.For prompt x, the student samples y ∼πθ(⋅∣x), and the teacher provides a token-level probability distribution at each position.
Mechanism: The student minimizes token-level KL divergence between the teacher’s and its own distributions along the on-policy trajectory.This objective applies the teacher’s token-level supervision directly to the student-generated response.
Motivation: OPD avoids the distribution mismatch between training and inference because it trains on the student’s own trajectories rather than teacher-generated data.This distinguishes OPD from off-policy distillation, which trains on teacher-generated data.