Source-linked AI summary
Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding
Taewon Yun, Jisu Shin, Jeonghwan Choi, Seunghwan Bang, Hwanjun Song
TL;DR
Long-CoT reasoning is expensive to deploy, while post-hoc curation misses collaboration and dynamic exploration among heterogeneous teachers. CoRD instead performs step-wise collaborative decoding with predictive perplexity and beam search, producing higher-quality supervision and students that approach or surpass teacher-level performance. Its evaluation is primarily limited to monolingual AIME24 and AIME25 benchmarks and SFT-based distillation.
Problem
Long-CoT distillation must identify high-quality, dynamically evolving reasoning trajectories, but curation-based methods generate complete traces post hoc and fail to exploit heterogeneous teacher collaboration.
Method
CoRD uses prompt-guided step segmentation, predictive perplexity scoring, and beam search to collaboratively construct and retain promising reasoning trajectories across teachers.
Results
CoRD produces higher-quality reasoning data and student models that approach or surpass teacher performance across the evaluated reasoning benchmarks.
Takeaways & Limitations
Step-wise collaboration and progress-aware evaluation support richer Long-CoT supervision and improved student performance under moderate compute budgets.
Takeaways & Limitations
Evaluation primarily uses monolingual AIME24 and AIME25 benchmarks, leaving generalization to multilingual settings unclear.
Abstract
from arXiv · showhide
Distilling large reasoning models is essential for making Long-CoT reasoning practical, as full-scale inference remains computationally prohibitive. Existing curation-based approaches select complete reasoning traces post-hoc, overlooking collaboration among heterogeneous teachers and lacking dynamic exploration, which leads to redundant sampling and missed complementary reasoning. We introduce CoRD, a collaborative multi-teacher decoding framework that performs step-wise reasoning synthesis guided by predictive perplexity-based scoring and beam search. This enables heterogeneous LRMs to jointly construct coherent reasoning trajectories while efficiently preserving diverse, high-potential hypotheses. Experiments show that CoRD produces higher-quality reasoning data and achieves near teacher-level student performance with fewer, structured supervision signals, without substantial efficiency overhead. CoRD further generalizes well to out-of-domain and open-ended settings. The dataset and model are available at \href{https://github.com/DISL-Lab/CoRD}{https://github.com/DISL-Lab/CoRD}.
1 Introduction
CoRD addresses the cost and curation limits of Long-CoT distillation by collaboratively constructing reasoning step by step with heterogeneous teachers. Predictive perplexity, beam search, and structured segmentation support adaptive exploration and produce higher-quality distilled supervision.
- Long-CoT reasoning is costly to deploy, making distillation into smaller student models essential for practical use.
- Long-CoT trajectories are difficult to curate because they span thousands of tokens, evolve dynamically, and create exponentially growing search spaces.PRM- and MCTS-based approaches can prematurely discard initially suboptimal paths that may contain transferable deliberative patterns.
- Curation-based methods generate complete traces before selection, wasting computation and missing complementary strategies from heterogeneous teachers.
- CoRD replaces post-hoc selection with collaborative, step-wise decoding that lets heterogeneous teachers jointly construct strategically evolving trajectories.
- Predictive perplexity evaluates candidate steps by how well the ground-truth answer is predicted from the current prefix, enabling early adaptive selection without full trajectories.Beam search preserves multiple high-potential paths, including trajectories requiring later strategic shifts or self-corrections.
- Across five close-ended or open-ended benchmarks, CoRD produces higher-quality reasoning data and students that approach or surpass teacher performance.The framework combines prompt-guided segmentation, predictive perplexity scoring, and beam search, with each mechanism empirically outperforming alternative designs.
2 Related work
Related work develops reasoning distillation, test-time scaling, and multi-model collaboration, but prior collaborative approaches mainly curate or integrate responses rather than jointly constructing Long-CoT trajectories step by step.
- Test-time scaling improves reasoning by allocating additional inference computation, supporting longer deliberation, multiple attempts, and iterative self-reflection.
- Reasoning distillation transfers complete teacher reasoning trajectories to lightweight students through sequence-level supervision rather than token-level logit matching.PRMs filter incorrect steps and MCTS combines correctness-based filtering with exploration for short-CoT settings.
- Collaborative distillation uses teacher diversity through collective MCTS, direct response integration, or simple curation, with recent LRM methods relying mainly on curation.
3 Multi-Teacher Reasoning Distillation
The multi-teacher distillation formulation contrasts post-hoc trajectory selection with CoRD’s step-wise collaboration, where teachers propose and select candidate steps to compose complementary reasoning paths.
- In curation-based distillation, each teacher generates a complete Long-CoT trajectory, and the highest-quality trajectory is selected for each training instance.The quality function Q(x, τ) determines the selected trajectory.
- Post-hoc evaluation cannot let multiple teachers collaboratively explore or refine reasoning paths during trajectory construction.
- At each step, every teacher proposes a candidate next step conditioned on the current reasoning prefix.
- A selection criterion evaluates each prefix extended with a candidate step from any teacher, constructing the distillation dataset from selected extensions.
- Step-wise collaboration composes complementary reasoning steps from multiple teachers but requires decisions about segmentation, quality evaluation, and search-space management.
- The formulation assumes one trajectory per teacher for simplicity, although each teacher can generate multiple trajectories.
- Table 1 compares line-break, prefix, and prompt-guided segmentation using distinct colors.
4 CoRD: Collaborative Reasoning Decoding for Reasoning Distillation
CoRD reformulates multi-teacher reasoning distillation as step-wise collaborative decoding, using teacher-proposed steps, predictive perplexity, and beam search to explore and retain promising reasoning paths. Its design addresses inconsistent step structure, local-selection limits, and efficiency trade-offs relative to alternative decoding strategies.
- Step-wise collaborative decoding: CoRD treats each reasoning step as a decoding unit and teacher-proposed steps as the vocabulary for collaborative autoregressive exploration.This replaces conventional token-level vocabulary with reasoning-step proposals from multiple teachers.
- Prompt-guided step segmentation: Prompt-guided segmentation inserts explicit markers to produce semantically coherent, functionally distinct steps with consistent granularity across teachers.The method embeds "<think> ### Step" in the initial prompt.
- Perplexity-based step selection: Predictive perplexity scores candidate steps by estimating how well the ground-truth answer is predicted after extending the current reasoning prefix.Higher scores indicate that the extended trajectory better predicts the correct answer, with scores bounded in [0, 1].
- Step-wise decoding with beam search: Beam search retains the top-B partial trajectories instead of committing to one locally optimal path, preserving alternatives that may enable later strategic shifts.At each step, every beam prefix is extended with candidate teacher steps before the highest-scoring trajectories are selected.
- Computational complexity: CoRD has complexity O(TKMB), while greedy decoding is its beam-size-1 special case and curation has complexity O(TKB).With cached key-value states, each expansion requires only an incremental forward pass.
- Computational complexity: CoRD has lower complexity than MCTS but higher complexity than greedy decoding or curation, while its meta-prover overhead is negligible in practice.The paper reports that CoRD yields higher-quality Long-CoT trajectories than simply increasing greedy or curation sample budgets.
5 Evaluation
CoRD evaluates step-wise collaborative reasoning against post-hoc multi-teacher baselines across reasoning-quality, distillation, efficiency, and generalization settings. It consistently yields stronger reasoning data and student performance, while predictive perplexity, beam search, and teacher specialization explain its advantages.
- Reasoning Quality: CoRD achieves the highest answer accuracy and predictive perplexity for generated Long-CoT reasoning, with stronger advantages under heterogeneous teachers.Diverse teacher signals interact step by step to reinforce complementary reasoning, suppress unstable trajectories, and explore alternatives.
- Evaluation Setup: CoRD collaboratively combines heterogeneous teacher signals during step-wise decoding, unlike Curation’s independent selection and Integration’s post-hoc trajectory merging.The comparison isolates whether teachers reason independently, merge complete traces afterward, or collaborate during decoding.
- Collaboration Dynamics: Teacher selection specializes by reasoning phase: R1-Qwen-32B and QwQ-32B dominate early phases, while Phi4-Reasoning-Plus increasingly takes over late phases.Shared-prefix conditioning lets predictive perplexity assess candidate alignment with the current trajectory context.
- Student Performance: CoRD delivers the highest Pass@1 across student sizes and teacher settings, and its 32B student surpasses every individual teacher on AIME24 and AIME25.The result indicates that step-wise collaboration can preserve and enhance reasoning patterns beyond simple teacher imitation.
- Quality Metrics: Predictive perplexity correlates strongly with student performance, whereas answer accuracy alone fails to produce comparable distillation gains.Integration achieves higher answer accuracy than Curation but performs significantly worse because it compresses reasoning into short-form CoT.
- Generalization: CoRD outperforms original curated datasets on both benchmarks, with particularly larger gains on the more challenging AIME25.The comparison uses equal amounts of original curated reasoning and CoRD data.
- Efficiency: CoRD runs in 49.0% of MCTS’s wall-clock computation time while achieving substantially higher reasoning quality than Curation at modest additional cost.The efficiency comparison reports negligible meta-prover overhead.
- Generalization: CoRD outperforms other methods in Pass@1 on MATH500 and TaTQA, indicating transfer beyond AIME to in-domain and table-based out-of-domain reasoning.TaTQA requires table-based reading comprehension, unlike the arithmetic structure shared by MATH500 and AIME.
6 Conclusion
CoRD reframes reasoning distillation as dynamic, step-wise decoding that enables collaborative trajectory construction among teacher LRMs and improves student performance under moderate compute budgets.
- CoRD enables teacher LRMs to collaboratively construct reasoning trajectories through dynamic, step-wise decoding.
Limitations
The evaluation primarily covers monolingual AIME24 and AIME25, leaving generalization to multilingual settings unclear; the distillation setup also uses only supervised fine-tuning.
- The evaluation primarily focuses on monolingual AIME24 and AIME25 benchmarks, so multilingual generalization remains unclear.
- The distillation setup uses only supervised fine-tuning, leaving preference-learning extensions for future work.
Ethical Considerations
The training data are generated by publicly available reasoning models and contain no human subjects or sensitive information, so the authors report no additional ethical concerns.
- All training data are generated by publicly available LRMs without human subjects or sensitive information.
- The authors report no additional ethical concerns during data collection or training.
Scientific Artifacts
The experiments use four reasoning models, specified datasets and checkpoints, predictive-perplexity prompting, and an integration prompt to construct coherent Long-CoT traces.
- Reasoning generation uses four language models, including public Hugging Face checkpoints and a proprietary model accessed through paid OpenAI APIs.
- The main experiment uses LIMO-v1, LIMO-v2, and S1k-1.1 datasets containing 817, 800, and 1000 samples, respectively.
- Predictive perplexity is computed by inserting partial reasoning and the ground-truth answer into a prompt.
- The Integration baseline merges individual outputs from multiple LRMs into a coherent reasoning trace consistent with Long-CoT characteristics.
B Results with Other Meta-provers
CoRD’s results show that meta-prover selection, step-wise collaboration, and beam search shape reasoning quality and distillation performance. Across teacher configurations, CoRD improves reasoning data quality, while post-hoc integration remains unreliable for very long reasoning contexts.
- Results with Other Meta-provers: Weaker meta-provers reduce reasoning quality and distillation performance, making meta-prover selection important within the teacher pool.The strongest teacher is used as the meta-prover because it is already available during distillation and requires no additional reward-model training or deployment dependency.
- Results with Other Homogeneous Teacher Model Setups: CoRD consistently improves overall data quality over Curation and Integration in homogeneous single-teacher settings.The result indicates that its gains arise from organized collective step-wise reasoning rather than stochastic diversity under the compute budget.
- Results with Other Homogeneous Teacher Model Setups: CoRD consistently outperforms baseline frameworks when distilling a teacher from a different LRM family.The additional evaluation uses DeepSeek-R1-Distill-Llama-8B, whose architecture and pretraining pipeline differ from the Qwen-based teachers in the main experiments.
- Post-hoc Integration with Stronger Integrator: A stronger integrator yields only modest improvements and still fails to reconstruct coherent Long-CoT structures.The Integration baseline must compress nearly 30K teacher tokens into a trajectory exceeding 4K tokens, often collapsing into short and shallow Short-CoT outputs.
- Collaboration Dynamics: Prompt-guided step units align collaboration with semantic reasoning phases, while MCTS rewards drive convergence toward globally stronger teachers and reduce exploration of weaker but occasionally effective ones.Figure 4 analyzes teacher hit rates across step units, and Figure 5 compares the resulting collaboration dynamics under MCTS and beam search.
G.4.1 Wall-clock Time Analysis
CoRD achieves a favorable wall-clock trade-off by adding only modest cost relative to Curation while avoiding MCTS’s repeated full rollouts. Matching CoRD’s compute with additional Curation sampling does not recover its reasoning quality or student performance.
- Wall-clock Time Analysis: CoRD adds only modest computational cost relative to Curation while invoking the meta-prover far less frequently than MCTS.It avoids full rollouts by advancing reasoning at the step level, making the additional meta-prover cost small.
- Wall-clock Time Analysis: Curation ×2 doubles completions from four to eight and raises generation cost from 168.3s to 288.7s to match CoRD’s budget.The best trajectory is selected post-hoc from the eight candidates using predictive perplexity.
- Wall-clock Time Analysis: Even at matched compute, Curation ×2 remains below MCTS and CoRD in predictive perplexity, and its student performance does not improve.The comparison attributes the gap to discarded trajectories and computational waste in post-hoc pipelines.
H Additional Experimental Details for PubMedQA
The PubMedQA study tests CoRD on open-ended, evidence-grounded reasoning that differs qualitatively from mathematical Long-CoT tasks. Under controlled distillation and judge-based evaluation, CoRD retains its advantage across teacher configurations.
- Additional Experimental Details for PubMedQA: PubMedQA requires domain-specific, paragraph-level reasoning grounded in scientific evidence and long, free-form conclusions.The authors construct a task-specific distillation dataset and compare Curation, Integration, and CoRD for open-ended answers.
- Additional Experimental Details for PubMedQA: The dataset retains difficult and complex questions, operationalized by low success rates and reasoning length, and samples three complete trajectories per question.Trajectories are sampled at temperature 0.6 from Llama-3.1-8B-Instruct or Qwen2.5-7B-Instruct without explicitly constraining generation length.
- Additional Experimental Details for PubMedQA: All distillation methods use identical teacher pools, sampling settings, student training configurations, and distillation procedures, varying only generation and meta-prover prompts.A golden grounded paragraph is included in the generation prompt to factor out retrieval ability and isolate the distillation comparison.
- Additional Experimental Details for PubMedQA: CoRD consistently produces higher-quality reasoning traces and stronger distillation performance than the baselines under both teacher configurations.Answer accuracy and student performance are assessed with an LLM judge because exact matching cannot capture diverse valid open-ended formulations.