PaperlayerView exact revision on arXiv

Source-linked AI summary

Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation

Taehoon Kim, Seunggeun Cho, Dongsu Han

arXiv:2610.09587v1cs.LGcs.AIcs.CL

TL;DR

Distillation struggles to transfer reasoning quality to compact models when it relies on outcome rewards or a single teacher. CRD combines teacher cross-feedback, step-level assessment, and coherent step stitching, then trains students with quality-weighted optimization. CRD-4B reports strong benchmark results using 50K training examples, while evaluation uses one teacher pair.

  • Problem

    Outcome-based rewards do not distinguish logically sound intermediate steps from lucky correct answers, and teacher flaws can constrain student reasoning.

    Source: §1 · p. 1

  • Method

    CRD refines reasoning across teachers, scores steps independently of final answers, stitches coherent high-quality paths, and trains students using RQO.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    CRD-4B reaches 97.3% on MATH-500 and 70.3% on AIME’25 using 50K training examples, 12× fewer final training trajectories than standard distillation.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Step-level synthesis enables CRD to solve problems where individual teachers fail.

    Source: §5 · p. 10

    The evaluation uses one teacher pair, DeepSeek-R1 and Gemini 2.5 Flash; other combinations remain to be explored.

    Source: Limitations and Future Work · p. 10

Abstract

from arXiv · show

Reasoning capabilities are critical for advancing Large Language Models, yet current approaches either require massive computational budgets or struggle to effectively distill reasoning to smaller models. Standard distillation methods rely on outcome-based rewards, failing to distinguish between sound reasoning and lucky guesses. We propose Collaborative Reasoning Distillation (CRD), a framework that enhances reasoning in compact models through three innovations: (1) interactive cross-feedback where teachers iteratively critique each other's reasoning, (2) fine-grained step-wise quality assessment capturing logical validity independent of final answers, and (3) coherence-aware step stitching that synthesizes complementary strengths. Students are trained via Reasoning Quality Optimization (RQO) with budget constraints. Our model, CRD-4B, achieves 97.3% on MATH-500 and 70.3% on AIME'25, surpassing baselines while using only 50K training examples, up to 12 times smaller than the datasets of comparable models.

1. Introduction

The introduction presents CRD as a way to transfer reasoning to compact students by combining complementary teachers, step-level quality assessment, and coherent synthesis. It reports gains across mathematical, scientific, and programming benchmarks using 50K training examples.

1. Introduction

  • Distillation can pass teacher flaws to students, while simple multi-teacher aggregation or voting lacks a mechanism to select and combine high-quality steps.The introduction frames this as a ceiling on achievable performance and notes that different models have distinct reasoning strengths.Source: §1 · p. 1
  • Standard distillation rewards binary correctness or format compliance, so lucky correct answers can conceal invalid intermediate reasoning.Single-teacher reasoning chains also have limited diversity, bias propagation, and inconsistent quality; longer test-time reasoning raises latency and computational cost.Source: §1 · p. 1
  • Rather than relying on extensive test-time computation, CRD assesses intermediate-step logic and synthesizes reasoning across teachers.The proposed approach aims for comparable reasoning quality with fewer final training trajectories and shorter reasoning at inference.Source: §1 · p. 1
  • CRD combines teachers with complementary strengths, step-level reasoning assessment, and synthesis of high-quality reasoning patterns.The framework trains students on the curated data using Reasoning Quality Optimization with curriculum-based budget constraints.Source: §1 · p. 2
  • CRD-1.5B improves over its student baseline by 7.2% on MATH-500, 6.8% on GPQA Diamond, and 10.3% on LiveCodeBench, using 50K training examples.On Qwen3, CRD-1.7B gains 2.0% on AIME’24 and 5.4% on AIME’25 over same-parameter baselines.Source: §1 · p. 2

2. Methodology

CRD turns outputs from multiple teachers into a curated dataset by refining their reasoning, assessing step quality, and stitching coherent paths. Students learn from these trajectories through quality-weighted optimization and progressively constrained token budgets.

2. Methodology

  • CRD combines iterative teacher feedback, multi-judge step assessment, coherence-aware stitching, and student training with budget-constrained RQO.The methodology treats intermediate reasoning quality as distinct from final-answer correctness.Source: §2 · pp. 2–3

2.1. Interactive Multi-Teacher Cross-Feedback

  • Cross-feedback asks teachers to critique peers’ chains and refine them; mathematical accuracy rose from 64.1% to 83.5%, and code success from 52.3% to 73.9%.Performance converges after 2–3 rounds, with additional iterations providing diminishing returns.Source: §2.1 · p. 3

2.2. Step-wise Reasoning Quality Assessment

  • Step-wise assessment assigns continuous scores for logical validity and reasoning efficiency, including in solutions that reach correct answers.Two judges score each step twice; the minimum and maximum of four scores are discarded and the remaining scores averaged.Source: §2.2 · p. 3

2.3. Coherence-Aware Step Stitching

  • Thinking Path selects a hybrid sequence of teacher steps that balances local quality with coherence, rather than choosing each step independently.Embedding similarity aligns steps across teachers, with lookahead merging for granularity differences; stitching turns 528 of 2,694 incorrect pre-stitch trajectories correct (19.6%).Source: §2.3 · p. 4

2.4. Reasoning Transfer via RQO

  • RQO weights preference learning by the quality margin between winning and losing reasoning chains, giving larger margins stronger learning signals.This uses step-level scores to distinguish degrees of reasoning quality rather than only correct from incorrect outcomes.Source: §2.4 · pp. 4–5
  • A curriculum progressively tightens response budgets from 32K to 10K to 1K tokens, avoiding strict length constraints from the start.The soft constraint is intended to let students first learn longer reasoning patterns and then compress them without performance degradation.Source: §2.4 · p. 5

3. Evaluation

CRD improves reasoning across model sizes and benchmarks while using a compact curated dataset. Ablations attribute gains to cross-teacher feedback, step-level synthesis and judging, and budget-aware training.

3. Evaluation

  • Evaluation spans mathematics, coding, and graduate-level STEM benchmarks, using DeepSeek-R1 and Gemini 2.5 Flash Thinking as teachers and dense and MoE student models.The training set contains 50K instances from four sources.Source: §3 · p. 5

3.1. Performance Across Model Sizes

  • 97.3% on MATH-500 and 70.3% on AIME’25 for CRD-4B; CRD-30B-A3B also surpasses DeepSeek-R1 on math and code with 12× fewer active parameters.CRD-4B also outperforms a 32B distilled model on math and code benchmarks while using 8× fewer parameters.Source: Table 1 · p. 6, §3.1 · p. 6

3.2. Data Efficiency: Quality Over Quantity

  • At 50K examples, CRD-1.5B reaches 35.1% on AIME’25, exceeding DeepSeek-R1-Distill’s 22.0% on 600K reasoning samples and ProRL’s 33.3% on 136K samples.CRD-1.5B improves from 28.1% at 10K to 35.1% at 50K; it uses 12× and 2.7× less data than the respective baselines.Source: Figure 2 · p. 7, §3.2 · p. 8

3.3. Ablation Study: Understanding Each Component

  • Cross-Feedback adds +3.4–7.1%, including +7.1% on AIME’25 and +5.9% on LiveCodeBench; Thinking Path adds +1.1–3.5%.Cross-Feedback repairs errors across teachers, while Thinking Path can combine one teacher’s creative setup with another’s correct computation.Source: §3.3 · p. 8
  • Optimal Budget adds +0.2–0.6% and Judging adds +1.4–1.9%; curriculum length constraints support efficient reasoning, while judges score step quality independently of final answers.Judging can distinguish high-quality from mediocre paths even when both reach correct answers.Source: §3.3 · p. 8
  • On the same 50K Thinking Path trajectories, RQO adds consistent gains across four benchmarks, averaging +1.15 points over naive SFT.Naive SFT already reaches 90.5% on MATH-500, indicating that much of the gain comes from curated data.Source: §3.3 · pp. 8–9, §3.2 · p. 8, Table 5 · p. 9
  • Performance peaks at 𝜆= 0.7; at 𝜆≤0.5 MATH-500 drops to 76.0%, while 𝜆> 0.8 yields disconnected chains at 82.3%.Smaller 𝜆 overweights coherence, whereas larger 𝜆 underweights it.Source: Figure 4 · p. 9, §3.3 · p. 9
  • CRD consistently outperforms single-teacher distillation, with particularly large gains of +14–18% on AIME’24 and +10% on LiveCodeBench.The single-teacher baselines use the same scoring and student-training pipeline but omit cross-feedback and cross-teacher step stitching.Source: Figure 3 · p. 9, §3.3 · p. 9

4. Related Work

Prior work studies reasoning distillation through Chain-of-Thought, multiple teachers, iterative refinement, and step evaluation. CRD combines cross-teacher refinement with step-level synthesis and budget-aware reasoning.

Reasoning and Distillation

  • CRD combines cross-teacher feedback and step-level synthesis across mathematics, coding, logic, and STEM, building on work in CoT distillation, multi-model transfer, and iterative refinement.The passage also situates CRD alongside LIMO and contrastive representation distillation.Source: §4 · p. 9

Multi-Teacher Distillation Methods

  • CRD differs from chain-level multi-teacher methods by refining teacher outputs before distillation and scoring reasoning steps explicitly, reporting 3.6–23% gains over those methods on identical data.The related-work comparison names TinyLLM, FAIR, and MAGDi.Source: §4 · p. 10

Efficient Reasoning Distillation

  • CRD complements test-time scaling by improving distilled supervision, while its Optimal Budget curriculum can encourage shorter student reasoning for latency-sensitive deployment.The paper states that students can use a fraction of teacher-model tokens without sacrificing problem-solving capability.Source: §4 · p. 10

LLM-as-a-Judge

  • Step-level evaluation provides denser quality signals than outcome-based evaluation, helping distinguish logically sound reasoning from lucky guesses.The related work cites learned evaluation criteria as a foundation for its multi-judge approach.Source: §4 · p. 10

5. Conclusion

CRD transfers reasoning from multiple foundation models to compact students through cross-feedback, step-level quality assessment, and coherence-aware synthesis. It reports strong benchmark results for CRD-4B using 50K training examples.

5. Conclusion

  • CRD-4B reaches 97.3% on MATH-500, 70.3% on AIME’25, and 57.4% on LiveCodeBench using 50K examples, 12× fewer final training trajectories than standard distillation.The conclusion also states that step-level synthesis enables solving problems where individual teachers fail.Source: §5 · p. 10
  • CRD transfers reasoning through multi-teacher cross-feedback, fine-grained quality assessment, and coherence-aware step synthesis.The framework is intended for compact student models.Source: §5 · p. 10
  • The reported benchmark results and teacher-failure examples support the conclusion that step-level synthesis can combine reasoning beyond what an individual teacher produces.The passage does not quantify the teacher-failure cases.Source: §5 · p. 10

Limitations and Future Work

The supplied material describes limitations and future work alongside review prompts, a mathematical scoring rubric, and the coherence-aware path algorithm. It details how steps are assessed and combined, while noting boundaries in the current evaluation and rubric.

Limitations and Future Work

  • CRD may inherit teacher and judge biases, stitching does not guarantee every chain is logically correct, and evaluation covers one teacher pair without DPO comparison.Reported training cost excludes teacher generation and curation; future work proposes larger teacher pools, learned quality models, broader comparisons, formal stitching bounds, and a larger human audit.Source: Limitations and Future Work · p. 10

A. Cross-Feedback Review Prompt

  • The cross-feedback prompt asks teachers to review peer reasoning in math, code, or logic, identify errors and strengths, and produce an improved chain.Its instructions emphasize rigorous review of logical gaps, unstated assumptions, and edge cases.Source: A. Cross-Feedback Review Prompt · p. 15

B. Mathematical Scoring Rubric (T-LCM)

  • T-LCM classifies problem domains before scoring reasoning steps, and groups errors as execution errors (6.0–7.9), reasoning errors (3.0–5.9), or logic breakdown (0.0–2.9).The error groups distinguish recoverable calculation or transcription mistakes from constraint or conceptual errors and severe failures such as hallucinations or unjustified leaps.Source: B. Mathematical Scoring Rubric (T-LCM) · p. 16, §B.2 · p. 16

B.1. Score Range and Decision Boundaries

  • The scoring prompt evaluates a step using the problem, prior steps, and current step, returning a 0.0–10.0 score, error type, and rationale with rubric decision boundaries.The associated table is identified as a T-LCM scoring rubric with decision boundaries; its cell values are not supplied here.Source: Table 6 · p. 16, §B.4 · p. 17

B.3. Domain-Specific Checklists

  • The mathematical checklist adapts to domain: it checks algebraic validity and restrictions, geometry properties, number-theory constraints, and combinatoric counting and boundary cases.These checks include preconditions for theorems, diagram consistency, divisibility, and overcounting adjustments.Source: §B.3 · p. 16

C. Coherence-Aware Thinking Path Algorithm

  • Thinking Path selects a chain by jointly optimizing aggregated step quality and transition coherence, using Viterbi-style dynamic programming rather than greedy step selection.The node score aggregates JudgeLM and xVerify scores; coherence is computed from consecutive-step embedding similarity, and the stated complexity is O(T·N^2), with negligible overhead for the described settings.Source: C. Coherence-Aware Thinking Path… · p. 17, §C.1 · p. 17

C.2. Implementation

The implementation aligns teacher steps and uses dynamic programming to reconstruct a path scored for quality and transition coherence. Its design sets a quality–coherence balance and gives a complexity bound for typical settings.

C.2. Implementation

  • The implementation aligns steps, initializes a dynamic-programming table with normalized scores, updates transitions using quality and coherence, then reconstructs a chronological path subject to a score threshold.Transition coherence is rescaled embedding cosine similarity; the implementation uses a final threshold tau = 7.0.Source: §C.2 · p. 18

C.3. Key Design Choices

  • The joint objective weights step quality by λ and transition coherence by 1 − λ; λ = 0.7 assigns 70% weight to quality and 30% to coherence.The paper describes this balance as avoiding high-scoring but disconnected steps.Source: §C.3 · p. 18
  • Viterbi-style dynamic programming provides global optimization in O(T·N^2) time, efficient for the reported setting of N = 2 teachers and T < 20 steps.The implementation passage says the space complexity is O(T·N) for the dynamic-programming table and backpointers.Source: §C.3 · pp. 18–19

D. Training Data Composition

Table 7 gives the number of training instances drawn from each source and is identified as the composition of the CRD-50K training dataset.

D. Training Data Composition

E. Benchmark Descriptions

CRD is evaluated across nine benchmarks spanning mathematics, code generation, STEM reasoning, planning, and expert-level knowledge; the MATH-500 subject analysis reports improved step correctness and accuracy across the examined subjects.

E. Benchmark Descriptions

  • The evaluation covers nine benchmarks across mathematics, code generation, STEM reasoning, planning, and expert-level knowledge.Examples include multi-step mathematics, execution-tested code, graduate-level science, procedurally generated planning, and text-only expert questions.Source: E. Benchmark Descriptions · p. 19

F.1. MATH-500 Subject Breakdown

  • On a 50-problem algebra slice, CRD-1.5B reaches 92.2% step correctness versus 85.8% for DeepSeek-R1-1.5B.Precalculus improves by 6–8 percentage points in function analysis and limit computations, while number theory improves by 5–7 points in divisibility and modular arithmetic.Source: §F.1 · p. 19

F.2. Scaling Analysis

CRD's MATH-500 gains vary with student scale and benchmark difficulty: gains shrink on Qwen3 MATH-500 as the base model saturates, while AIME’25 gains persist.

F.2. Scaling Analysis

  • CRD improves MATH-500 by 7.2 points (83.9 →91.1) on DeepSeek-R1-Distill-Qwen-1.5B.Source: §F.2 · p. 19
  • For Qwen3, MATH-500 gains are +1.1 at 1.7B, +0.3 at 4B, and +0.3 at 30B-A3B, while AIME’25 gains are +5.4, +4.7, and +2.3, respectively.The passage attributes the shrinking MATH-500 gains to base-model saturation and says CRD's marginal benefit diminishes on near-saturated benchmarks but remains on harder AIME’25.Source: §F.2 · p. 20

G. Multi-Round Cross-Feedback Analysis

Cross-feedback performance converges after two to three rounds, and the judge analysis compares a dual-judge setup with individual evaluators while noting remaining scoring risks.

G. Multi-Round Cross-Feedback Analysis

H. Judge Reliability

  • On MATH-500, the full dual-judge configuration improves the student by 1.4 points, versus 0.6 with xVerify alone, 0.4 with JudgeLM alone, and 0.6 with GPT-4o alone.The setup combines two heterogeneous judges, two evaluations per judge, a 25% trimmed mean, and domain-specific verification.Source: H. Judge Reliability · p. 20
  • Re-scoring with GPT-4o differs from the aggregated score by a mean absolute 0.428 on the 0–10 scale; correlated evaluator biases may remain, and plausible but incorrect reasoning may still score highly.Source: H. Judge Reliability · p. 20

I. Stitched Training Example

The stitched trajectory replaces a correct opening chain that assumes only an exterior-center configuration with a corrected continuation. On a held-out MATH-500 example, CRD-1.5B reaches the correct answer while the baseline does not.

I. Stitched Training Example

  • The original chain derives its first two steps correctly but assumes only an exterior-center configuration; cross-feedback supplies a corrected continuation stitched as A1 → A2 → B3 → B4 → B5.Table 9 identifies A as the original chain and B as the cross-feedback continuation.Source: §I · p. 20, Table 9 · p. 20

J. Student Output Examples

  • CRD-1.5B returns p(8) = 3/56, while DeepSeek-R1-Distill-Qwen-1.5B returns −657 after an invalid factorization.The CRD solution preserves the degree-7 structure and uses Q(1) and Q(−1) to determine the remaining linear factor.Source: J. Student Output Examples · pp. 20–21

Related papers

Loading 2610.09587v1…