From Solver Trajectories to Teaching Trajectories: Cognition-Aligned Reasoning Distillation

1 Introduction

• Large reasoning models generate chains of thought, but smaller models may struggle to acquire advanced reasoning through reinforcement learning alone.Source: §1 · p. 1
• Failed steps and long, complex rationales can hinder student learning; shorter, simpler chains may better match small models’ capacity.The introduction reports that no-failed-step trajectory subsets generally outperform failed-step subsets and produce shorter outputs.Source: §1 · p. 2
• CARD reformats teacher solver trajectories into teaching trajectories aligned with student cognition, without repeated teacher sampling.It selectively reformats trajectories with detected failed steps and leaves the others unchanged.Source: §1 · p. 3
• CARD-reformatted trajectories are reported to outperform raw teacher trajectories in-domain and achieve substantial gains in out-of-distribution generalization.Source: §1 · p. 3

Trajectory-Based Knowledge Distillation

• Trajectory-based distillation trains students on teacher-generated rationales, while filter-based methods select among existing rationales.The cited examples include MCC-KD and MoRSD.Source: §2 · p. 3
• Small language models do not consistently benefit from long teacher-generated chains of thought, and shorter, simpler chains can outperform direct imitation of frontier models.LS-Mixture SFT pairs original long chains with structure-preserving short rewrites.Source: §2 · p. 3

Quality of Chain-of-Thought

• Work on chain-of-thought quality identifies redundant tokens and failed steps as distinct concerns; failed steps are abandoned exploratory branches.Feng et al. [2025] report that Failed-Step Fraction predicts accuracy more reliably than length or review ratio, and removing failed steps improves subsequent reasoning accuracy.Source: §2 · pp. 3–4

3 Preliminaries

• The setup treats the teacher as accessible through sampled outputs and frames distillation as student training on those outputs.Source: §3 · p. 4
• Each teacher sample pairs a rationale with an answer, and a processing operator constructs the distillation dataset.Source: §3 · p. 4
• Filter-based methods select teacher samples using answer distributions or difficulty metrics.Source: §3 · p. 4

4.1 CARD Reformatter

• CARD applies a trained Reformatter to each question and original rationale, producing a cognition-aligned rationale while leaving the answer unchanged.The Reformatter is optimized with a GRPO-based objective.Source: §4.1 · p. 4, §4.3 · p. 6

4.2 Reward Design

• The Distribution Alignment Reward combines minimized KL terms for student anticipation of the rationale and preservation of the answer distribution.Lower values of both KL terms indicate greater accessibility; the answer-distribution term checks that the reformatted rationale remains coherent with answer generation.Source: §4.2 · p. 5
• The Logical Coherence Reward uses Failed-Step Fraction (FSF), while failed-step and leakage checks gate the reward; candidates failing either check receive zero.FSF is the fraction of failed steps among all rationale steps; FSF = 0 indicates a structurally clean chain.Source: §4.2 · p. 6

4.3 Reformatter Training and Dataset Construction

• After training, CARD reformats rationales with FSF > 0 and retains trajectories with FSF = 0 in the distillation dataset.The student is then trained on that dataset with the standard SFT objective.Source: §4.3 · p. 6

5.2 Main Results

• CARD improves Math Avg over Vanilla SFT by +2.36/+2.47/+2.42 points and General Avg by +3.54/+1.35/+14.73 points across Llama3.2-1B, Llama3.2-3B, and Qwen2.5-7B, respectively.The reported gains are measured against Vanilla SFT; Table 1 compares methods trained on the same DeepSeek-R1-671B trajectory pool.Source: Table 1 · p. 7, §5.2 · p. 8

5.1 Experimental Setup

• Experiments use three student models and a shared DeepSeek-R1-671B trajectory pool; CARD reformats the 8,819 trajectories with FSF > 0 in the Vanilla SFT training set.The setup partitions DeepMath-103K into 5,000 questions for Reformatter training and 30,000 questions for student SFT, with trajectories over 8,192 tokens discarded.Source: §5.1 · pp. 6–7

5.3 Comparison with FSF Filtering

• On Qwen2.5-7B, CARD exceeds Replace-FSF by 1.48 Math Avg and 13.52 Gen Avg points, supporting gains beyond passive filtering.Drop-FSF discards trajectories with detected failed steps, while Replace-FSF substitutes trajectories without detected failed steps at the original example budget.Source: §5.3 · p. 8

5.4 Robustness across Random Seeds

• Across three Qwen2.5-7B training seeds, CARD improves both aggregates in every run, with paired gains of 2.60 ± 0.22 Math Avg and 14.16 ± 0.94 Gen Avg points.The reported error bars are sample standard deviations across seeds.Source: §5.4 · p. 8

5.5 Cross-Model Transferability of CARD

• The matched Llama3.2-3B reward student performs best across benchmarks, while a Qwen2.5-7B-trained reformatter reaches 33.02 Math Avg and 55.24 General Avg.Those results are within 0.90 and 0.59 points, respectively, of the matched reformatter.Source: §5.5 · p. 8

5.6 Analysis: Dynamics, Ablation and Robustness

• CARD-reformatted rationales yield lower SFT loss than Vanilla SFT, rrat decreases during GRPO training, and student PPL shifts lower after reformatting.The PPL comparison uses Qwen2.5-7B on 5,000 trajectories before and after reformatting.Source: §5.6 · p. 9, Figure 4 · p. 9
• Random reformatting peaks at Gen Avg 27.37/42.86/52.66 for Llama3.2-1B/3B and Qwen2.5-7B, below FSF-guided selection’s 31.01/55.83/71.07.ID performance follows the same trend for Llama3.2-3B and Qwen2.5-7B; the Llama3.2-1B Math Avg gap is marginal.Source: §5.6 · p. 9, Figure 5 · p. 9
• On Qwen2.5-7B, removing answer-behavior preservation produces the largest Math Avg drop, 54.22 to 52.07, while removing failed-step penalties produces the largest Gen Avg drop, 71.07 to 66.31.Removing rrat or Ileak also degrades performance.Source: §5.6 · p. 10, Table 5 · p. 10

Conclusion

• Across three student models and eight benchmarks, CARD outperforms raw solver-trajectory distillation in in-domain mathematical reasoning and achieves even larger gains in out-of-domain generalization.The conclusion attributes mitigation of catastrophic forgetting induced by task-specific SFT to cognition-aligned rationales.Source: §6 · p. 10
• CARD uses a GRPO-trained Reformatter guided by Distribution Alignment and Logical Coherence rewards to align rationales with student predictions and reduce misleading failed steps.Source: §6 · p. 10

Limitations

• Evaluation focuses on mathematical reasoning with students up to 7B parameters; larger students and multi-turn agentic tasks remain unevaluated.The authors cite limited computational resources and identify scaling behavior and interactive settings as future work.Source: §6 · p. 10

A.1 Rationale Reformatter

• The reformatting prompt asks the Reformatter to retain core logic, clarify leaps, remove failed branches, rephrase the source, and output only the rewritten reasoning.Source: §A.1 · p. 14

A.2.1 Reasoning Information Leakage

• Reasoning information leakage occurs when the Reformatter reveals awareness of the reformatting task instead of producing a clean teaching trajectory.Source: §A.2.1 · p. 14

A.2.2 Format Penalty

• A lightweight binary format check screens each rollout before reward computation for task-meta language and problem or reasoning markers.Source: §A.2.2 · p. 15

A.2.3 FSF Judge

• The FSF judge parses rationales into Graphviz reasoning graphs and defines FSF as the fraction of failed nodes; a Qwen3-4B judge is fine-tuned on 49,000 DOT-labeled examples.The examples are built from rationales annotated with GPT-OSS-120B.Source: §A.2.3 · pp. 15–16

A.3 Training Environment and Code Structure

• CARD is implemented in Python 3.12 with PyTorch 2.6.0 and CUDA 12.4, using the VERL reinforcement learning framework.Source: §A.3 · p. 16
• The first stage trains a GRPO Reformatter with distribution-alignment and logical-coherence rewards to rewrite teacher reasoning; the second trains the student by SFT on reformatted trajectories.Source: §A.3 · p. 17
• The codebase includes evaluation and result-aggregation tools, GRPO training components, and inference pipelines for rollout, judging, and FSF tagging.Source: §A.3 · p. 17

B.1 Benchmark Descriptions

• Mathematical evaluation spans seven MATH subjects and includes grade-school GSM8K problems requiring multi-step arithmetic reasoning and basic algebra.Source: §B.1 · p. 17
• Other mathematical benchmarks include AGI-MATH standardized-exam tasks, AIME 2024 and 2025 sustained reasoning problems, and AMC 2023 high-school competition questions.Source: §B.1 · p. 17
• Out-of-domain evaluation uses MMLU-Pro without mathematics and ARC-Challenge science questions, including ARC as a probe for forgetting of general scientific reasoning after math-focused SFT.Source: §B.1 · p. 18

B.2 Baseline Descriptions

• Baselines share the candidate trajectory pool and a 30,000-example budget where applicable; Vanilla SFT trains directly on raw DeepSeek-R1-671B trajectories.Source: §B.2 · p. 18
• Alternative baselines select or modify rationales through answer consistency, student difficulty, small-teacher replacement, or token reduction.The methods include MCC-KD, MoRSD, Mix Small Teacher, and Mix TokenSkip.Source: §B.2 · p. 18

• The CARD Reformatter is trained with GRPO as one component of the experimental setup.Source: C Experimental Details · p. 18
• Knowledge distillation through supervised fine-tuning is a separate component of the setup.Source: C Experimental Details · p. 18
• The experimental details also include the computational cost of the full pipeline.Source: C Experimental Details · p. 18

C.1 CARD Reformatter Training via GRPO

• GRPO trains the CARD Reformatter on rewriting instances, with rollout, actor adaptation, reward composition, and FSF judge inference settings specified.The configuration uses a 9000-token prompt limit, 8192-token response limit, and LoRA with rank 64; rationale and answer alignment weights are 0.75 and 0.25.Source: §C.1 · pp. 18–19

C.2 Knowledge Distillation via SFT

• SFT uses a shared two-epoch schedule, with full fine-tuning for models up to 3B and LoRA for 7B students.The learning rates are 2e-05 for models up to 3B and 2e-04 for 7B models.Source: §C.2 · p. 19

C.3 Computational Cost

• The Qwen2.5-7B CARD pipeline is estimated at 416 A100 GPU-hours using eight GPUs over 52 wall-clock hours.The cost summary is specific to this student setting.Source: §C.3 · p. 20

C.4 Full Experiment Results

• The full experiments cover FSF filtering, three-seed evaluation, cross-model transferability, CARD component ablations, and FSF-judge robustness.The tables report filtering comparisons on Qwen2.5-7B, seed comparisons on Qwen2.5-7B, transferability on Llama3.2-3B, and ablations and judge robustness on Qwen2.5-7B with LoRA; the supplied passages do not provide outcome values.Source: Table 7 · p. 20, Table 8 · p. 20, §C.4 · p. 20

D Case Study

• In the algebra case study, CARD’s reformatted rationale expands the expression, derives coefficient constraints, and checks integer factor pairs to identify (a, b) = (8, −3).The original rationale includes intermediate multiplication and verification steps; the reformatted version explicitly checks factor pairs against the coefficient equation.Source: D Case Study · p. 22

NeurIPS Paper Checklist

• The paper limits its conclusions to mathematical reasoning and students up to 7B parameters, noting that larger-student scaling may require additional compute and engineering.It also reports analysis of robustness to FSF judge bias and computational cost for the representative Qwen2.5-7B setting.Source: NeurIPS Paper Checklist · p. 23
• The paper says its experimental pipeline is reproducible through disclosed protocols and configurations, supplementary training and evaluation code, and reported compute requirements.For the representative Qwen2.5-7B setting, it reports one node with 8 NVIDIA A100 PCIe 40GB GPUs and a total of 52 wall-clock hours (416 A100 GPU-hours).Source: NeurIPS Paper Checklist · pp. 24–26
• The paper reports no error bars for its main experiments because each model was trained only once due to the computational cost.The authors state that this follows prior work.Source: NeurIPS Paper Checklist · p. 26
• Existing models, datasets, benchmarks, and software are credited at first mention and used under their published licenses and terms.The paper states that no terms of use were violated.Source: NeurIPS Paper Checklist · p. 28
• The CARD framework code and FSF judge training set are documented through their artifacts, training and reproducibility details, and accompanying instructions.The FSF set is released in anonymized form; documentation covers its data source, annotation pipeline, target format, and judge fine-tuning configuration.Source: NeurIPS Paper Checklist · p. 29
• The paper reports no crowdsourcing or human-subject research; evaluations use public benchmarks and automated judging or answer extraction.It states that no human annotators were recruited and that IRB approval is not applicable.Source: NeurIPS Paper Checklist · p. 29
• LLMs are central to the experiments and CARD method as teacher and student models, the Reformatter, and the lightweight FSF judge.The paper says these uses are documented and that no additional undisclosed LLM assistance affected the methodology or results.Source: NeurIPS Paper Checklist · p. 30