Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition

1 Introduction

• The study targets a gap in evidence about distillation on domain-specific, expert-curated problem sets with lower contamination risk.Prior studies often use generic benchmarks such as MATH-500 or GSM8K, which may include pretraining data.Source: §1 · p. 2
• The work applies knowledge distillation to undergraduate problems from the John O’Bryan Mathematics Competition at Northern Kentucky University.The paper describes the corpus as spanning multiple difficulty levels and mathematical topics.Source: §1 · p. 2
• 69.43% ± 0.17%: LoRA fine-tuning with early stopping improves Qwen2.5-7B from 64.67% and generalizes to MATH-500.The contributions also report token-budget effects across six levels and roughly 40% of failures as formatting errors.Source: §1 · p. 2
• The pipeline digitizes 2011–2025 problems, generates and verifies teacher reasoning traces, fine-tunes a quantized Qwen2.5-7B student, and evaluates competition and MATH-500 performance.A diagnostic 1,000-iteration run informed five independent 200-iteration runs and a multi-level token-budget experiment.Source: §1 · p. 2

2.1 Knowledge Distillation

• Knowledge distillation transfers teacher knowledge to a student; soft targets encode class relationships, and PAC theory provides a sample-complexity justification.The section also surveys feature-level, relation-based, and task-specific objectives.Source: §2.1 · p. 3

2.2 Distillation for Reasoning LLMs

• Prior reasoning-distillation studies report gains from teacher-generated chain-of-thought traces, implicit reasoning supervision, and parameter-efficient adaptation.Examples include GSM8K and grade-school or symbolic reasoning tasks.Source: §2.2 · p. 3

2.3 Vision-Language Distillation

• The VLV auto-encoder uses a frozen diffusion model as an information bottleneck and an LLM to decode language representations into image captions.The paper draws a parallel between this fixed-teacher strategy and its use of DeepSeek-R1 as an API teacher.Source: §2.3 · p. 3

2.4 Response Length and Reasoning Quality

• Prior work links explicit intermediate reasoning and inference-time compute to improved performance, motivating measurement of accuracy under constrained reasoning tokens.The cited work also examines how structuring reasoning steps affects theorem-proving accuracy.Source: §2.4 · p. 4

2.5 Mathematical Benchmarks and Competition Problems

• Competition problem sets offer expert curation, graded difficulty, and reduced pretraining-contamination risk compared with standard mathematical benchmarks.The John O’Bryan corpus spans algebra, calculus, combinatorics, and linear algebra across three difficulty tiers.Source: §2.5 · p. 4

3.1 Theoretical Foundation

• Because teacher logits are unavailable through the API, generated step-by-step reasoning traces serve as supervised targets and reduce the objective to cross-entropy fine-tuning.The standard objective combines cross-entropy against labels and KL divergence between teacher and student distributions.Source: §3.1 · p. 4

3.2 Dataset Construction

• The 671-problem corpus is split chronologically: 2011–2021 for training, 2022–2023 for validation, and 2024–2025 for held-out testing.Problems with missing answer keys, illegible diagrams, or ambiguous phrasing were removed.Source: §3.2 · p. 5

3.3 Dual-Agent Teacher Framework

• Two DeepSeek-R1 agents work sequentially: a solver produces a step-by-step response ending with a final answer, and a verifier judges it; accuracy was 91.4% on 651 competition questions and 94.0% on MATH-500.Source: §3.3 · p. 5

3.4 LoRA Fine-Tuning Configuration

• The student is a 4-bit Qwen2.5-7B-Instruct model adapted with LoRA while its base weights remain frozen.The configuration uses rank 8 and trains 11.534M parameters, or 0.151% of the base model’s 7,615.617M parameters.Source: §3.4 · p. 5

3.5 Multi-Run Training Protocol

• Validation loss reached 0.374 at iteration 200, then rose to 0.826 by iteration 1,000 while training loss fell toward zero, motivating early stopping at 200.Source: §3.5 · p. 5
• Five independent runs used different random seeds and 200-iteration limits, with other hyperparameters held constant; each merged model was evaluated separately.Peak memory was 12.247 GB per run.Source: §3.5 · p. 5

3.6 Multi-Level Token Budget Experiment

• The token-budget evaluation uses unrestricted R1 as the 69.43% baseline and progressively smaller budgets from 400 tokens at R2 to 100 at R6.Correctness was judged with the same DeepSeek-R1 judge, and accuracy included only questions with verified correct ground-truth answers.Source: §3.6 · p. 6

4 Dataset Description

• 671 problems from 2011–2025 cover three competition sections, with a chronological split intended to prevent temporal leakage.Freshman/Sophomore has 265 problems, Junior/Senior 268, and Two-Person Speed 118.Source: §4 · p. 6
• The dataset includes competition records and uses MATH-500 as an out-of-domain evaluation set, with no MATH-500 problems in training.Table 1 summarizes corpus statistics.Source: §4 · p. 6, Table 1 · p. 6
• 4.51 to 5.79: DeepSeek-R1’s average yearly difficulty ratings span this range, indicating moderate and consistent difficulty.Fig. 1 rates difficulty on a 1–10 scale and marks years at or above the mean of 5.19 with darker bars.Source: §4 · p. 6, Figure 1 · p. 7

5 Evaluation Metrics

• Answer accuracy is the share of problems where the model’s answer matches the ground truth.The displayed ratio uses the number of correct answers over total problems judged.Source: §5 · p. 7
• Task-level metrics from the main distillation experiment are reported as mean ± standard deviation across five runs.Correctness is assessed by a separate DeepSeek-R1 judge rather than string matching because equivalent answer forms can differ.Source: §5 · p. 7
• Test perplexity is computed as PPL = exp(Ltest).The supplied definition identifies Ltest as the test loss.Source: §5 · p. 7

6.1 Teacher Model Performance

• DeepSeek-R1 performs well across all three competition sections, supporting its use as a source of chain-of-thought data.The supplied text does not give section-level accuracy values.Source: §6.1 · p. 7, Table 2 · p. 8

6.2 Training Dynamics and Overfitting

• Validation loss reaches 0.374 at iteration 200, then rises to 0.826 by iteration 1,000 as training loss falls toward zero.This diagnostic run identifies iteration 200 as the early-stopping point; the authors describe later training as shifting toward copying training sequences.Source: Figure 2 · p. 8, §6.2 · p. 8, Table 3 · p. 9

6.3 Model Accuracy Comparison

• 69.43% ± 0.17% mean competition accuracy from five 200-iteration runs exceeds the base Qwen2.5-7B’s 64.67%.The 1,000-iteration model reaches 67.59%, and the two-person speed section is weakest at 64.3%.Source: Table 4 · p. 9, §6.3 · p. 9

6.4 Test Loss and MATH-500 Generalization

• 73.1% ± 0.18% on MATH-500 for the five fine-tuned models exceeds the base model’s 70.1%.The subject breakdown is strongest on Algebra (93.5%) and weakest on Intermediate Algebra (52.1%).Source: §6.4 · pp. 9–10, Table 5 · p. 10

6.5 Effect of Response Length on Accuracy

• Accuracy falls from 69.43% at unrestricted R1 (mean 220 words) to 41.9% at R6 (100-token budget, mean 31.2 words).The R2-to-R3 decline is 1.3 points, while the decline accelerates at R4 and below as meaningful intermediate steps no longer fit.Source: Table 6 · p. 10, §6.5 · p. 10
• Two-person speed accuracy falls 34.5 points from R2 (57.4%) to R6 (22.9%); year-level accuracy also varies across reasoning levels.The speed section’s multi-step numerical problems rely on intermediate calculations; Fig. 3 reports per-year accuracy across levels.Source: Table 7 · p. 11, §6.5 · p. 11, Figure 3 · p. 11

6.6 Architectural Compatibility and Inference

• Applying the Qwen2.5-7B adapters to Qwen2.5-3B produces a projection-shape mismatch, confirming the adapters are architecture-specific.On a representative 19-gon prompt, the fused model computes 3,060° using 228 tokens, at 31.8 tokens per second and 4.4 GB peak memory.Source: §6.6 · p. 12

Early Stopping Threshold

• 69.43% at iteration 200 falls to 67.59% by iteration 1,000, while test perplexity rises from 3.519 to 6.280.The authors identify early stopping as decisive in separating the generalizing model from the overfitted one.Source: §7 · p. 12

Seed Stability

• Standard deviations were 0.17 pp on competition accuracy and 0.18 pp on MATH-500 accuracy across five 200-iteration runs.The authors report that the narrow spread indicates the improvement is not an artifact of a favorable initialization.Source: §7 · p. 12

Token Budget Levels

• Removing roughly half the words (R1 to R2, 220 to 119.8 words) costs 7.5 accuracy points; cutting to R4 (55.6 words) costs 18.3 points.Below R4, the curve flattens, suggesting a shift to heuristic answer-guessing below approximately 50 words.Source: §7 · p. 12
• The two-person speed section loses 34.5 points from R2 to R6, versus 15 points for Freshman/Sophomore and 19 for Junior/Senior.Source: §7 · p. 12

Error Type Distribution

• Among 197 errors, 59.9% were reasoning failures, 32.0% extraction or formatting failures, and 8.1% rounding or type mismatches.The authors estimate roughly 40% of failures are formatting-related and suggest a structured parser or light post-processing could recover some cases.Source: §7 · p. 12

• The 4.76 pp competition gain is mirrored by a 3.0 pp gain on MATH-500, with five-run competition standard deviation of 0.17 pp.Algebra and number theory improve substantially, while geometry, precalculus, and intermediate algebra remain weak.Source: §8 · p. 13
• Accuracy falls 27.5 points from R1 (220 words, 69.43%) to R6 (31.2 words, 41.9%), with the sharpest decline between R3 and R4.The R3-to-R4 change is 96.5 to 55.6 words and 60.6% to 51.1%, indicating a practical lower bound of roughly 50–100 words for multi-step reasoning.Source: §8 · p. 13
• The two-person speed section is most sensitive to length reduction because its problems require carrying intermediate numerical results.Conceptual problems in Freshman/Sophomore and Junior/Senior are more robust; easier years also sustain accuracy at shorter budgets while harder years collapse sooner.Source: §8 · p. 13

• A single North American institution supplies the primary corpus, which may bias the model toward Western pedagogical conventions and reasoning styles.Using DeepSeek-R1 as both teacher and verifier may also introduce stylistic dependency in the training data.Source: §9 · p. 13
• At 69.43% accuracy, the model is not ready for fully autonomous educational use; the authors recommend human review before sharing outputs with students.Source: §9 · p. 13
• Tight response-length constraints substantially degrade accuracy on multi-step problems, a caution for inference-system configuration.Source: §9 · p. 13

• LoRA fine-tuning with early stopping improves Qwen2.5-7B from 64.67% to 69.43% ± 0.17%, while MATH-500 accuracy is 73.1% ± 0.18% versus 70.1% base.The DeepSeek-R1 dual-agent framework produces CoT training data with 91.40% verified accuracy.Source: §10 · p. 13
• At R6 (31.2 words, 41.9%), accuracy establishes a practical lower bound of roughly 50–100 words for multi-step competition-level reasoning, with the steepest drop between R3 and R4.Source: §10 · p. 14
• Future work will target larger, more diverse corpora, section-stratified sampling, and training on compressed CoT traces.The compressed-trace direction aims to help maintain accuracy under tighter token budgets.Source: §10 · p. 14
• Approximately 40% of failures stem from formatting rather than mathematical errors, making post-processing a potential path to further gains.Source: §10 · p. 14