PaperlayerView exact revision on arXiv

Source-linked AI summary

SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic Alignment in Reinforcement Learning

Zhuochun Li, Yuelyu Ji, Yiming Zeng, Daqing He

arXiv:2608.26550v1cs.CL

TL;DR

Reinforcement learning-based distillation must balance sparse rewards that inadequately guide multi-step reasoning against expensive neural PRMs. SPEAR addresses this with training-free symbolic milestone alignment for dense sequence-level rewards, and experiments report consistent gains across math, science, and commonsense tasks.

  • Problem

    Reinforcement learning-based distillation lacks an efficient reward that provides dense guidance for multi-step reasoning without the computational cost of neural PRMs.

  • Method

    SPEAR projects teacher rationales into domain-adaptive symbolic anchors and uses longest common subsequence alignment to reward ordered student reasoning paths.

  • Results

    Experiments across math, science, and commonsense benchmarks show SPEAR consistently outperforms baselines when integrated into standard reinforcement learning pipelines.

  • Takeaways & Limitations

    SPEAR provides dense process supervision with a favorable efficiency–performance tradeoff for sequence-level on-policy distillation.

  • Takeaways & Limitations

    Exact symbolic matching may under-reward semantically equivalent but structurally different reasoning, while rule-based extraction is sensitive to parsing and formatting errors.

Abstract

from arXiv · show

Reinforcement learning-based knowledge distillation has the potential to transfer complex reasoning from teacher to student models, yet it currently faces a critical dilemma: researchers must choose between sparse outcome-based rewards, which provide insufficient logical guidance, or expensive neural Process Reward Models (PRMs) for dense signals. We resolve this by introducing SPEAR (Symbolic Process Evaluation and Alignment Reward), a training-free and plug-and-play process reward method for sequence-level on-policy distillation. SPEAR projects natural-language reasoning traces into domain-adaptive symbolic milestones, providing an efficient proxy for process-level reasoning alignment. By utilizing the longest common subsequence (LCS) to align student explorations with teacher milestones, SPEAR provides a dense, order-aware reward signal that enforces logical consistency without the need for an external neural verifier. Our experiments across math, science, and commonsense reasoning tasks demonstrate that SPEAR effectively bridges the reasoning gap between student and teacher models via sequence-level distillation with efficient dense process rewards. Our code and data are available at: https://github.com/zhuochunli/SPEAR.

1 Introduction

The paper identifies inadequate process guidance and high verification costs as barriers to reinforcement learning-based distillation, then introduces SPEAR as a symbolic, dense-reward solution. SPEAR aligns domain-adaptive teacher milestones with student trajectories and improves reasoning distillation across math, science, and commonsense benchmarks.

  • Off-policy distillation can encourage style mimicry and exposure bias, causing failures when students encounter inference errors absent from training data.
  • Sparse outcome rewards provide insufficient guidance for multi-step logic, whereas dense PRMs or token-level divergence impose substantial computational overhead.
  • Existing process-supervision methods mainly target formally verifiable mathematics and programming, leaving scientific and commonsense reasoning difficult to supervise.
  • SPEAR projects teacher reasoning traces into domain-adaptive symbolic trajectories and uses LCS to align student paths with teacher milestones.The framework supports computational processes for math, causal dependencies for science, and entity state-transitions for commonsense reasoning.
  • SPEAR is training-free and enables efficient sequence-level on-policy distillation without the computational burden of process reward models.
  • Experiments across math, science, and commonsense benchmarks show that integrating SPEAR consistently outperforms baselines.The reported results support separating logical acquisition from linguistic expression.

2 Related Work

RLVR has become a primary approach for inducing multi-step reasoning, while recent policy-optimization refinements improve training efficiency and stability. However, these methods typically rely on sparse binary outcome rewards, which can encourage redundant-token overthinking.

  • RLVR has become a primary driver for inducing multi-step reasoning behaviors in large language models.
  • GRPO refinements such as Dr. GRPO and DAPO improve the efficiency and stability of reinforcement-learning reasoning training.Dr. GRPO addresses length-related optimization bias, while DAPO uses dynamic trajectory reweighting to reduce sampling and length-induced biases.
  • Sparse binary outcome rewards can lead models to generate thousands of redundant tokens without valid logical progress.The passage describes this behavior as overthinking, in which models maximize rewards without making valid logical progress.

3 Method

SPEAR reframes reasoning distillation as alignment between domain-adaptive symbolic milestones rather than token-level imitation. It combines order-aware LCS process rewards with accuracy and format gating to provide dense supervision during on-policy reinforcement learning.

  • 3.2 Domain-Adaptive Symbolic Anchors: SPEAR represents reasoning traces as sequences of symbolic milestones, allowing students to align with teacher logic while retaining flexibility in linguistic expression.The projection maps natural-language traces into low-dimensional symbolic trajectories grounded in domain-specific structure.
  • 3.2.1 Formal Logic and Mathematical States: Math anchors capture symbolic expressions and assignments while excluding bare numbers, emphasizing ordered derivations rather than constant overlap.The extraction focuses on transitions such as isolating variables before substitution.
  • 3.2.2 Causal Dependency Anchors for Science: Science anchors use verb-centered dependency tuples to preserve directional relations between entities, while uniqueness constraints prevent repetitive reasoning loops.Each tuple represents a governing verb with its subject and object when available.
  • 3.2.3 Open-Domain State Transitions: Commonsense anchors encode State-Action transitions from noun phrases and syntactic heads, capturing chronological changes while ignoring modifiers and tense variation.These anchors function as logical key-frames for tracking evolving world states.
  • 3.3 Sequential Alignment via LCS-F1: LCS-F1 aligns student and teacher milestone sequences by order, balancing precision against spurious anchors with recall against omitted teacher milestones.Dynamic programming makes the alignment computable during real-time rollouts, and the resulting reward penalizes logical reversals.
  • 3.4 Gated Composite Reward Formulation: The composite reward gates accuracy and reasoning alignment by format compliance, while partial milestone credit supplies dense shaping when the final answer is incorrect.The phased structure checks required reasoning and answer blocks before applying the weighted reward components.

4 Experiments

The experiments evaluate SPEAR across mathematical, scientific, and commonsense reasoning using two student architectures and comparisons against SFT and outcome-based RL baselines. Results show stronger and more consistent gains on mathematics, while benefits vary by task and baseline.

  • Experimental Setup: Experiments cover GSM8K and MATH for mathematics, GPQA for science, and CommonsenseQA for commonsense reasoning.The study uses Llama-3-8B-Instruct and Qwen3-4B as student models, with DeepSeek-V3.2 as teacher.
  • Baselines: The comparison includes SFT distillation, GRPO, Dr. GRPO, DAPO, and Logic-RL baselines.The RL baselines primarily use sparse outcome-based rewards, while Logic-RL provides partial format or answer rewards.
  • Main Results: Distillation generally improves base performance across both student architectures, but its effect is task-dependent.Mathematical reasoning shows the most consistent gains, whereas improvements are smaller for already strong or near-saturated models.
  • Main Results: Qwen3-4B falls from 83.95% to 80.90% and Llama3-8B-Instruct from 75.67% to 74.75% on CommonsenseQA under SFT.The reported drops indicate that directly learning teacher natural-language rationales is not uniformly beneficial for commonsense reasoning.
  • Main Results: 33.83% on GPQA lets SFT for Llama-3-8B-Instruct exceed all three plain RL baselines.The paper attributes this pattern to scientific multiple-choice tasks where direct explanation imitation can transfer domain knowledge beyond outcome-only exploration.
  • SPEAR Advantages: SPEAR consistently improves over the baselines across tasks, student models, and RL baselines, with the clearest gains on mathematics.CommonsenseQA shows the smallest improvement, and format- or final-answer-based partial rewards are reported as insufficient for reliable distillation.

5 Discussion

SPEAR offers competitive performance with substantially lower process-supervision cost than neural PRMs, while ablations show that its reward components and task-specific symbolic alignment matter. The qualitative case study illustrates how sequence-aware matching supplies feedback that accuracy alone misses, and moderate cold-start SFT improves subsequent RL.

  • Performance–efficiency analysis: SPEAR remains competitive with neural PRMs across MATH and GPQA while using only lightweight spaCy/LCS processing instead of an additional billion-parameter model.SPEAR is within 1–2% of the math-specialized PRM across three RL backbones and outperforms it on GPQA by an average of 3.87%.
  • Reward-function ablation: Removing accuracy reward causes the largest ablation drop, including MATH performance falling from 30.00 to 25.40.Removing the format gate also consistently lowers performance, while outcome-only supervision is insufficient on both benchmarks.
  • Reward-function ablation: MATH performs best at λ = 0.5 and GPQA at λ = 0.75, whereas λ = 1.0 degrades performance.The results indicate that reasoning-alignment weight must be tuned by task rather than maximized universally.
  • Reward-function ablation: Replacing symbolic anchors with raw-token LCS or removing order information with Jaccard overlap causes around 2% performance drops.These ablations support combining abstraction through symbolic anchors with sequential, order-aware alignment.
  • Reward-function ablation: Generic surface markers reduce accuracy by 2.22% on average, showing that domain-aware milestone extraction supplies more informative reasoning signals.The paper contrasts task-specific extraction with markers such as “however,” “wait,” and “thus.”
  • Cold-start SFT: A moderate 30% SFT cold start yields the best subsequent SPEAR-based RL performance, while larger SFT fractions gradually reduce it.The study uses Qwen3-4B on MATH, progressively increasing SFT data before applying GRPO with SPEAR.
  • Qualitative case study: In the GPQA case study, LCS matches two ordered teacher–student anchors and penalizes a mismatched third step despite identical final answers.The combined reward therefore distinguishes imprecise reasoning terminology that accuracy reward alone treats identically.

6 Conclusion

The paper introduces SPEAR as a training-free reward framework for sequence-level on-policy distillation and evaluates it across math, science, and commonsense reasoning. Its conclusion emphasizes efficient dense supervision and the importance of balanced reasoning weight, order-aware symbolic alignment, and moderate SFT initialization.

  • SPEAR provides dense process supervision with a favorable efficiency–performance tradeoff across math, science, and commonsense tasks.
  • Balanced reasoning weight, order-aware symbolic alignment, and a moderate SFT cold start are important for effective distillation.

Limitations

SPEAR’s symbolic anchor extraction and exact subsequence matching may miss semantically equivalent reasoning paths. It also assumes that teacher-generated rationales reliably represent reasoning trajectories, despite potentially redundant or suboptimal intermediate steps.

  • Symbolic anchor extraction and exact subsequence matching may fail to capture semantically equivalent but structurally different reasoning trajectories.
  • Valid alternative reasoning paths that diverge from the teacher’s symbolic ordering may receive lower rewards, especially in open-ended reasoning tasks.
  • SPEAR assumes teacher-generated rationales provide reliable reasoning trajectories, although language-model explanations may contain redundant or suboptimal intermediate steps.

Ethics Statement

The authors report limited AI-assistant use for text polishing and code implementation, while retaining responsibility for the research. They also describe dataset splits, compute, model access, and hyperparameter selection procedures.

  • Ethics Statement: AI Assistants were used only for minor text polishing and part of code implementation.The authors state that ideas, experiments, analyses, and discussions were conducted solely by them.
  • Data and evaluation: Datasets were sourced from Hugging Face or official repositories, with benchmark-specific training and test splits.MATH uses its official training set and MATH-500 for testing, while GPQA uses gpqa_main and gpqa_extended for training and gpqa_diamond for testing.
  • Model selection: Hyperparameters were tuned on 5% held-out development subsets, while benchmark evaluation splits remained reserved for final evaluation.The tuning procedure used random seed 731 and varied λ over {0, 0.25, 0.5, 0.75, 1.0}.

D Analysis of LCS-F1 Alignment

SPEAR’s LCS-F1 reward balances matching teacher milestones with avoiding unsupported student anchors. The analysis shows why precision or recall alone permits degenerate trajectories, while domain-specific extraction produces ordered symbolic trajectories for scoring.

  • LCS-F1 design: LCS-F1 combines alignment precision and teacher-milestone recall to penalize unsupported student anchors and missing teacher anchors.Precision is |LCS|/|A_s|, while recall is |LCS|/|A_t|.
  • Degenerate strategies: Precision alone rewards a short matching prefix, whereas recall alone rewards verbose trajectories that include unsupported anchors.The teacher trajectory example uses A_t = [a1, a2, a3] to illustrate both failure modes.
  • Interpretation: The harmonic mean reserves full credit for complete, concise alignment rather than incomplete or verbose matching.Table 9 contrasts precision, recall, and LCS-F1 using unsupported anchors x, y, and z.
  • Trajectory construction: Algorithm 2 projects teacher and student responses into ordered symbolic trajectories before passing them to the LCS-F1 calculation.The extraction procedure includes domain-specific rules for expressions, governing verbs, noun chunks, and appended anchors.

F Cross-Benchmark Transfer to AlpacaEval 2.0

SPEAR transfers beyond its original reasoning benchmarks using the unchanged commonsense state-transition extractor on AlpacaEval 2.0. The extractor improves several RL baselines and outperforms Logic-RL across the tested optimization frameworks.

  • Evaluation: AlpacaEval 2.0 evaluates Qwen3-4B with the official length-controlled win rate using the unchanged commonsense state-transition extractor.The experiment tests whether the open-domain extractor generalizes to instruction following.
  • Results: SPEAR also outperforms the Logic-RL reward under each tested optimization framework.The authors interpret these results as evidence that the extractor captures reusable structure rather than a benchmark-specific heuristic.

G Measured Resource and Training Overhead

SPEAR has low reward-scoring overhead because symbolic extraction and LCS-F1 matching run on CPU without a neural reward model. Compared with a neural PRM, it avoids extra GPU memory, rollout inference, and separate process-labeled supervision.

  • Resource requirements: SPEAR adds no neural reward model, running spaCy extraction and LCS-F1 matching on CPU with negligible additional GPU memory overhead.PRM scoring instead retains an additional 7B reward model in GPU memory and performs one extra transformer pass per rollout.
  • Training overhead: SPEAR derives rewards directly from teacher trajectories, whereas neural PRM training generally requires separately process-labeled supervision.PRM800K is cited as containing 800K step-level annotations.
  • Measured overhead: 30.04% versus 1.87% is the measured per-step time increase for Qwen2.5-Math-PRM versus SPEAR.The complete SPEAR run took 23 hours and 42 minutes with 76.59 GB peak memory per GPU.
Loading 2608.26550v1…