PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Next-Chunk Reasoning RL Really Better than SFT? Revisiting Training Strategies under no-CoT Data

Yinhao Tang, Youqing Fang, Yanan Sun, Jiangning Liu, Ziyi Wang, Xun Zhao, Weiming Zhang, Bin Liu, Kuikun Liu, Wenwei Zhang, Kai Chen

arXiv:2608.23256v1cs.AI

TL;DR

The paper asks whether next-chunk reasoning RL’s gains reflect the RL formulation or simply better exposure to no-CoT data. It compares NCR with Mixed SFT in a controlled pre-RLVR setup and finds that Mixed SFT reaches a higher post-RLVR ceiling at over 60× lower training compute. The results also show that pre-RLVR accuracy is not a reliable proxy for post-RLVR performance.

  • Problem

    Existing evaluations mainly compare next-chunk reasoning RL with no-CoT-only SFT, leaving unclear whether gains arise from the RL formulation or from more effective exposure to no-CoT data.

  • Method

    The paper compares NTR, NSR, Sequential SFT, and Mixed SFT from a common base model before applying a shared RLVR stage.

  • Results

    Mixed SFT reaches the highest post-RLVR ceiling on both in-domain and out-of-domain reasoning benchmarks while requiring over 60× less training compute than next-chunk reasoning RL.

  • Takeaways & Limitations

    No-CoT training strategies should be judged by their post-RLVR performance rather than intermediate accuracy alone, because Mixed SFT has the lowest pre-RLVR accuracy but the highest final ceiling.

  • Takeaways & Limitations

    The study uses mathematical training data and benchmarks, so transfer to code, broader scientific reasoning, multilingual tasks, and other domains remains open.

Abstract

from arXiv · show

Recent work proposes next-chunk reasoning RL for leveraging no-CoT data---corpora such as worked solutions and textbook derivations that contain reasoning-rich content but lack explicit chain-of-thought annotations. The method trains a model to generate implicit reasoning traces and rewards them by their ability to predict the next chunk of text. While promising, existing evaluations primarily compare against conventional SFT baselines, leaving open whether the gains come from the RL formulation itself or from more effectively exposing the model to no-CoT data. We address this question with a controlled study of next-chunk reasoning RL and a simple but previously overlooked alternative: Mixed SFT, a single supervised fine-tuning stage that jointly trains on no-CoT and long-CoT data. Despite its simplicity, Mixed SFT achieves a clearly higher post-RLVR performance ceiling than next-chunk reasoning RL while requiring over 60 times less training compute. The advantage is consistent across in-domain mathematical reasoning and out-of-domain reasoning tasks. Moreover, we show that higher pre-RLVR accuracy does not necessarily translate into higher post-RLVR accuracy, highlighting the need to evaluate no-CoT training strategies in the context of the full post-training pipeline.

1 Introduction

The paper revisits whether next-chunk reasoning RL itself improves the use of no-CoT data, comparing it with Mixed SFT under a shared RLVR pipeline. Mixed SFT achieves the highest post-RLVR ceiling with substantially less compute, while the analysis questions intermediate accuracy as a reliable predictor.

  • Motivation: No-CoT corpora contain reasoning-relevant knowledge but lack explicit reasoning traces, creating a central question about how to convert them into useful training signal.The paper discusses worked solutions, textbook derivations, and research papers as examples of such data.
  • Research gap: Existing NCR comparisons mainly use no-CoT SFT baselines, leaving unclear whether reported gains come from RL or better exposure to no-CoT data.The paper argues that no-CoT-only SFT is not a suitable reference for the subsequent RLVR stage because it can disrupt the long-CoT format.
  • Approach: Mixed SFT jointly trains on no-CoT and long-CoT data in one stage, whereas Sequential SFT uses two stages and NCR applies rewards at token or sentence granularity.The comparison moves all strategies before a common RLVR stage and starts them from the same pretrained base model.
  • Findings: Over 60× less training compute accompanies Mixed SFT’s clearly higher post-RLVR ceiling on in-domain math and out-of-domain reasoning.The paper presents this as evidence that Mixed SFT is simpler, more effective, and substantially cheaper than next-chunk reasoning RL.
  • Findings: Mixed SFT has the lowest pre-RLVR accuracy yet the highest post-RLVR ceiling, showing that intermediate accuracy need not predict final pipeline performance.The authors therefore evaluate no-CoT strategies using the full post-training pipeline rather than only intermediate checkpoints.
  • Analysis: The analysis suggests NCR objectives can collapse into locally predictable completion, whereas Mixed SFT preserves no-CoT signal that RLVR can amplify.These mechanisms are offered as explanations for the different outcomes of the no-CoT training strategies.

2 Experimental Setup

The experiments use a unified base model, curated mathematical no-CoT and long-CoT data, five initialization strategies, and a common RLVR stage. Models are evaluated before and after RLVR on in-domain and out-of-domain reasoning benchmarks, with post-RLVR accuracy as the primary metric.

  • Models: Qwen3-30B-A3B-Base serves as the common initialization, avoiding prior post-training confounds from instruction-tuned or otherwise differing starting checkpoints.The authors choose a base checkpoint to ensure a clean, unified initialization.
  • Training data: The training corpus contains 152K correct long-CoT trajectories totaling approximately 1.95B tokens and 421K no-CoT solutions totaling approximately 0.53B tokens.Long-CoT data is generated with DeepSeek-V3.2 annotations, while no-CoT data comes from brief original AoPS solutions containing derivations without explicit reasoning traces.
  • Training strategies: The study compares NTR, NSR, Sequential SFT, Mixed SFT, and Reasoning SFT, then applies the same GRPO-based RLVR stage with exact-match rewards to every strategy.NTR and NSR are initialized from Reasoning SFT; a standalone no-CoT SFT stage is excluded because it disrupts the long-CoT format.
  • Evaluation: Post-RLVR accuracy is the primary metric across six in-domain competition-mathematics benchmarks and three out-of-domain reasoning benchmarks.The evaluation includes both pre-RLVR and post-RLVR checkpoints, with post-RLVR accuracy representing the ceiling enabled by each initialization.

3 Main Results

Mixed SFT outperforms next-chunk reasoning RL after RLVR across in-domain and out-of-domain reasoning while using substantially less compute. It also exceeds Sequential SFT, and pre-RLVR accuracy proves unreliable for predicting post-RLVR potential.

  • Mixed SFT outperforms next-chunk reasoning RL on both in-domain and out-of-domain post-RLVR benchmarks.
  • In-domain reasoning: 67.4 post-RLVR average accuracy gives Mixed SFT a 3.1-point lead over NTR and a 3.7-point lead over NSR across six in-domain benchmarks.
  • OOD generalization: 9.24, 60.98, and 75.84 are Mixed SFT’s highest post-RLVR accuracies on HLE, GPQA-Diamond, and MMLU-Pro, respectively.
  • Training efficiency: Over 60× more GPU hours are required by both NTR and NSR than by SFT on the same no-CoT data.NTR and NSR require online rollouts, reward computation, and policy optimization; NSR additionally uses a generative reward model.
  • Data combination: Mixed SFT achieves a higher final ceiling than Sequential SFT, showing that how no-CoT data is organized with reasoning data matters.Mixed SFT starts lowest before RLVR, trails early, then overtakes the other strategies and finishes highest on all three representative benchmarks.
  • Evaluation timing: 27.5 pre-RLVR accuracy for Mixed SFT rises to 61.1 post-RLVR accuracy, so intermediate accuracy does not reliably indicate final performance.Mixed SFT starts roughly 20 points below the other methods yet achieves a 33.7-point improvement, more than three times larger than any other method.

4 Analysis

The analysis finds that next-chunk reasoning RL does not add genuine reasoning gains over Mixed SFT, while Mixed SFT’s initial accuracy drop is transient and recoverable during RLVR. Sequential SFT instead loses no-CoT knowledge across stages.

  • NTR analysis: NTR’s high-entropy filter does not reliably identify reasoning-hard tokens, allowing reconstruction rewards without genuine reasoning.The analysis directly tests whether selected high-entropy tokens correspond to difficult reasoning targets.
  • NTR analysis: NTR outputs become shorter, more deterministic, and template-like during training, indicating local completion rather than long-horizon reasoning.NSR exhibits the same sentence-level tendency toward direct continuation of preceding text.
  • NTR analysis: Entropy-controlled NTR reaches a slightly lower post-RLVR ceiling than vanilla NTR on every benchmark.Preventing entropy collapse does not supply a better solution and therefore does not raise NTR’s ceiling.
  • NTR analysis: Adding NTR or NSR between Mixed SFT and RLVR leaves final performance almost unchanged, showing no additional reasoning ability that RLVR can amplify.This rules out the explanation that Mixed SFT wins only because it exposes the model to more no-CoT tokens.
  • Mixed SFT: Mixed SFT recovers from the lowest pre-RLVR accuracy to the highest post-RLVR ceiling because RLVR repairs a transient output-format mismatch.Joint training temporarily destabilizes reasoning markers and answer formatting, but format compliance rises during RLVR.
  • Sequential SFT: Sequential SFT’s second long-CoT stage partially overwrites no-CoT knowledge absorbed during its first stage, and RLVR does not recover the lost signal.A post-RLVR retention probe compares the two pipelines on problems drawn from the no-CoT training corpus.

5 Related Work

Related work has used SFT to inject instruction, short-response, and long-CoT data, while next-chunk reasoning RL adapts RL objectives to broader no-CoT text. This paper studies Mixed SFT as an initialization strategy for subsequent RLVR.

  • SFT for reasoning post-training: Reasoning-oriented SFT evolved from instruction-response tuning to explicit CoT training through bootstrapping, distillation, and large-teacher transfer.Examples include STaR, Distilling Step-by-Step, and DeepSeek-R1-Distill.
  • SFT for reasoning post-training: Recent methods mix long-CoT and non-CoT data within one SFT stage, but prior work generally applies such mixtures after reasoning-oriented RL or as standalone recipes.This paper instead examines raw no-CoT plus long-CoT Mixed SFT before a subsequent RLVR stage.
  • Next-chunk reasoning RL: Next-chunk reasoning RL converts no-CoT text into an RL-optimizable task by rewarding generated reasoning for predicting subsequent content.RPT uses prefix-matching rewards for next-token reasoning, while later work extends the objective to larger chunks or sentences.

6 Discussion and Conclusion

Across six in-domain and three out-of-domain reasoning benchmarks, Mixed SFT achieves the highest post-RLVR ceiling despite the lowest pre-RLVR accuracy, while requiring over 60× less training compute than next-chunk reasoning RL.

  • Discussion and Conclusion: Mixed SFT reaches the highest post-RLVR ceiling across in-domain and out-of-domain benchmarks while requiring over 60× less training compute.The comparison uses a unified RLVR budget and the same pre-trained base model.
  • Discussion and Conclusion: Next-chunk reasoning RL’s gains do not arise from genuine reasoning: its targets are largely locally predictable and its traces collapse into template-like completion.Mixed SFT preserves the no-CoT signal that RLVR later amplifies and avoids the cross-stage forgetting limiting Sequential SFT.
  • Discussion and Conclusion: Higher pre-RLVR accuracy does not imply higher post-RLVR accuracy, so no-CoT strategies should be evaluated through the full RLVR pipeline.For compute-constrained practitioners, the conclusion points toward data composition within SFT rather than additional RL stages.

A Limitations

The study uses a fixed no-CoT/long-CoT mixing ratio and evaluates mathematical no-CoT data and mathematical benchmarks, leaving broader recipe and domain transfer questions open.

  • Mixing ratio: The experiments use one fixed Mixed SFT mixing ratio and do not test how ratio changes affect post-RLVR ceilings, pre-RLVR drops, or knowledge-format trade-offs.A systematic sweep, including settings where one data source is scarce, remains future work.
  • Domain coverage: Both the no-CoT training corpus and evaluation benchmarks are mathematical, so transfer to code, broader scientific reasoning, multilingual tasks, and other domains remains untested.The paper leaves open whether Mixed SFT is equally effective when no-CoT data has different structural and token-level properties.

B Training Details

The study compares SFT and next-chunk reasoning objectives before a shared RLVR stage, using distinct data mixtures, reconstruction targets, rewards, and GRPO optimization.

  • The experiments use NVIDIA H200 GPUs, with 64 GPUs for every RL stage.
  • The long-CoT corpus contains 152K examples and roughly 1.95B tokens, while the no-CoT corpus contains 421K examples and roughly 0.53B tokens.
  • RLVR: All compared objectives are followed by the same RLVR stage, optimized with GRPO and rule-based exact-match rewards on DAPO-Math-17K.
  • Reasoning SFT uses long-CoT data, Sequential SFT uses two consecutive stages, and Mixed SFT unions no-CoT and long-CoT data in one stage.
  • NTR: NTR generates reasoning traces and continuations from entropy-filtered prefixes, rewarding byte-level prefix matches to the ground-truth continuation.
  • NSR: NSR generates reasoning traces and next-sentence predictions from sentence-segmented no-CoT data, using a generative reward model for binary semantic judgments.

C Evaluation Details

Evaluation uses exact-match answer extraction across competition mathematics and out-of-domain reasoning benchmarks, with avg@n as the main metric and pass@n reported separately.

  • Answers are extracted from \boxed{} or the final-answer span after </think>, then compared with canonical answers by exact match.
  • The evaluation covers AIME, HMMT, IMO-Answer, HLE, GPQA-Diamond, and MMLU-Pro, spanning mathematical, scientific, humanities, and multi-domain reasoning.
  • In-domain mathematics: AIME, HMMT, and IMO-Answer use 32 samples per problem and report avg@32.
  • Out-of-domain reasoning: HLE and MMLU-Pro use one sample per question and report pass@1, equivalent to avg@1.
  • Out-of-domain reasoning: GPQA-Diamond uses four responses per problem and reports avg@4.
  • The main table reports avg@n, while the appendix reports pass@n, the at-least-one-correct rate over the same samples.

D Pass@n Per-Benchmark Results

Pass@n results show a ceiling effect after RLVR, while Mixed SFT can exhibit format instability before RLVR despite retaining useful no-CoT mathematical knowledge.

  • Post-RLVR pass@n scores saturate, with top-performing methods reaching 93.33 on AIME 2024/2025 and compressing differences on HMMT and IMO-Answer.
  • Because pass@n is an upper-bound metric, avg@n remains the primary criterion for distinguishing expected per-sample quality across mature strategies.
  • Mixed SFT combines explicit long-CoT formatting with structurally varied no-CoT solutions, creating conflicting output conventions before RLVR.
  • Before RLVR, observed Mixed SFT failures include multiple <think> segments and direct answers that omit the <think> block.
  • These format failures disrupt reasoning and lower solution quality even though the underlying mathematical knowledge from no-CoT data remains intact.

F Full Per-Benchmark Results for the Mixed SFT Ablation

Across all eight benchmarks, inserting NTR or NSR between Mixed SFT and RLVR does not improve post-RLVR accuracy over applying RLVR directly to Mixed SFT.

  • Across all eight benchmarks, an intermediate NTR or NSR stage after Mixed SFT produces comparable or lower post-RLVR accuracy than direct RLVR.

G Entropy-Controlled NTR: Full Results

The entropy-controlled NTR variant applies stochastic filtering and adaptive advantage down-weighting to suppress entropy and length collapse, but its post-RLVR ceiling is lower than vanilla NTR across all evaluated benchmarks. The accompanying examples illustrate the reconstruction objectives and the risk that local completion can satisfy rewards without genuine reasoning.

  • Stochastic filtering of easy groups: Stochastic filtering retains rollout groups with probability 1 when their success rate is at most 0.25, then decreases retention linearly to 0.1 at a success rate of 1.The intervention targets already-easy prompts and avoids a hard threshold.
  • Adaptive down-weighting of positive advantages: When batch entropy is below 0.5, the entropy-controlled variant scales positive advantages by 0.75 while leaving negative advantages unchanged.This relatively increases the contribution of negative advantages to counter drift toward a single peaked template.
  • Results: Across all nine benchmarks, suppressing entropy and length collapse lowers the final RLVR ceiling rather than raising it.The entropy-controlled variant performs slightly worse than vanilla NTR on every benchmark.
  • Pre-RLVR comparison: NTR and NSR are evaluated from the same Reasoning SFT checkpoint during pre-RLVR training on AIME 2024, AIME 2025, and GPQA Diamond under a 32K-token context.NTR uses token-level prefix matching, whereas NSR uses a sentence-level generative judge.
  • Mechanistic interpretation: Representative NTR cases show that even high-entropy targets can often be solved by short local completion over preceding tokens rather than long-horizon inference.Such cases can satisfy the reconstruction reward without genuine reasoning.
  • Training examples: The examples distinguish long-CoT samples with step-by-step trajectories from no-CoT samples with compressed solutions, alongside NTR and NSR reconstruction prompts.The displayed prompts place intermediate thinking inside <think>...</think> tags and ask for token- or sentence-level continuation.
Loading 2608.23256v1…