PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Complete Reasoning Traces for Post-Training

Jaehui Hwang, Sangdoo Yun, Byeongho Heo, Dongyoon Han

arXiv:2609.07103v1cs.CL

TL;DR

The paper investigates whether LLMs need complete reasoning traces during post-training, since intermediate trace segments may be redundant. It analyzes attention and segment removal, then shows that endpoint-focused training improves SFT and also benefits RL and on-policy distillation.

  • Problem

    It is underexplored whether LLMs benefit from learning complete, often lengthy reasoning trajectories during post-training.

  • Method

    The paper combines pilot trajectory truncation, attention analysis, controlled segment removal, and endpoint-focused training across SFT, RL, and on-policy distillation.

  • Results

    Endpoint-focused training improves reasoning performance, while masking middle segments also improves RL and on-policy distillation; step-level masking gains +2.2 versus +0.7 for token-level masking on average.

  • Takeaways & Limitations

    Intermediate reasoning segments are often redundant, so retaining trajectory endpoints can provide effective post-training supervision across multiple training paradigms.

  • Takeaways & Limitations

    Performance degrades with excessive truncation, while overly conservative removal limits benefits; approximately 20% filtering is a heuristic operating point.

Abstract

from arXiv · show

Large language models (LLMs) are often post-trained on pre-collected reasoning trajectories to improve their reasoning capability. Such trajectories tend to be long due to complex, interwoven paths, which often include detours on the path toward the answer. However, it has been underexplored whether LLMs indeed benefit from learning complete trajectories in post-training, such as supervised fine-tuning (SFT). Starting from our pilot study, we find that full trajectories provide only limited benefit, while partial trajectories are effective even under heavy truncation. We analyze redundancy in reasoning trajectories through attention-based analyses and controlled token-removal studies, both of which show that intermediate tokens contribute minimally to final reasoning quality. This suggests that avoiding redundant information may allow LLMs to internally infer coherent alternatives by inferring missing steps from their internal knowledge, given known trajectory endpoints. Furthermore, we show that training LLMs using endpoints leads to consistent changes in reasoning behavior, and that it also benefits post-training methods based on reinforcement learning or on-policy distillation, highlighting the need to revisit complete reasoning traces. Code is available at https://github.com/naver-ai/revisiting-trace.

1 Introduction

The paper asks whether complete reasoning trajectories are necessary for effective post-training, given that LRM traces can be lengthy, uninformative, and misleading. It finds that transitional segments are often redundant and that endpoint-focused training can improve post-training across SFT, RL, and distillation.

  • Complete reasoning trajectories may be unnecessary because LRM traces are often lengthy, less informative, and sometimes misleading.
  • The paper studies whether coarse trimming of contiguous reasoning segments enables more effective SFT with fewer training traces.
  • Answer tokens place little attention on transitional steps, and skipping those steps preserves answer quality.
  • Endpoint-focused training allows pretrained LLMs to infer missing transitional reasoning from internal knowledge and known trajectory endpoints.
  • The proposed endpoint-focused approach outperforms several competitive filtering methods on OpenThoughts and s1K, and extends beyond SFT to RL and distillation post-training.

2 Related work

Related work shows that reasoning quality does not always increase with longer chains or all reasoning tokens. Existing approaches compress or selectively weight reasoning, while this paper contrasts them with endpoint-based trajectory trimming.

  • Longer reasoning chains can hurt accuracy beyond a threshold because of overthinking, and the shortest sampled chain is often more reliable than the longest.
  • Selective token-importance methods focus training on high-utility reasoning tokens, improving data efficiency through compacted or selectively supervised reasoning.
  • This paper differs from token-scoring, budgeted-RL, and guidance-policy approaches by trimming contiguous trajectory segments and retaining endpoints.

3 Background

The paper studies SFT from machine-generated reasoning traces and tests whether retaining trajectory endpoints while removing intermediate segments improves training. Its pilot study finds that prefix-plus-suffix trajectories outperform full, prefix-only, and suffix-only variants, motivating broader segment analyses.

  • 3. Background: SFT trains a smaller model on teacher-generated input, answer, and reasoning-trace tuples by minimizing their negative log-likelihood.
  • 3. Background: The problem setting focuses on post-training approximately 32B-parameter LLMs with pre-collected traces, while treating SFT as a practical alternative or precursor to RL.
  • 3.2 Pilot Study: Are All Traces Performance-Relevant?: The pilot compares complete, prefix-only, suffix-only, and prefix-plus-suffix trajectories using s1K-1.1 across AIME24, GPQA-D, and MATH500.
  • 3.2 Pilot Study: Are All Traces Performance-Relevant?: Prefix-plus-suffix trajectories consistently outperform prefix-only and suffix-only variants and achieve higher average performance than full trajectories.
  • 3.2 Pilot Study: Are All Traces Performance-Relevant?: Coarse truncation can remain effective without hand-tuned or complicated filtering methods.
  • 3. Background: The subsequent analysis investigates why removing substantial transitional segments can improve SFT and motivate a more effective training paradigm.

4 Revisiting Reasoning Trajectories

The paper finds that intermediate reasoning segments contribute less than trajectory endpoints and introduces endpoint-based SFT, which retains beginning and ending segments while removing redundant middle content. Analyses and experiments show that this strategy can match or outperform full-trajectory SFT and extend to other post-training objectives.

  • Attention weight analysis: Attention peaks at trajectory beginnings and endings, while intermediate steps receive weaker attention during answer generation.Layer-wise patterns shift from candidate answers in early layers to joint problem-and-reasoning references, then toward the beginning for answer formatting and consolidation.
  • Segment ablation analysis: Removing intermediate segments yields the highest similarity to full-trajectory answers, whereas removing beginning or ending segments lowers similarity.The ablation compares beginning, centered intermediate, and ending spans without fine-tuning.
  • Answer perplexity analysis: Replacing intermediate steps decreases answer perplexity, while replacing beginning or ending steps increases it.This indicates that intermediate steps provide limited supervision for predicting final answers and may be self-filled by the model.
  • Learning from trajectory segments: Preserving both prefix and suffix segments consistently outperforms retaining only one endpoint and can outperform full-trajectory SFT.These findings motivate Endpoint-based SFT, which retains trajectory beginnings and endings while omitting redundant intermediate segments.
  • Additional results on general benchmarks: E-SFT generally outperforms standard SFT on language-centric and code-generation benchmarks despite domain-specific SFT degrading general-language performance.The results suggest that skipping redundant intermediate reasoning steps helps preserve general linguistic ability during domain-specific fine-tuning.
  • Comparison with filtering methods: Resource-intensive filtering methods do not surpass E-SFT, while similarity-based filtering can remove important early definitions or problem setups.LLM-based compression also performs worse and requires extra computation, whereas E-SFT provides a simpler filtering design.
  • Beyond SFT: Masking the middle part improves performance beyond GRPO, and masking reasoning steps outperforms masking individual tokens in on-policy distillation.The step-level mask improves average performance by +2.2 versus +0.7 for the token-level mask.

5 Why Does the Proposed Method Work?

E-SFT improves reasoning by reducing redundant detours and helping models complete missing reasoning steps. Training with fewer, more informative tokens also lowers perplexity relative to standard SFT.

  • Generation behavior: E-SFT suppresses unnecessarily lengthy reasoning detours, enabling more accurate solutions with fewer tokens.In another example, E-SFT succeeds where standard SFT fails despite both producing the same intermediate equations.
  • Reasoning quality: 22% of E-SFT trajectories were preferred over original s1K-1.1 trajectories, compared with 8% for the original data, while 70% were judged equivalent.The evaluation compared unnecessary repetition and smoothness of logical flow using GPT-OSS-120B.
  • Token-length patterns: E-SFT adapts reasoning length to baseline trajectories: it shortens outputs longer than 15k tokens and increases token usage for shorter trajectories.This pattern accompanies the reported preference results for E-SFT trajectories.
  • Training dynamics: E-SFT achieves consistently lower perplexity during training while using fewer tokens than standard SFT.The authors interpret this as evidence that intermediate segments are redundant and make optimization harder.

6 Conclusion

The paper argues that complete reasoning trajectories are not always necessary for post-training. Endpoint-based supervision improves reasoning under SFT and also benefits reinforcement-learning and distillation objectives.

  • Conclusion: Full trajectories, especially intermediate segments, provide no clear SFT benefit because those segments are redundant for training.The conclusion motivates skipping learning signals from the middle of trajectories.
  • Conclusion: E-SFT trains on incomplete trajectories by retaining endpoint segments while skipping the middle reasoning segments.This approach is presented as an effective SFT method.
  • Conclusion: Masking middle segments also improves reasoning performance under GRPO and on-policy distillation.The conclusion extends the finding beyond standard supervised fine-tuning.

A Discussion and Limitation

E-SFT’s effectiveness depends on model capability and the amount of retained trajectory. The discussion identifies an operating range for truncation and relates redundant middle content to weak attention in long contexts.

  • Model scale: E-SFT benefits appear only above roughly 60–65 base performance on s1K-1.1, while OpenThoughts3-100K lowers this threshold for weaker models.The reported relationship is between general benchmark performance and reasoning gains from E-SFT over standard SFT.
  • Model scale: Models need sufficient baseline capacity to infer trimmed reasoning steps; weaker models gain little or nothing from E-SFT.The authors connect this boundary to the need to complete missing intermediate reasoning.
  • Truncation sensitivity: Excessive truncation degrades performance, whereas conservative truncation limits potential gains, implying an appropriate operating range.The retained beginning and ending segments are varied to study this sensitivity.
  • Truncation sensitivity: Filtering approximately 20% of the original reasoning trajectories provides a practical trade-off that generalizes across datasets and step- or token-based cutoffs.The authors describe this as a simple heuristic rather than a principled cutoff rule.
  • Attention analysis: E-SFT’s redundant middle segments and the “Lost in the Middle” effect may share insufficient attention to middle tokens, despite differing mechanisms.The paper characterizes its own mechanism as redundancy-driven attention drop rather than attention-driven failure.

B.1 Details of Segment Ablation Analysis

The analysis removes beginning, middle, or ending trajectory segments at varying ratios and examines whether double-newline boundaries represent meaningful reasoning steps. It also characterizes the functions and limitations of the trimmed middle region.

  • Segment ablation analysis: Segment ablation removes beginning, intermediate, or ending portions at controlled ratios to test their effects on answer generation.The study uses complete trajectories from 100 GPQA-D problems and varies the ablation ratio, such as 10% or 20%.
  • Step and block construction: Reasoning steps are defined by splitting trajectories at double newline characters, with context-aware paraphrasing applied to selected center blocks.The compression procedure exposes three preceding and three following blocks while shortening only a 10-block center window.
  • Step and block construction: 33.5% and 28.0% of sampled positions marked semantic transitions across double newlines, versus 2.5% and 1.0% within steps.These figures correspond to s1K-1.1 and OpenThoughts3, respectively, supporting double newlines as useful but imperfect semantic boundaries.
  • Functional composition: Planning concentrates in retained early regions, while conclusion concentrates in retained late regions; the trimmed middle contains the highest deduction shares, 54% and 39%.The middle also contains more backtracking, at 14% and 13%, whereas removing either end would discard setup or final-answer content.
  • Functional composition: The functional analysis does not establish whether individual steps are redundant, but frequent deduction and backtracking make the middle more likely to contain revised or discarded exploratory attempts.The authors frame this as a characterization of regional composition rather than a direct redundancy test.

C.3 Implementation details and experimental settings

E-SFT preserves the first and last trajectory segments while removing the middle, using dataset-specific retained lengths and evaluating models across three reasoning benchmarks.

  • E-SFT construction: E-SFT preserves the first and last n reasoning steps and removes the middle total steps −2n steps.The setting uses n=100 for s1K-1.1 and n=200 for OpenThoughts3.
  • E-SFT construction: The retained lengths remove about 20% of tokens in both datasets, based on average trajectory lengths of 234 and 461 steps.The corresponding values are n=100 for s1K-1.1 and n=200 for OpenThoughts3.
  • Evaluation: Evaluation uses AIME24, GPQA-D, and MATH with temperature 0.6, three generations per instance, and averaged results.The detailed results are organized by task, model, and method.

C.4 Detailed results of E-SFT and Llama family

Detailed experiments report strong E-SFT performance and examine its transfer across model families, while additional analyses relate gains to pretrained model performance and use external judging of reasoning quality.

  • Detailed results: E-SFT consistently achieves strong performance across the detailed task, model, and method results.The reported benchmarks are AIME24, GPQA-D, and MATH.
  • Llama-family results: E-SFT maintains performance on Llama-3.2-3B-Instruct and improves Llama-3.1-8B-Instruct by +1.34% on average.These results use the OpenThoughts3-100K dataset and suggest transfer beyond Qwen-based models without performance degradation.
  • Conditions for effectiveness: The experiments compare E-SFT gains over full-trajectory SFT with general task performance across Qwen and OpenThoughts3 model settings.The comparison examines how model capacity and initialization relate to observed gains.
  • Reasoning-quality evaluation: An external GPT-OSS-120B judge compares paired reasoning responses in both presentation orders and treats order-switched preferences as equivalent.The prompt evaluates naturalness, unnecessary repetition, and smoothness while holding the final answer constant.

C.7 Experimental details of §4.4

The experiments compare E-SFT with reinforcement learning and on-policy distillation under specified training settings, reporting averaged accuracy across three benchmarks.

  • Training settings: GRPO and OPD are each trained for 100 steps with a learning rate of 1e-6.The settings differ in batch construction, rollout or generation count, context length, and loss configuration.
  • Evaluation: Decoding performance is summarized as averaged accuracy across AIME24, GPQA-D, and MATH.The table reports performance for E-SFT decoding experiments.

D Decoding with E-SFT for efficient inference

The method truncates the middle of a generated reasoning trajectory before answer generation, preserving its beginning and ending while reducing inference cost. Experiments report practical speed improvements, including a 4.5% reduction in GPQA Diamond evaluation time.

  • Method: The model generates the trajectory up to answer generation, then removes a selected middle segment before producing the final answer.The evaluated removal ratios are 25%, 33%, 50%, and 75%.
  • Method: For r=0.5, the method removes the central 50% of tokens while preserving equal beginning and ending fractions.The retained portions provide both the initial problem setup and concluding consolidation region.
  • Efficiency: Middle truncation shortens the effective context seen by answer tokens and downstream modules, reducing computation for long-form reasoning.GPQA Diamond trajectories typically contain around 8.1k intermediate tokens, which a naive two-stage pipeline re-prefills in full.
Loading 2609.07103v1…