PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Step Length Confounding in LLM Reasoning Data Selection

Bing Wang, Rui Miao, Chen Shen, Shaotian Yan, Kaiyuan Liu, Ximing Li, Xiaosong Yuan, Sinan Fan, Jun Zhang, Jieping Ye

arXiv:2604.06834v1cs.CLcs.AI

TL;DR

Naturalness-based selection for long-CoT reasoning data systematically favors longer reasoning steps, because low-probability first tokens are diluted in longer steps. The paper introduces ASLEC-DROP and ASLEC-CASL to correct this bias, and experiments show consistent improvements over existing naturalness-based selection across models and benchmarks.

  • Problem

    Naturalness-based selection for long-CoT datasets prefers samples with longer reasoning steps rather than higher-quality ones, exposing a step length confounding problem.

  • Method

    ASLEC-DROP drops first-token probabilities, while ASLEC-CASL applies causal debiasing regression to remove their confounding effect from selection scores.

  • Results

    Across four LLMs and five evaluation benchmarks, the proposed methods consistently outperform existing naturalness-based selection and mitigate step length confounding.

  • Takeaways & Limitations

    Correcting first-token effects improves naturalness-based selection for long-CoT reasoning SFT data within the evaluated datasets, models, and benchmarks.

  • Takeaways & Limitations

    Whether deeper confounding factors exist and whether on-policy data selection remains correlated with response length require further exploration.

Abstract

from arXiv · show

Large reasoning models have recently demonstrated strong performance on complex tasks that require long chain-of-thought reasoning, through supervised fine-tuning on large-scale and high-quality datasets. To construct such datasets, existing pipelines generate long reasoning data from more capable Large Language Models (LLMs) and apply manually heuristic or naturalness-based selection methods to filter high-quality samples. Despite the proven effectiveness of naturalness-based data selection, which ranks data by the average log probability assigned by LLMs, our analysis shows that, when applied to LLM reasoning datasets, it systematically prefers samples with longer reasoning steps (i.e., more tokens per step) rather than higher-quality ones, a phenomenon we term step length confounding. Through quantitative analysis, we attribute this phenomenon to low-probability first tokens in reasoning steps; longer steps dilute their influence, thereby inflating the average log probabilities. To address this issue, we propose two variant methods: ASLEC-DROP, which drops first-token probabilities when computing average log probability, and ASLEC-CASL, which applies a causal debiasing regression to remove the first tokens' confounding effect. Experiments across four LLMs and five evaluation benchmarks demonstrate the effectiveness of our approach in mitigating the step length confounding problem.

1 Introduction

The paper identifies step length confounding in naturalness-based selection for long-CoT reasoning data: average log probability favors longer reasoning steps rather than higher-quality samples. It proposes ASLEC-DROP and ASLEC-CASL, which intervene on first-token probabilities, and evaluates them across multiple models and benchmarks.

  • Long-CoT SFT datasets can contain incorrect reasoning steps and overly complex trajectories, motivating data-selection methods for higher-quality subsets.
  • Naturalness-based methods systematically prefer samples with longer reasoning steps rather than higher-adaptability samples, a phenomenon termed step length confounding.
  • Longer steps dilute low-probability first tokens, increasing overall step log probability and making longer-step examples more likely to be selected.
  • ASLEC-DROP removes first-token probabilities from the global average log probability, while ASLEC-CASL uses regression to remove their confounding effect.
  • Across four LLMs and five evaluation benchmarks, ASLEC-DROP and ASLEC-CASL consistently outperform naturalness-based selection and mitigate step length confounding.

2 Preliminary Experimental Analysis on Step Length Confounding

Naturalness-based selection methods consistently prefer reasoning samples with longer steps, creating step length confounding. The analysis attributes this pattern to low-probability first tokens whose influence is diluted in longer steps.

  • Experimental Setup: The experiments analyze naturalness-based selection using log probabilities, perplexity, and local log probabilities across generated reasoning responses.The setup evaluates responses generated by four long-CoT LLMs and selects data using representative naturalness-based criteria.
  • Observed Confounding: Naturalness-based methods consistently select samples with longer reasoning steps rather than higher-adaptability samples.Selected responses have longer step lengths, while unselected responses average approximately 30 tokens per step.
  • Observed Confounding: Longer reasoning steps receive higher average step-level log probabilities, showing a monotonic relationship between step length and score.The relationship is measured using Qwen3-4B-Base on steps produced by different LLMs.
  • Cause Analysis: Each reasoning step begins with a consistently lower-probability first token, and longer steps reduce that token’s proportion of the overall score.The first token often branches among alternative reasoning paths, producing higher entropy and lower log probability.
  • Cause Analysis: Step length confounding occurs because diluted first-token penalties inflate average log probabilities, making longer-step samples more likely to be selected.The paper also reports a weaker correlation between selection and total response length than between selection and step length.

3 The Proposed Method

The proposed ASLEC method addresses step length confounding by intervening on first-token probabilities. It offers a direct first-token-dropping variant and a causal-debiasing variant that removes the estimated confounding effect.

  • Method Overview: ASLEC defines selection metrics for choosing responses aligned with the trained reasoning LLM while avoiding step-length confounding.Candidate responses can differ in correctness, reasoning quality, verbosity, and step length.
  • ASLEC-DROP: ASLEC-DROP computes the geometric-mean probability after dropping the first token of each reasoning step.This directly mitigates the bias but discards potentially informative first-token signals.
  • ASLEC-CASL: ASLEC-CASL models the raw log-probability score with first-token terms and a confounding factor defined as the first-token proportion.The parameters are estimated by ordinary least squares, with residual noise represented separately.
  • ASLEC-CASL: ASLEC-CASL removes the estimated confounder effect from each instance’s raw score to obtain a deconfounded metric for downstream selection.The final score is computed as the raw score minus the estimated coefficient times the confounding factor.

4 Experimental Evaluation

Experiments across two reasoning SFT datasets, four source and target LLM settings, and five benchmarks show that ASLEC variants outperform naturalness-based selection while reducing step-length disparities.

  • Main Results: 6.28% and 9.08% average accuracy gains over Local LP were achieved on the two datasets by both ASLEC variants.The experiments compare against the SOTA naturalness-based Local LP method.
  • Main Results: ASLEC-CASL consistently outperformed ASLEC-DROP, including in cross-domain evaluation on GPQA after training on LIMO-v2.The result is attributed to preserving informative first-token probability patterns through causal debiasing.
  • Main Results: The performance gains were larger on LIMO-v2 than AceReason-1.1-SFT, indicating particular effectiveness when data or model capacity is limited.The authors connect this setting to the stronger negative effect of low-quality samples on generalization.
  • Performance on Alleviating Confounding: ASLEC-DROP and ASLEC-CASL produced markedly smaller selected-versus-unselected step-length disparities than prior approaches.Both methods intervene directly on first-token probabilities, supporting a link between those probabilities and step-length confounding.
  • Comparing Min and Max Probabilities: Selecting samples with the highest scasl values outperformed selecting samples with the lowest values across the evaluated models and benchmarks.This preserves the naturalness-based preference for samples better aligned with target-model capabilities after debiasing.
  • Linear Regression Results: The largest fitted γ was -1.284, corresponding to a 6.22% reduction in per-token probability for a 0.05 first-token-ratio difference.The all-data regression yielded γ = -0.680, corresponding to a 3.34% reduction.
  • Implementation: ASLEC-DROP adds no computational overhead, while ASLEC-CASL uses a lightweight regression that typically fits in a few seconds.The regression has only a small number of parameters and imposes negligible overall pipeline cost.

5 Related Works

Related work develops long-chain-of-thought data through supervised fine-tuning and increasingly uses model confidence for naturalness-based selection alongside heuristic rules.

  • Long-CoT Data Construction: Supervised fine-tuning on large, diverse datasets generated by stronger LLMs has become a standard route to eliciting long-chain-of-thought reasoning.Existing pipelines construct broad question sets and generate responses from strong models.
  • Data Selection: Naturalness-based approaches select data using model confidence scores to favor samples better adapted to the target model.Recent work also reports limitations of these methods on reasoning data.

6 Conclusion

The paper identifies step length confounding in naturalness-based selection, attributes it to diluted low-probability first-token effects, and introduces ASLEC methods that improve selection outcomes.

  • Conclusion: Naturalness-based selection systematically favors longer reasoning steps instead of higher-quality samples, a bias termed step length confounding.The phenomenon is presented as a limitation of naturalness-based selection for long-CoT datasets.
  • Conclusion: Low-probability first-token influence is diluted in longer sequences, inflating their average log probabilities.This mechanism explains why step length can distort naturalness-based selection scores.
  • Conclusion: ASLEC-DROP and ASLEC-CASL respectively drop or causally debias first-token probabilities when computing selection scores.Experiments across two datasets, four LLMs, and five benchmarks report consistent improvements over existing naturalness-based methods.

Limitations

The paper identifies unresolved confounding factors and leaves open whether its findings generalize to on-policy data selection. Additional analysis indicates total response length has a much weaker effect than step length.

  • The influence of first-token probabilities is identified as a key limitation, while deeper confounding factors remain an open question.
  • Whether on-policy data selection, where the student model generates its own samples, still correlates strongly with response length requires further exploration.
  • 9.8K versus 15.4K tokens: selected data had shorter average total responses than unselected data in preliminary experiments.The values refer to selected and unselected data, respectively.
  • Response length nevertheless showed rising average log probabilities, with longer responses receiving higher values across the examined models.The observed trend contrasts with the preliminary selection pattern based on total response length.
  • Tokens toward the end of responses had monotonically higher average log probabilities than tokens near the beginning.The paper attributes this pattern to increasing target-LLM confidence in continuation tokens.

A.3 Step Length Significantly Matters Response Length for Data Selection

The analysis finds that total response length contributes little to selection bias compared with step length. Removing its regression term barely changes performance, supporting its generally negligible influence in this setting.

  • The total-length confounder γ2|o_i| is approximately two orders of magnitude smaller than the step-length confounder γZ_i.
  • Removing γ2|o_i| from the selection criterion causes little change in model performance.The comparison uses the criterion s_casl_i − γZ_i − γ2|o_i|.
  • The analysis concludes that total response length has a generally small, potentially negligible influence on selection.
  • The experiments distinguish source LLMs that generate SFT data from target LLMs trained on that data.
  • The source models include QwQ-32B, Qwen3-32B, DeepSeek-R1-Distill-Qwen-32B, and gpt-oss-120b, while target models vary in size and type.

B.2 Data Sampling and Filtering

The study samples and filters reasoning data from LIMO-v2 and AceReason-1.1-SFT, then evaluates selection strategies through target-model fine-tuning and benchmark testing.

  • The experiments use LIMO-v2 and AceReason-1.1-SFT as the two data sources.
  • LIMO-v2 contains 800 filtered mathematics problems, with five diverse correct responses generated per problem by each source LLM.
  • AceReason-1.1-SFT contributes 10k sampled mathematics problems, each paired with one correct response from each of four source LLMs.
  • Data generation uses top-p sampling with p = 0.95, temperature 1.0 for gpt-oss-120b, and temperature 0.6 for the other source models.
  • Target models are fine-tuned with full-parameter training, batch size 32, maximum sequence length 32K, Adam optimization, and six epochs.
  • AceReason-1.1-SFT shows pronounced selected-versus-unselected step-length differences, while step-level log probability increases monotonically with step length.

C.2 Different Step Splitting Methods

The paper tests whether step-length confounding persists across alternative response-splitting methods. It does, although period-based splitting produces the most distinct distribution differences.

  • Different Step Splitting Methods: Responses are split using periods and spaces, double newlines, or external tools such as NLTK.
  • Different Step Splitting Methods: Across all tested splitting methods, selected and unselected data retain clear differences in step-length distributions.
  • Different Step Splitting Methods: Period-based sentence splitting produces the most distinct selected-versus-unselected distribution differences.
  • Different Step Splitting Methods: On AceReason-1.1-SFT, causal-regression parameters show high γ values, indicating that step-length confounding persists.
  • Different Step Splitting Methods: The result β1 ≪ β2 is consistent with the paper’s earlier conclusions.

D.1 Convergence Analysis

The convergence analysis finds that the proposed method outperforms the existing naturalness-based GRACE approach and baseline selection strategies. Additional results indicate that its selected data remains effective across difficulty settings and on Llama3-3B.

  • Convergence Analysis: GRACE consistently converges to a higher loss than the proposed debiasing method.The analysis also reports that the debiasing approach selects data with greater naturalness.
  • Baseline Comparisons: The comparison evaluates Uniform, High/Low Difficulty, and other selection strategies on the LIMO-v2 dataset.Uniform samples 4k trajectories across four source LLMs, while High/Low Difficulty selects the longest or shortest trajectories by response length.
  • Baseline Comparisons: The proposed selected data consistently achieves superior effectiveness compared with the evaluated selection strategies.Longer, more difficult examples improve supervised fine-tuning performance but contain more redundant or noisy reasoning steps than the proposed selection.
  • Generalizability: Training Llama3-3B on ASLEC-CASL-selected data consistently demonstrates the effectiveness of the proposed method.This experiment assesses whether the method generalizes beyond Qwen models.
Loading 2604.06834v1…