Capacity-Dependent Effects of Data Selection for Reasoning

Cuong Dang; Hoang Anh Just; Ruoxi Jia

Source-linked AI summary

Capacity-Dependent Effects of Data Selection for Reasoning

Cuong Dang, Hoang Anh Just, Ruoxi Jia

arXiv:2608.13721v1Aug 13, 2026cs.LGcs.AIcs.CL

TL;DR

Response selection for reasoning fine-tuning lacks a universal likelihood preference across students and training horizons. This paper studies the issue across model capacities using experiments, learning-dynamics analysis, and theory, finding a capacity-dependent Fast-Fit/Slow-Gain pattern.

Problem
Candidate responses differ in how well they match the student distribution, making response selection important for optimization stability, sample efficiency, and acquired reasoning behavior.
Method
The paper combines controlled experiments across 1.5B–8B students with learning-dynamics analysis and a capacity-constrained theoretical view of distillation.
Results
High-likelihood data yields faster, more stable early improvements, while low-likelihood data surpasses it for larger students given longer training.
Takeaways & Limitations
Effective reasoning data selection should account for model capacity and computing budget rather than universally favoring high-likelihood supervision.
Takeaways & Limitations
The theoretical analysis models low-capacity students with a feasible parameter space represented as a subspace of the full parameter space.

Abstract

from arXiv · show

In reasoning supervised fine-tuning, candidate responses for the same instruction can differ substantially in how well they match the student's current distribution. Recent likelihood-based response selection methods suggest that responses closer to the student distribution provide more effective supervision, motivating the hypothesis that high-likelihood responses may generally be preferable for fine-tuning. In this paper, we revisit this intuition and show that the value of likelihood-based data selection depends critically on model capacity and training duration. Through controlled experiments on mathematical reasoning, using students ranging from 1.5B to 8B parameters and supervision generated by stronger teacher models, we observe a clear \emph{capacity-dependent} ``{\color{SMALLCOLOR}\textbf{Fast-Fit}} / {\color{LARGECOLOR}\textbf{Slow-Gain}}'' pattern. High-likelihood data provides faster and more stable early improvements, especially for smaller models, but low-likelihood data becomes increasingly beneficial for larger models when training is allowed to continue longer. To explain this phenomenon, we analyze learning dynamics, showing that small models often fail to absorb low-likelihood supervision and instead fall into shallow or repetitive behaviors, while larger models are better able to move toward the teacher distribution under such data. We further provide a capacity-constrained theoretical view of distillation that clarifies how data difficulty, data span, and student capacity jointly govern transfer. Overall, our findings show that effective data selection for reasoning should be aware of model capacity and computing budget rather than based on a single universal preference for high-likelihood supervision.

1 Introduction

The introduction frames response selection in reasoning SFT as a capacity- and training-duration-dependent problem rather than a universal preference for high-likelihood data. It motivates this view through heterogeneous supervision, learning-dynamics analysis, and a capacity-constrained theory of distillation.

Motivation: Candidate responses for one instruction vary in alignment with the student distribution, affecting optimization stability, sample efficiency, and learned reasoning behavior.Some responses are close to the student’s current capabilities, whereas others are substantially harder to reproduce.
Central hypothesis: The paper hypothesizes that data-selection effectiveness depends on whether student capacity is sufficient to benefit from challenging supervision.The study examines this interaction through model capacity and learning dynamics.
Learning dynamics: High-likelihood supervision often produces rapid early gains, whereas low-likelihood supervision can yield slower but greater improvements for sufficiently capable models trained longer.The comparison motivates treating training duration as a key condition in response-selection decisions.
Main finding: The introduction reports a capacity-dependent effect in which high-likelihood data is not universally optimal for reasoning.Larger models can benefit substantially from low-likelihood data because it pushes them beyond their current distribution toward the teacher’s reasoning behavior.
Contributions: The paper explains the observed selection effects through optimization and learning dynamics and formalizes distillation using initialization, knowledge gap, data space, and student capacity.These factors jointly govern knowledge transfer in the proposed capacity-constrained view.

2 Related Works

Prior work establishes SFT, rationale training, and reasoning distillation as effective approaches for eliciting multi-step reasoning. This work situates likelihood-based data selection within context-dependent selection research while adding model capacity alongside training duration as determinants of whether high- or low-likelihood supervision is preferable.

SFT, Distillation & Data Engineering for Reasoning: SFT research shows that training on rationales can substantially improve multi-step reasoning, including through iterative bootstrapping on successful rationales.The passage cites Zelikman et al. (2022) as demonstrating iterative generation and fine-tuning on successful rationales.
Context-Dependent Data Selection: Curriculum and self-paced learning research finds that harder examples are not universally optimal, motivating adaptive or easy-to-hard schedules.The usefulness of harder examples is described as varying with the learner’s current stage rather than following a fixed policy.
Context-Dependent Data Selection: This work shows that the relative value of high- versus low-likelihood supervision depends jointly on model capacity and training duration.It introduces model capacity as a new theoretical axis, complementing prior analyses centered on data size and learning stage.

3 Background

Section 3 establishes teacher-generated supervision as a common training setup and formalizes candidate response collection across instructions and teachers. It then defines likelihood-based selection using the base student distribution, contrasting highest- and lowest-likelihood response datasets.

Teacher-generated supervision: Smaller language models are commonly trained on supervision generated by stronger teacher models.Examples include AceReason-Nemotron 1.1 trained on DeepSeek-R1 data and DASD-4B-Thinking supervised with gpt-oss-120b responses.
Candidate response construction: For each instruction, the framework collects one or more responses from every teacher in the teacher set.The candidate pool is defined over an instruction pool Q and teacher conditional distributions πθT(y | x).
Likelihood-based selection: Likelihood-based selection chooses, for each question, the response with the highest log-likelihood under the base student distribution.The base student model is denoted πθ0(y | xi), with θ0 representing its parameters.
Selection strategies: The study compares datasets formed from highest-likelihood responses, Dhigh, and lowest-likelihood responses, Dlow.Both selected datasets are used to examine likelihood-based response selection.

4 Main Empirical Results

High-likelihood data produces faster early gains, particularly for smaller students, but longer training reveals capacity-dependent benefits for low-likelihood data in larger models. The results show a “Fast-Fit / Slow-Gain” pattern in which model capacity and computing budget determine effective data selection.

Early-stage advantage: High-likelihood selection wins on almost all datasets during the first epoch, with the clearest early advantage for 1.5B and 3B students.The reported score is the best across four checkpoints saved during the first epoch.
Early-stage advantage: High-likelihood data consistently yields faster early gains, especially for smaller models.This is identified as Observation 1 in the empirical results.
Longer-horizon reversal: After five epochs, high-likelihood training remains dominant for the smaller 1.5B, 3B, and 4B students, while longer training produces a capacity-dependent reversal.The five-epoch result selects the best outcome among 20 checkpoints.
Longer-horizon reversal: For high-capacity models, high-likelihood data peaks early, whereas low-likelihood data improves later and can surpass it.This delayed improvement is identified as Observation 2.
Capacity-dependent learning: Under low-likelihood training, the 1.5B model repeats the question and an incorrect answer, the 3B model repeats reasoning, and the 7B model completes the solution correctly.The example links model capacity to progressively better learning from difficult supervision.
Fast-Fit / Slow-Gain: High-likelihood data is best for fast, compute-efficient early learning, whereas low-likelihood data becomes more valuable when capacity and training budget are sufficiently large.The combined pattern is summarized as “Fast-Fit / Slow-Gain.”

Dual-view Learning Dynamics Analysis

The learning dynamics are capacity-dependent across teacher and original-model views: small students struggle to move toward the teacher, while larger students show greater alignment, stability, and clearer selection effects.

Learning Dynamics under Teacher View: Under the teacher view, high-likelihood training increases teacher loss for the 1.5B student, while low-likelihood training leaves it roughly unchanged.This indicates the 1.5B model lacks sufficient capacity to move meaningfully closer to the teacher under either data selection strategy.
Learning Dynamics under Teacher View: Low-likelihood data moves larger models closer to the teacher but fails to do so for small models.The paper identifies this as capacity-dependent teacher alignment.
Learning Dynamics under Original Model View: Under the original-model view, low-likelihood training produces a broader, more divergent loss pattern for the 1.5B model while substantially overlapping its initial loss range.This suggests fine-tuning pushes many generated answers away from the base distribution without fully leaving its initial loss range.
Learning Dynamics under Original Model View: As model size increases, fine-tuned behavior becomes more stable, generalization improves, and high- versus low-likelihood training differences become more pronounced.The paper summarizes these effects as capacity-dependent distribution shift and generalization.

6 Theoretical Mechanism of Capacity-Constrained Distillation

The theory explains capacity-dependent distillation through two constraints: data expose only part of the teacher’s knowledge, and low-capacity students can use only the exposed knowledge within their representable subspace. This clarifies why low-likelihood supervision may help larger students but overwhelm smaller ones.

Motivation: The framework accounts for why identical data selection can produce different outcomes across student capacities.The section specifically uses the theory to explain why low-likelihood data may benefit larger models while failing to help smaller ones.
High-capacity student solution: Distillation transfers only the component of the teacher–student knowledge gap contained in the data span.Theorem 6.1 interprets the data as a mirror that exposes teacher knowledge through the data span.
Low-capacity student solution: For low-capacity students, useful supervision is restricted to the portion of the data span lying within the feasible student subspace.Theorem 6.2 formalizes this additional projection constraint, beyond what the data reveal about the teacher.
Capacity-aligned data: Successful low-capacity distillation requires exposed teacher knowledge to lie within the student’s representable subspace.Capacity-aligned data must both reveal teacher knowledge and remain representable by the student.
Connection to low-likelihood data: Low-likelihood data may expose knowledge beyond the student’s current distribution, but small models may be unable to represent or optimize it.When likelihood is too low, the supervision can exceed the student’s feasible capacity and become ineffective.

7 Conclusion · A Missing Proofs · A.1 Setting

The paper concludes that data-selection benefits are capacity-dependent: high-likelihood data supports faster early gains, whereas low-likelihood data increasingly helps larger models. Its formal setting models teacher–student distillation with linear classifiers, low-capacity students restricted to a subspace, and gradient-based optimization.

7 Conclusion: High-likelihood data produces faster and more stable early improvements, especially for smaller models, while low-likelihood data increasingly benefits larger models.The experiments span models from 1.5B to 8B parameters and establish a capacity-dependent “Fast-Fit / Slow-Gain” pattern.
A.1 Setting: The formal distillation setting uses binary classification with input space X, label space Y = {0, 1}, and input distribution D.The teacher–student notation is introduced for the theoretical analysis.
A.1 Setting: The teacher h∗: X →Y is a fixed linear classifier, and the student is likewise represented as a linear classifier.Both models are defined in the teacher–student notation used by the setting.
A.1 Setting: Low-capacity students are modeled as an s-dimensional subspace S ⊂Rd, with PS denoting orthogonal projection from the teacher parameter space onto S.The paper motivates this projection assumption using capacity-reduced student architectures in model compression and distillation.
A.1 Setting: Distillation constructs a transfer set {(xi, yi)}n from inputs xi sampled i.i.d. from D and associated soft labels.The soft labels are described as more informative than standard hard 0/1 labels.
A.1 Setting: The student is trained by minimizing a normalized cross-entropy loss whose normalization constant makes the minimum of L1 equal to 0.The optimization notation includes the normalized loss and its normalization convention.
A.1 Setting: For optimization analysis, the model is parameterized by matrices Wi(τ), whose values evolve under gradient descent through a differential equation, with w(τ) = WN(τ) · · · W1(τ).The formulation tracks each layer matrix over time τ ∈[0, ∞).

A.2 Properties of Cross-Entropy Loss

The appendix characterizes cross-entropy optimization through gradient-span constraints, global-minimum conditions, and curvature properties on restricted sublevel sets. Under full-rank data, the loss satisfies restricted strong convexity and a restricted Polyak–Łojasiewicz condition.

Gradient updates: The gradient update always lies in the span of the training inputs, ∇L1(w) ∈ span(X).The gradient is a linear combination of the training inputs.
Global minima: The global minimum of the cross-entropy loss is 0, attained exactly by weights satisfying the stated per-example zero-loss conditions.The proof reduces zero loss to matching the teacher logits on every training example.
Restricted curvature: For full-rank X, every sublevel set admits a positive restricted strong-convexity constant µ.The Hessian is lower-bounded on the relevant restricted directions using the data matrix spectrum.
Restricted curvature: For full-rank X, every sublevel set also admits a constant c > 0 satisfying a restricted Polyak–Łojasiewicz inequality.The condition is applied over directions whose differences lie in span(X).

A.3 Property of Low-Capacity Distilled Learning

Under a capacity constraint, distilled learning depends only on the training-data components that project into the student’s parameter space. For a directly parameterized linear student, optimization remains within the affine space formed by the initialization and the training-data span.

Projected-data equivalence: Theorem A.6 states that empirical risk depends on training inputs only through their projections onto the student space F.Components orthogonal to F are invisible to the student.
Projected-data equivalence: A low-capacity student can learn teacher behavior only through the training-data component that survives projection into its parameter space.This limits the effective data span available for distillation.
Space of distilled models: For a directly parameterized linear classifier, the model trajectory satisfies w(τ) ∈ w(0) + span(X).Thus, training stays in the affine subspace obtained by translating the data span by the initialized parameter w(0).

A.4 Optimal Distillation Solution of a One-Layer Linear Student Model · A.5 Optimal Distillation Solution of a Deep Linear Student Model · B Experiment Setup

The theory characterizes distillation as projection of the teacher–student gap onto the data span, with capacity constraints replacing the span by an effective projected span. Under stated assumptions, both high-capacity and low-capacity deep linear students converge toward the corresponding optimal solution.

A.4 Optimal Distillation Solution of a One-Layer Linear Student Model: The high-capacity one-layer student converges to ˆw = w0 + PX (w∗−w0) as τ →∞.The solution transfers the teacher–student gap only through its component in the training-data span.
A.4 Optimal Distillation Solution of a One-Layer Linear Student Model: Training can transfer only teacher knowledge exposed by the data, because the transferred knowledge gap is projected onto the data span.When the data span covers the knowledge gap, the corresponding information can be transferred.
A.4 Optimal Distillation Solution of a One-Layer Linear Student Model: Under capacity constraint F, the one-layer student converges to ˆwF = w0 + PXF (w∗−w0) as τ →∞.The capacity constraint induces an effective data span through projected data PF X.
A.5 Optimal Distillation Solution of a Deep Linear Student Model: For a high-capacity deep linear student, the predictor converges within ϵ of the one-layer optimum ˆw for all sufficiently large t, when n ≥d and the stated initialization conditions hold.The theorem assumes balanced initialization and L1 (w (0)) < L1 (0).
A.5 Optimal Distillation Solution of a Deep Linear Student Model: The high-capacity deep linear dynamics remain in an L (w (0))-sublevel set and converge in objective, with L (t) →0 as t →∞.The sublevel-set result follows from non-positive objective derivative, while the objective convergence is stated after establishing exponential decay.
A.5 Optimal Distillation Solution of a Deep Linear Student Model: The low-capacity deep linear student converges arbitrarily close to the capacity-constrained optimum associated with its effective projected data span.Under the same assumptions as the high-capacity theorem, for every ϵ > 0 there exists τϵ > 0 yielding the stated closeness.

B.1 Training … C.4 Generalization to Llama family

Across Qwen and Llama students, likelihood-based selection exhibits a capacity-dependent Fast-Fit/Slow-Gain pattern: high-likelihood data helps early, while low-likelihood data increasingly benefits larger models with longer training. Experiments use full-parameter SFT from pretrained checkpoints, with results supported across model families.

B.1 Training: Students are trained with full-parameter supervised fine-tuning from corresponding pretrained checkpoints on selected high- or low-likelihood subsets.Experiments run on 2 NVIDIA A100 GPUs using DeepSpeed ZeRO-3, FlashAttention-2, and gradient checkpointing for long-context training.
C.1 Small Qwen models show Fast-Fit without eventual crossover: For Qwen2.5-1.5B, high-likelihood training reaches 21.26 after 0.25 epoch and 24.16 at epoch 1, while low-likelihood training reaches 12.13 and 14.77.After additional training, low-likelihood performance peaks at only 16.50 by epoch 2.25, showing no eventual crossover.
C.2 Qwen2.5-7B is the actual transition regime: Qwen2.5-7B is transitional: high-likelihood data achieves 44.09 within the first epoch versus 37.05 for low-likelihood data, whose trajectory continues improving with optimization.This places the model between smaller students that struggle with low-likelihood supervision and larger students that benefit from it consistently.
C.3 Qwen3-8B gives the strongest evidence for Slow-Gain: Qwen3-8B shows Slow-Gain most clearly: high-likelihood training leads at epoch 0.25 with 66.16 versus 57.72 for low-likelihood training, which later overtakes it.Low-likelihood training overtakes high-likelihood training at epoch 2.25 and remains superior through epoch 5.
C.3 Qwen3-8B gives the strongest evidence for Slow-Gain: For Qwen3-8B, high-likelihood performance improves only 0.48 points from its early best of 66.16 to 66.64 overall, whereas low-likelihood performance rises 8.07 points from 62.62 to 70.69.The contrast captures fast early fitting for high-likelihood data and slower but larger eventual gains for low-likelihood data.
C.4 Generalization to Llama family: Llama results show the capacity-dependent effect beyond Qwen: Llama3.2-3B favors high-likelihood training on 9 of 10 benchmarks, with Minerva the sole low-likelihood advantage at 20.82 versus 20.77.The supplied results further state that low-likelihood training never surpasses high-likelihood training for this model.

C.5 Best-Checkpoint vs. Trajectory-Level Comparison of 7B and 8B via Peak Accessibility and Persistent Superiority Metrics · C.6 From Fast-Fit to Sustained Slow-Gain: A Capacity-Dependent Transition

The comparison distinguishes independently selected best-checkpoint peaks from trajectory-wide superiority, while extended results identify a capacity-dependent transition from rapid high-likelihood gains in small models to harder low-likelihood learning.

C.5 Best-Checkpoint vs. Trajectory-Level Comparison of 7B and 8B via Peak Accessibility and Persistent Superiority Metrics: Table 2 reports a best-checkpoint envelope, so its ten column scores need not come from one common checkpoint.This limitation is especially relevant when interpreting model trajectories.
C.5 Best-Checkpoint vs. Trajectory-Level Comparison of 7B and 8B via Peak Accessibility and Persistent Superiority Metrics: For Qwen2.5-7B, low-likelihood training reaches higher benchmark-wise peak performance on 6 out of 10 datasets.The passage characterizes this as a reversal relative to high-likelihood training.
C.5 Best-Checkpoint vs. Trajectory-Level Comparison of 7B and 8B via Peak Accessibility and Persistent Superiority Metrics: Persistent superiority requires low-likelihood training to remain better than high-likelihood training throughout the entire post-early-training region, not merely achieve a higher peak.The early stage is defined as the first epoch, t ≤ 1; later checkpoints comprise Tlate.
C.6 From Fast-Fit to Sustained Slow-Gain: A Capacity-Dependent Transition: Extended results indicate three distinct learning regimes as model capacity increases.The reported regimes are introduced as a capacity-dependent interpretation of Tables 4–9.
C.6 From Fast-Fit to Sustained Slow-Gain: A Capacity-Dependent Transition: In the low-capacity regime, Qwen2.5-1.5B, Qwen2.5-3B, and Llama3.2-3B show a high-likelihood Fast-Fit / No Catch-up pattern.These smaller models benefit rapidly from examples close to their current distribution, while low-likelihood supervision remains difficult to absorb.
C.6 From Fast-Fit to Sustained Slow-Gain: A Capacity-Dependent Transition: Tables 4–9 compare high- and low-likelihood selection across checkpoints or epochs for Qwen2.5-1.5B, Qwen2.5-3B, Qwen2.5-7B, Qwen3-8B, Llama3.2-3B, and Llama3.1-8B.The tables mark early-stage best values, all-training best values, and cells satisfying both where specified.