PaperlayerView exact revision on arXiv

Source-linked AI summary

How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum

Chu-Cheng Lin, Eugene Ie

arXiv:2604.25907v2cs.LGcs.AI

TL;DR

The paper develops a unified Tsallis-loss account of why SFT precedes RLVR and why RLVR alone stalls at cold start. It analyzes the continuum's escape–noise trade-off, introduces GARL and PAFT for rationale-free fixed-q training, and reports regime-dependent benchmark behavior.

  • Problem

    SFT-then-RLVR lacks a unified theoretical account explaining its ordering and why RLVR-only training stalls when initial success probability is near zero.

  • Method

    The paper defines the Tsallis loss continuum J_Q, whose parameter q controls per-instance amplification, and derives GARL and PAFT estimators for fixed-q training without annotated rationales.

  • Results

    High q escapes cold start faster but memorizes noise, whereas low q is noise-robust but slower; GARL mitigates cold-start stalling, while PAFT remains stable on HotPotQA and MuSiQue where GARL destabilizes.

  • Takeaways & Limitations

    The results support a regime-dependent training strategy: use high-q GARL at cold start, then select low-q GARL or sufficiently high-q PAFT according to warm-start stability, with q annealed toward zero near convergence.

  • Takeaways & Limitations

    Experiments use Qwen 3 0.6B, three benchmarks, and fixed q; warm-start GARL collapse and PAFT stability are verified only at 0.6B, and annealed-q schedules remain unvalidated.

Abstract

from arXiv · show

SFT-then-RLVR is widely used for post-training reasoning models, but why this specific ordering, and why RLVR-only stalls at cold start, have lacked a unifying theoretical account. We provide that account under a unified loss family $J_Q$ using the Tsallis $q$-logarithm. $J_Q$ is a single-parameter family that interpolates between RLVR (at $q{=}0$, the \textit{exploitation pole}) and the log-marginal-likelihood over latent trajectories (at $q{=}1$, the \textit{density-estimation pole}), under which the standard pipeline corresponds to a stepwise $q{=}1 \to 0$ schedule. All members share the same per-example gradient direction, differing only by a per-instance amplification $P_θ^{-q}$ that reweights each instance independently of the learning rate. Under gradient flow analysis, we show that the exploitation pole requires $Ω(\frac{1}{p_0})$ time to escape cold start but is robust to label noise, while the density-estimation pole escapes in $Θ\big(\log(\frac{1}{p_0})\big)$ but memorizes label noise. This separation explains how SFT ($q{=}1$) first moves the model out of the cold-start regime, followed by the more robust RLVR ($q{=}0$), under the SFT-then-RLVR paradigm. We further derive two Monte Carlo estimators that directly optimize fixed-$q$ on the $J_Q$ continuum, without annotated rationales: Gradient-Amplified RL (GARL) and Posterior-Attenuated Fine-Tuning (PAFT), with shared bias $O\big(\frac{q}{M P_θ^q}\big)$ but different variance and stability properties. On FinQA, HotPotQA, and MuSiQue, GARL at sufficiently high $q$ substantially mitigates cold-start stalling, escaping cold start where GRPO fails entirely. In warm start, GARL at low $q$ dominates FinQA where training is stable; on HotPotQA and MuSiQue, GARL destabilizes and PAFT at $q{=}0.75$ remains stable, reaching $47.9$ \texttt{m@16} on HotPotQA ($+13.9$ over GRPO).

1 Introduction

The paper explains SFT-then-RLVR through a Tsallis-loss continuum whose commitment parameter controls per-instance gradient amplification, balancing cold-start escape against noise robustness. It introduces GARL and PAFT, with experiments showing high-q GARL escapes cold start and stable warm-start gains from GARL or PAFT depending on benchmark.

  • 1 Introduction: The JQ family interpolates between exploitation at q=0 and density estimation at q=1, with shared gradient directions reweighted per instance by P_θ^-q.Higher q amplifies gradients on unfamiliar, low-probability examples; global learning-rate changes cannot reproduce this per-example reweighting.
  • 1 Introduction: High q enables fast cold-start escape but memorizes label errors, whereas low q is noise-robust but escapes cold start slowly.The poles require Θ(log(1/p0)) and Ω(1/p0) escape times, respectively.
  • 1 Introduction: The standard SFT-then-RLVR recipe corresponds to switching from q=1 to q=0, using amplification first and robust filtering afterward.Intermediate q can instead cold-start models directly under JQ without annotated rationales.
  • 1 Introduction: GARL and PAFT are Monte Carlo estimators for fixed-q training without annotated rationales; GARL has lower variance, while PAFT is more stable in some warm-start settings.Both estimators share bias O(q/(M P_θ^q)).
  • 1 Introduction: GARL at sufficiently high q escapes cold start where GRPO fails entirely, while warm-start gains range from +7.0 to +13.9 maj@16 over GRPO.PAFT reaches 47.9 m@16 on HotPotQA, compared with GRPO’s 34.0.

2 Setup and the JQ Loss Family

The paper defines JQ for latent reasoning trajectories using the Tsallis q-logarithm, connecting reward maximization and density estimation. Its shared gradient geometry shows that q changes only per-example amplification, while the endpoints differ in boundedness, noise behavior, and distributional coverage.

  • 2 Setup and the JQ Loss Family: For latent reasoning trajectories, P_θ is the model’s success probability for the target output, with J0=1−E_D[P_θ] and J1=−E_D[log P_θ].Under exact-match supervision, minimizing J0 maximizes expected reward.
  • 2 Setup and the JQ Loss Family: The Tsallis q-logarithm defines JQ, recovering exploitation loss J0 at q=0 and density-estimation loss J1 at q=1.For q<1 the per-example loss is bounded and noise-robust; q=1 is unbounded and fits label errors.
  • 2 Setup and the JQ Loss Family: Increasing q imposes stronger penalties on non-uniform success and yields an escort-distribution minimizer that interpolates from coverage at q=1 toward mode-seeking as q approaches 0.The q=1 endpoint is the unique strictly proper scoring rule in JQ.
  • 2 Setup and the JQ Loss Family: All JQ members share one per-example gradient direction and differ through P_θ^-q amplification, which increasingly emphasizes unfamiliar examples as q rises.The two factorizations motivate GARL and PAFT.

3 Commitment Dynamics under Gradient Flow

Gradient-flow analysis attributes cold-start behavior to the amplification exponent P_θ^-q: q=0 is provably slow, while q=1 escapes exponentially faster in the initial success probability. The same amplification accelerates commitment to both clean and noisy labels, motivating the SFT-then-RL switch.

  • 3 Commitment Dynamics under Gradient Flow: Under gradient flow, escape from cold start takes Ω(1/p0) at q=0 but Θ(log(1/p0)) at q=1.The separation follows from the P_θ^-q amplification in the success-probability dynamics.
  • 3 Commitment Dynamics under Gradient Flow: The analysis tracks single-example success probability under continuous-time gradient flow, isolating amplification while assuming bounded score and, for tight rates, a positive lower score bound.It does not fully model multi-example language-model optimization.
  • 3 Commitment Dynamics under Gradient Flow: The exponent 2−q controls the cold-start divergence because the score norm is q-independent in the derived dynamics.This yields slower divergence at low q and faster escape at high q.
  • 3 Commitment Dynamics under Gradient Flow: The same mechanism accelerates noise fitting: q=1 memorizes contamination logarithmically, while noise-fitting time diverges as q approaches 0.Thus amplification speeds both clean and corrupted commitment rather than selectively improving correctness.
  • 3 Commitment Dynamics under Gradient Flow: SFT-then-RL acts as a q=1→0 switch: SFT escapes cold start through amplification, then RL suppresses later noise commitment.The continuum replaces this hard switch with smooth interpolation.

4 Gradient Estimators for JQ

The paper derives two Monte Carlo estimators for fixed-q optimization of J_Q: GARL uses prior samples and amplification, while PAFT uses posterior-resampled trajectories and attenuation. Both share finite-sample bias that worsens with q and low P_θ, but PAFT can offer more stable warm-start training despite higher variance.

  • GARL: GARL samples latent trajectories from the prior and amplifies the RL gradient using an estimate of P_θ^-q.At q=0 it recovers RB-REINFORCE, while q=1 gives the IWAE gradient estimator.
  • PAFT: PAFT approximately samples from the posterior, applies teacher forcing to posterior-weighted rationale-answer triples, and attenuates the gradient by an estimate of P_θ^(1−q).At q=1, attenuation vanishes and PAFT recovers the EM gradient update.
  • Implementation: Both estimators are drop-in replacements at the same rollout budget and require no extra forward passes.GARL adds scalar reweighting, whereas PAFT adds categorical resampling followed by teacher forcing on generated tokens.
  • Bias and consistency: Finite-sample bias arises because the estimators reuse samples in a numerator-denominator ratio, despite unbiased numerator and denominator components.At q=0, the bias vanishes exactly because the estimator becomes an unbiased sample mean.
  • Bias and consistency: Under bounded-score conditions, the shared bias scales as O(q/(M P_θ^q)), while the bound is not uniform over training and may shrink as P_θ approaches zero.The expansion is asymptotic at fixed P_θ and large M, so in cold start it indicates degradation direction rather than a uniform bound.
  • Variance and stability: GARL has lower variance, but PAFT can be more stable at warm start because posterior resampling excludes bad rationales before gradient formation.PAFT's higher variance preserves the semantic coherence associated with its fine-tuning endpoint.

5 Empirical Validation

Experiments test fixed-q JQ optimization across cold- and warm-start reasoning settings. High-q GARL escapes cold start where GRPO fails, while warm-start results favor stable method choices that vary by benchmark.

  • RQ1: Cold-start escape: GARL at q≥0.75 escapes cold start on Qwen 3 0.6B where GRPO and all q≤0.5 settings fail entirely.At 8B, the escape threshold shifts to q≥0.85.
  • RQ1: Cold-start escape: GARL at q=0.75 matches or exceeds prompted warm-start GRPO on every reported metric across FinQA, HotPotQA, and MuSiQue.The comparison is confounded because the methods use different prompting regimes.
  • RQ2: Warm-start utility: GARL is stable on FinQA, where q=0.25 reaches 38.7 m@16, +10.9 over GRPO.Performance is roughly flat across q∈[0,0.75], and higher q adds estimator bias and noise memorization.
  • Experimental scope: PAFT is not evaluated at cold start because posterior attenuation suppresses gradients and importance resampling suffers effective sample size near one.This bounds the empirical scope of PAFT to warm-start experiments.
  • RQ3: PAFT stability: 47.9 m@16: PAFT at q=0.75 beats GRPO by +13.9 on HotPotQA while remaining stable; PAFT also reaches 22.4 on MuSiQue, +7.0 over GRPO.GARL collapses on HotPotQA and its MuSiQue peak does not survive training.
  • RQ3: PAFT stability: PAFT at q=0.25 underperforms GRPO on MuSiQue, 9.0 versus 15.4, but its validation curve remains rising at training end.Higher q recovers speed while preserving the stability observed for PAFT.

6 Discussion and Future Work

The discussion presents JQ as a unified, regime-dependent view of SFT-then-RLVR and proposes a three-phase training recipe. It also identifies unvalidated schedules and limits the strongest stability evidence to 0.6B models.

  • Discussion: JQ recovers SFT-then-RLVR as a q=1→q=0 schedule and permits rationale-free training with intermediate q.GARL targets cold-start amplification, while PAFT targets warm-start stability.
  • Future recipe: The proposed recipe uses large-q GARL at cold start, stable low-q GARL on FinQA, and PAFT at q≥0.75 on HotPotQA and MuSiQue.As Pθ→1, annealing q→0 is proposed to recover the unbiased RB-RLOO estimator.
  • Future work: The annealed-q switches remain theory rather than validated empirical procedures.The paper explicitly identifies validating these switches as future work.
  • Limitations: Experiments use Qwen 3 0.6B, three benchmarks, and fixed q; warm-start GARL collapse and PAFT stability are verified only at 0.6B.The cold-start ordering also replicates at Qwen 3 8B, but 8B warm-start verification is ongoing.
  • Limitations: The convergence analysis is stylized, assuming single-example gradient flow, bounded score, and exact-match supervision; general rewards remain open.These assumptions constrain how directly the theoretical results transfer beyond the studied setting.
  • Related work: Related work places JQ among RL–MLE continua, weighted likelihood methods, noisy-label losses, and latent-variable estimators.The paper distinguishes its example-level marginal Pθ weighting from token-level gates and policy regularization.

B Proofs for Section 2: Setup and Background

The appendix establishes the connection between exact-match expected reward and the exploitation loss. It does so by identifying the success probability of the correct output for each supervised example.

  • Proof: Under exact-match reward, the expected reward equals the dataset expectation of the correct-output probability Pθ.The indicator reward selects the correct output before averaging over training examples.
  • Endpoint connection: The exploitation loss J0=1−E_D[Pθ] is therefore equivalent to maximizing expected exact-match reward.Both objectives are minimized or optimized through the same correct-output probability.

C Proofs for Section 2: Loss Landscape

For q>0, the Tsallis objective has a unique categorical minimizer and penalizes uneven success across examples. Its endpoints interpolate between recovering the data distribution at q=1 and concentrating on the most frequent output as q approaches 0.

  • Dispersion penalty: J_Q(θ,q) ≥ −log_q(P̄), with equality exactly when success probability is constant across examples.This establishes a dispersion penalty favoring uniform per-example success.
  • Minimizers: For q>0, strict convexity yields a unique interior minimizer of J_Q in the categorical model.The proof uses strict convexity and Lagrange multipliers; boundary points are suboptimal for q<1 and incur infinite loss at q=1.
  • Endpoint behavior: At q=1, the minimizer recovers the data distribution exactly, whereas q→0 concentrates all mass on the most frequent output.The exploitation endpoint is mode-seeking rather than distribution-matching.
  • Monotone sharpening: For 0<q<1, increasing q sharpens the minimizer toward outputs with larger target probabilities.The relative concentration grows with the inverse-q exponent.
  • Propriety: The Tsallis q-logarithmic scoring rule is strictly proper if and only if q=1.For q∈(0,1), the true distribution is not a maximizer.

D.1 Warm-up: exact analysis on the sigmoid model

The sigmoid warm-up reduces gradient-flow escape to an exactly solvable scalar problem. It confirms logarithmic escape at q=1 and inverse-initial-probability escape at q=0, while the general theorem provides weaker bounded-score guarantees.

  • Sigmoid dynamics: Under sigmoid gradient flow, the separable ODE gives exact cold-start escape times for the Tsallis loss.The dynamics follow from P(θ)=σ(θ) and the score norm ∥s(θ)∥²=(1−p)².
  • Exploitation pole: T_0=(1/p_0)(1+o(1)) for q=0.The remainder is O(log(1/p_0)), negligible relative to 1/p_0.
  • Density-estimation pole: T_1=log(1/p_0)(1+o(1)) for q=1.The dominant integral is logarithmic in the inverse initial success probability.
  • Relation to general bounds: The general theorem assumes only bounded score norms, whereas the sigmoid model yields exact 1+o(1) asymptotics.The sigmoid score norm converges to a known constant as p approaches zero.
  • General guarantee: With bounded score norms, q=1 has a logarithmic lower bound Ω(log(1/p_0)).This is the general-model counterpart to the sigmoid calculation.

D.3 Proof of Theorem 3.2: Tight cold-start escape rates

Tight gradient-flow rates show that higher q accelerates cold-start escape but applies the same amplification to incorrect supervision. Near optimality, convergence times become asymptotically q-independent under a score-norm condition.

  • Cold-start escape: Under upper and lower bounded score norms, T_q=Θ(p_0^{−(1−q)}) for q<1 and T_1=Θ(log(1/p_0)).The rates are tight as p_0 approaches zero.
  • Cold-start escape: For q<q′≤1, the cold-start speedup ratio T_q/T_q′ diverges as p_0 approaches zero.The difference in exponents makes larger q increasingly advantageous in the cold-start regime.
  • Near-optimality convergence: Near optimality, convergence time is the same for all q up to a correction vanishing with the initial error.This holds when the score norm depends on θ only through P_θ.
  • Label noise: The same P_θ^−q amplification accelerates commitment to clean and corrupted supervision by the same multiplicative factor.Noise-fitting therefore has a dynamical dual to cold-start escape.
  • Label noise: At q=1, the stable noise equilibrium fits the observed noise rate, while q→0 drives the equilibrium contamination toward zero.The equilibrium is both the static escort minimizer and the dynamic gradient-flow limit.
  • Label noise: For q=0, any positive initial contamination decays monotonically toward zero, so reaching a larger contamination target takes infinite time.This is the robust filtering behavior of the exploitation pole.

E Proofs and Pseudocode for Section 4: Monte Carlo Estimators

The estimator proofs establish consistency, a shared finite-sample bias scaling, and different variance properties for GARL and PAFT. Endpoint substitutions recover familiar RLVR, importance-weighted, SFT, and EM-style updates.

  • Consistency and bias: Under P_θ>0, finite gradient moments, and weights bounded away from zero, the Monte Carlo estimators are consistent.The proof uses the strong law and continuity of the plug-in map.
  • Consistency and bias: With bounded marginal and per-trajectory scores, the leading bias is O(q/(M P_θ^q)).The same P_θ^−q exponent appears in the cold-start amplification factor.
  • RLOO control variate: RLOO has the same expectation as the plug-in estimator for every M and preserves its bias expansion.The control variate changes the estimator form without changing its expected value.
  • Endpoint recovery: At q=0, GARL recovers Rao–Blackwellized REINFORCE, while q=1 recovers the IWAE gradient estimator.GARL analytically marginalizes the output label conditional on each latent trajectory.
  • Endpoint recovery: At q=0, PAFT becomes posterior-resampled SFT scaled by P_θ, whereas q=1 recovers an EM-style posterior update.The q=0 endpoint can downweight hard instances overly aggressively; q=1 removes that attenuation.
  • Variance and stability: PAFT and GARL have identical expectations and bias, but PAFT variance is at least GARL variance.The variance inequality is strict except when conditional PAFT variance vanishes almost surely.

F Additional Experimental Details

Experiments use filtered subsets of three reasoning benchmarks, standardized optimization and checkpoint selection, and additional analyses of GARL’s estimator behavior, limitations, and methodological scope.

  • Data construction: Training subsets retain examples below task-specific character-length caps, yielding separate train, validation, and test splits for FinQA, HotPotQA, and MuSiQue.The supplied passage specifies the filtering procedure and task-specific caps, but truncates the resulting subset sizes.
  • Evaluation: Checkpoints are evaluated every 50 steps and at training end, with the best m@16 validation checkpoint selected for reporting.This selection rule applies across the reported experiments.
  • Generation: Maximum generation lengths are 4096, 3072, and 2048 tokens for FinQA, HotPotQA, and MuSiQue, respectively, plus 128 answer tokens.The caps differ by benchmark while reserving a fixed answer-token allocation.
  • Additional analyses: GARL analyses examine cold-start escape, estimator effects, warm-start validation curves, and dataset-level dynamics governed by q-dependent weighting and the kernel Kij.The multi-example analysis is identified as a limitation because its convergence treatment is single-example, while annealing and richer posterior sampling remain open.
  • Scope: The work is methodological, using public Qwen 3 checkpoints and FinQA, HotPotQA, and MuSiQue without releasing new pretrained models or datasets.The proposed continuum and estimators target post-training without annotated rationales.
Loading 2604.25907v2…