Source-linked AI summary
Sharpen Before You Adapt: Data-Free Entry-State Sharpening for Test-Time Reinforcement Learning
Zhanming Zhang, Vinoth Selvendran
TL;DR
Diffuse checkpoint entry can make TTRL’s self-supervision less reliable and consume adaptation budget before reachable capability is expressed consistently. The paper prepares checkpoints with data-free training before TTRL and evaluates entry entropy across five Qwen3-4B checkpoints. Entry entropy rank-orders conversion efficiency, while the outcomes vary across preparation objectives and reachability constrains sharpening.
Problem
TTRL derives supervision from a checkpoint’s own samples, so a diffuse entry distribution can make self-consensus less reliable and consume limited adaptation steps before reachable capability is expressed consistently.
Source: §1 · p. 1Method
The paper uses data-free training before TTRL to prepare checkpoints and compares their entry states through entropy, conversion efficiency, and reachability.
Source: §1 · pp. 1–2, §3 · p. 3Results
Takeaways & Limitations
Entry entropy is a label-free signal for checkpoint preparation, but sharpening must be calibrated against the point where reachable capability begins to deteriorate.
Source: §7 · p. 7The evidence is observational across Qwen3-4B checkpoints under a fixed 15-step TTRL regime; a controlled preparation-strength sweep is needed to establish causal effects, and broader generalization remains open.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Test-time reinforcement learning (TTRL) adapts language models on unlabeled test problems using supervision derived from their own samples. This makes the checkpoint's \emph{entry state} consequential: a diffuse policy provides noisier self-supervision and may spend much of a limited adaptation budget merely concentrating probability mass before reliably expressing capability it already possesses. We propose \textbf{entry-state sharpening}: use data-free training \emph{before} TTRL to prepare a general-purpose checkpoint in a state that subsequent label-free adaptation can exploit more efficiently. The idea is not tied to one training recipe; different data-free objectives can move the same base model to different entry states. Across five data-free checkpoints derived from Qwen3-4B and evaluated under an identical 15-step TTRL protocol, entry policy entropy strongly rank-orders endpoint conversion efficiency, a reliability-to-reachability measure (Spearman $\rho=-0.90$; $\rho=-0.99$ after controlling for entry reachability). The contrast across objectives is striking: R-Zero remains diffuse at $3.39$ nats and finishes below the untuned base in 6/6 matched comparisons across MATH, GPQA, and AMC, whereas SPIRAL reaches $0.07$ nats and achieves the highest post-TTRL accuracy on MATH and GPQA despite its self-play stage using no math training data. An in-domain label-free self-distillation intervention further shows that the entry state can be deliberately sharpened. These results motivate treating checkpoint preparation as a \emph{state-control problem}: use data-free training to improve TTRL readiness, with entry entropy as a label-free control signal and reachable capability as the constraint.
1 Introduction
The checkpoint entering TTRL matters because its own distribution supplies supervision, and a diffuse policy can spend limited adaptation steps concentrating before expressing reachable capability reliably. The paper proposes data-free entry-state preparation and finds that entry entropy tracks conversion efficiency across checkpoints.
1 Introduction
- A diffuse entry policy makes self-consensus less reliable and can spend limited adaptation budget concentrating probability mass before reachable capability is consistently expressed.A checkpoint can therefore be strong at frozen inference yet poorly positioned for further self-improvement.Source: §1 · p. 1
- Entry-state sharpening uses data-free training before TTRL to prepare a general-purpose checkpoint for more efficient subsequent label-free adaptation.The proposal is broader than a particular recipe: self-play, self-evolution, and self-distillation can reshape the policy passed to TTRL.Source: §1 · p. 1
- The paper frames checkpoint preparation as state control and defines conversion efficiency η = mean@16/best@16, aiming to improve reliable expression without reducing reachable capability.The cited motivation is that reinforcement learning with verifiable or self-generated rewards often re-weights behaviors already accessible to the base model [35] [31] [8].Source: §1 · p. 2
- Spearman ρ = −0.90 between entry policy entropy and endpoint conversion efficiency across five checkpoints, strengthening to ρ = −0.99 after controlling for entry reachability.The ordering is stable across seeds (ρ = 1.00, p = 0.017), whereas raw-accuracy ordering is not.Source: §1 · p. 2
- R-Zero enters at 3.39 nats and finishes below the untuned base in 6/6 matched comparisons, while SPIRAL enters at 0.07 nats and yields the strongest post-TTRL checkpoint.SPIRAL never trains on math; the contrast shows that data-free objectives can produce very different entry states and TTRL outcomes.Source: §1 · p. 2
- Self-distillation moves Base entropy from 1.186 to 0.183 nats and conversion efficiency from 0.941 to 0.945, while high-strength reachability falls from 0.820 to 0.811.The intervention sharpens the entry state, but its reachability decline makes preserving reachable capability a constraint on preparation.Source: §1 · p. 2
2 Related Work
The paper distinguishes preparation before test-time learning from changes made within the adaptation loop. It situates entry-state sharpening among data-free self-improvement and entropy-control work, while emphasizing the short-horizon, label-free setting.
2 Related Work
- Prior test-time adaptation work uses self-supervised objectives, majority-vote rewards, or confidence- and uncertainty-based signals, including methods that regulate uncertainty within the adaptation loop [24] [40] [23] [19] [1] [5] [36] [39] [11] [30] [14] [37] [32] [12].This paper instead asks what state the checkpoint should have before adaptation begins, motivated by evidence that RL re-weights behaviors already accessible to the base policy [35] [31] [8].Source: §2 · p. 2
- Data-free self-improvement includes self-evolution, verifier-grounded self-play, self-instruction, and self-training, but these works primarily evaluate the resulting checkpoint as an endpoint [9] [21] [38] [3] [10] [34] [6].This study treats the checkpoint as an initialization for further test-time learning and measures the resulting state rather than only the upstream recipe.Source: §2 · pp. 2–3
- Preparation for downstream RL has precedents in domain-adaptive pretraining and mid-training, while SPIRAL inserts self-play before RLVR [7] [25] [29] [26] [4] [22] [15].Here, the downstream learner is label-free and operates on the test distribution.Source: §2 · p. 3
- Because self-training and distillation can reduce multi-sample diversity or coverage, sharpening is treated as controlled preparation rather than unconditional entropy minimization [13] [18] [17].The paper distinguishes entry concentration for short-horizon TTRL from entropy collapse in long-horizon RL, where exploration is often preserved [2] [33] [27] [16].Source: §2 · p. 3
3 Method: Entry-State Sharpening
Entry-state sharpening inserts data-free preparation before a fixed TTRL procedure. It compares different preparation objectives through checkpoint-level measures of entry entropy, conversion efficiency, and reachability.
3 Method: Entry-State Sharpening
- The framework inserts data-free preparation before TTRL, with the preparation objective m and its method-specific training strength τ_m defining the entry state.The entry policy is then passed to the test-time adaptation procedure on unlabeled target problems.Source: §3 · p. 3
- Preparation can use different operators, including self-play, self-evolution, and self-distillation, whose training strengths are not directly comparable.The methods are compared through a common checkpoint-level readout rather than their recipe-specific training budgets.Source: §3 · p. 3
- Conversion efficiency η is targeted for improvement while preserving reachability, separating how reliably capability is expressed from how much capability is reachable.Here, mean@16 is sampled accuracy and best@16 estimates reachability.Source: §3 · p. 3
- Entry policy entropy is measured before the first TTRL update using the checkpoint’s own rollouts from the first TTRL batch, without ground-truth labels.The rollouts use TTRL’s prompt template, tokenization, and rollout temperature.Source: §3 · p. 3
- The study contrasts game-based, environment-verified SPIRAL self-play with R-Zero’s uncertainty-driven curriculum and adds in-domain self-distillation on unlabeled target problems.The distillation stage retains traces consistent with the model’s self-consistency majority as pseudodemonstrations for supervised cross-entropy fine-tuning [10] [34] [6].Source: §3 · pp. 3–4
4 Experimental Setup
The experiments compare five checkpoints derived from Qwen3-4B under a shared, fixed 15-step TTRL protocol. Evaluation reports sampled accuracy, estimated reachability, and their conversion efficiency.
4 Experimental Setup
- All checkpoints use the same TTRL protocol: 15 optimizer steps, validation at steps 0, 5, 10, and 15, 32 reward votes, and 16 rollout/update and validation samples per question.The runs use matched seeds, identical training questions, and no ground-truth labels during adaptation.Source: §4 · p. 4
- The five-checkpoint roster comprises Base, R-Zero, SPIRAL@400, SPIRAL@32, and an in-domain self-distilled checkpoint, all produced without labeled target answers.SPIRAL@400 is the public multi-game release; SPIRAL@32 is an early Kuhn-Poker self-play endpoint.Source: §4 · p. 4
- For each validation question, the study reports mean@16, best@16, and conversion efficiency η; best@16 is a 1,000-resample bootstrap estimate of reachability.It is not the literal fraction of questions with one observed correct response; entry best@16 controls for capability in partial-correlation analyses.Source: §4 · p. 4
- TTRL optimizes majority-vote reward with GRPO [23] for 15 updates of 32 unlabeled problems each, using a 5 × 10^-7 actor learning rate with cosine decay.Rollouts use temperature 1.0; validation uses 16 samples per problem at temperature 0.6 and top-p = 0.95.Source: §4 · p. 4
5 Results
Across the five data-free checkpoints, entry entropy tracks how efficiently TTRL converts reachable capability into reliable output, while frozen accuracy alone does not identify readiness. Different preparation objectives produce distinct entry states, and sharpening must preserve reachable capability.
5.1 Frozen accuracy does not imply TTRL readiness
- R-Zero starts above Base on MATH but finishes below it, while SPIRAL@400 finishes highest; R-Zero also finishes below Base across all three domains.SPIRAL@400 reaches MATH final accuracy 0.787 and conversion efficiency 0.957 despite never training on math.Source: §5.1 · p. 4, Figure 1 · p. 5
5.2 Entry entropy rank-orders test-time conversion
- Entry entropy rank-orders endpoint conversion efficiency across five checkpoints (Spearman ρ = −0.900), strengthening to ρ = −0.986 after adjusting for entry best@16.The ordering is also stable across seeds, while raw-accuracy ranks are not.Source: §5.2 · pp. 4–5, Figure 2 · p. 6
- Conversion-efficiency checkpoint ranks are identical across seeds (ρ = 1.000), whereas final mean@16 ranks are not (ρ = 0.600).The exact p-values are 0.017 for η ranks and 0.350 for mean@16 ranks.Source: §5.2 · p. 5
- On GPQA, entry entropy rank-orders endpoint η at ρ = −1.000; on AMC, the relationship is directional at ρ = −0.800 and not significant at n = 5.The AMC ordering is qualified as directional rather than significant; the benchmark ordering is not identical across tasks.Source: §5.2 · p. 5, Table 2 · p. 6, Table 3 · p. 7
5.3 Data-free training effectively moves the entry state
- R-Zero and SPIRAL@400 produce opposite entry states, at 3.391 and 0.073 nats respectively, showing that data-free training alone does not determine TTRL readiness.R-Zero selects self-disagreement problems, whereas SPIRAL trains on environment-verified game outcomes.Source: §5.3 · p. 6
- Label-free self-distillation moves Base entropy from 1.186 to 0.183 nats and raises endpoint conversion efficiency from 0.941 to 0.945.At this preparation strength, final reachability falls from 0.820 to 0.811, so entropy is a calibration signal rather than a minimization target.Source: §5.3 · p. 6
- SPIRAL@32 enters at 0.344 nats with η = 0.937, compared with SPIRAL@400 at 0.073 nats and η = 0.957, offering directional evidence about stronger self-play.The paper characterizes this as directional evidence that stronger self-play concentrates policy and improves conversion.Source: §5.3 · p. 6
6 Limitations
The evidence is observational across checkpoints with different training histories, and experiments are limited to Qwen3-4B under a fixed 15-step TTRL regime. The study also uses mean token-level policy entropy as a practical proxy that does not fully capture answer- or solution-level diversity.
6 Limitations
- A controlled within-method preparation-strength sweep is still needed to establish whether calibrated entry-state sharpening causally improves final TTRL performance.The reported relationship is observational across checkpoints with different training histories.Source: §6 · p. 7
- Experiments cover the Qwen3-4B family and a fixed 15-step TTRL regime, leaving generalization across model families, horizons, and broader task distributions open.MATH provides the strongest matched-seed evidence; GPQA and AMC support the same directional pattern across additional domains.Source: §6 · p. 7
- Mean token-level policy entropy is a practical, label-free proxy for entry-state concentration, but does not fully capture diversity at the answer or solution level.Future work should test whether semantic or answer-level uncertainty more directly measures the reachable solution set.Source: §6 · p. 7
7 Conclusion
The paper frames data-free checkpoint preparation before TTRL as entry-state sharpening, motivated by TTRL’s conversion of reachable capability into reliable output. Across five checkpoints, entropy rank-orders conversion efficiency, while the results also show that sharpening must preserve reachable capability.
7 Conclusion
- Entry-state sharpening prepares a general-purpose checkpoint with data-free training before TTRL, shifting some concentration work upstream of limited adaptation.The premise is that TTRL largely converts reachable capability into reliable output.Source: §7 · p. 7
- Across five Qwen3-4B-derived checkpoints, entry policy entropy strongly rank-orders conversion efficiency, and appropriately prepared checkpoints can finish above Base after TTRL.SPIRAL achieves the highest final accuracy on MATH and GPQA, while the self-distilled checkpoint is marginally highest on AMC.Source: §7 · p. 7
- Entry entropy offers a label-free preparation signal, but the strength of sharpening must be balanced against deterioration in reachable capability.The self-distillation experiment deliberately shifts entry state while revealing this constraint.Source: §7 · p. 7