Source-linked AI summary
Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR
Chandak Chakma, Syed Nazmus Sakib, Nafiul Haque, Shifat E. Arman
TL;DR
Prior work reports that some difficult prompts with occasional successful solutions improve little, but the reliability of the labels and gradient comparisons behind that claim is uncertain. The paper models difficulty-label reproducibility and rechecks gradient similarity under matched correct-rollout counts. It finds persistent slow learning alongside unstable prompt selection and a reduced, but residual, gradient gap.
Problem
Prior work labels intermittently successful hard prompts unlearnable and links them to lower gradient similarity, while the reliability of prompt-set membership remains a measurement question.
Source: §1 · p. 1Method
The paper models how rollout budgets and aggregation affect difficulty-label reproducibility, then reassesses gradient similarity with correct-rollout counts matched.
Source: §1 · p. 2, §3.3 · p. 6, §3.4 · p. 7Results
Takeaways & Limitations
Difficulty labels require explicit evaluation budgets because aggregation can change prompt-set membership, which difficulty-aware methods use directly.
Source: §1 · p. 2The authors cannot test whether rewarded trajectories contain weak learning signal because training-time response strings were not stored.
Source: §3.5 · p. 9
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) has become an important approach for improving reasoning during post-training. Recent work suggests that some difficult prompts remain resistant to learning even when they occasionally produce correct solutions. We revisit this unlearnability phenomenon and find that the affected prompts do improve, at roughly one third of the learnable rate, while the difficulty-defined set used to study them is much less reproducible than expected. These difficulty labels are estimated from a limited number of sampled responses. Combining them across seeds can further change which prompts are selected instead of simply reducing measurement noise. We develop a sampling-based framework for quantifying this instability and determining how much evaluation is required for difficulty assignments to reproduce reliably. We also revisit the gradient-similarity evidence proposed to explain unlearnability and show that part of the observed separation arises because difficult prompts provide fewer correct rollouts from which their gradients can be estimated. Matching this sample count weakens the gradient difference but does not remove it. Overall, the slow-learning phenomenon survives our reanalysis, while both the prompts used to define it and the evidence used to explain it require more careful measurement.
1 Introduction
Because RLVR is widely used to improve reasoning, identifying which prompts learn slowly matters; prior work labeled some intermittently successful hard prompts “unlearnable.” This paper finds those prompts still improve, while both difficulty-set membership and gradient evidence depend on measurement choices.
1 Introduction
- RLVR is central to reasoning post-training, and prior work linked hard prompts that sometimes succeed but improve little to lower gradient similarity.If correct trajectories are available but learning remains limited, the constraint would be deeper than ordinary exploration or reward sparsity.Source: §1 · pp. 1–2
- At roughly one third of the learnable rate, labeled prompts still improve, while changing aggregation shifts the set from 74 to 228 prompts.Much of the disagreement across training seeds can be reproduced by resampling the same checkpoint.Source: §1 · p. 2
- 8 to 16 rollouts per prompt yield 0.63 to 0.73 agreement, while 0.90 agreement requires a projected 242 rollouts.The sampling-based predictions matched retrospective measurements and two prospectively frozen evaluations.Source: §1 · p. 2
- 2.327 times to 1.539 times: matching correct-rollout counts reduces the easy-to-unlearnable gradient-similarity ratio, but a substantial difference remains.Hard prompts provide fewer correct rollouts, so their prompt-level gradients are estimated from fewer samples.Source: §1 · p. 2
2 Related work
Related work frames this paper across RLVR’s learning limits, prompt difficulty as a training signal, explanations for hard prompts, and measurement reliability. The paper’s focus is whether prompt-level difficulty labels and gradient comparisons are reliable enough to support those claims.
RLVR and the limits of post-training
- RLVR’s limits remain debated: some work emphasizes sharpening base-model capabilities, while other work reports reasoning-boundary expansion and prompt-level slow learning.Chen et al. (2026) describe intermittently successful prompts that improve little and connect them to gradient-similarity differences.Source: §2 · p. 3
Difficulty as a training signal
- Difficulty estimates guide prompt replay, routing, reweighting, prioritization, and curriculum choices, but these methods infer difficulty from finite sampled outcomes.The methods differ in objective, yet share dependence on estimated prompt success rates.Source: §2 · p. 3
Why hard prompts fail to learn
- Prior work studies ways to make hard prompts more learnable and interprets lower gradient alignment as possible representation failure; this paper finds the comparison depends on correct-rollout counts.Matching sample counts reduces the reported gradient separation while leaving a residual difference that still requires explanation.Source: §2 · p. 3
Measurement and reproducibility
- This paper studies reproducibility of prompt-set membership under sampled-success-rate thresholds, rather than reproducibility of an aggregate benchmark score.The distinction matters because difficulty-aware methods act directly on prompt sets.Source: §2 · p. 3
3 Experimental Results
The published unlearnable prompts improve more slowly rather than remaining flat, while their membership is sensitive to evaluation and aggregation choices. Matching gradient sample counts weakens—but does not remove—the measured gradient separation, and the training comparison does not establish that set instability produces an equally large performance change.
3.1 Experimental setup
- Experiments use Qwen2.5-0.5B with GRPO on 1,023 MATH prompts, five seeds, and G = 8 rollouts; training and primary evaluation use a 1,024-token cap.The original scripts defaulted to 5,120 tokens, so the authors caution against transferring training-dependent magnitudes beyond their configuration.Source: §3.1 · p. 4
3.2 The slow-learning separation appears, but literal non-learning does not
- Du reward rises +0.0422 [+0.0259, +0.0590] per 100 optimizer steps versus +0.1119 for learnable prompts, about one third their rate.The slope, rather than terminal reward, is used because prompts can stop contributing once sampled rewards have zero variance, changing which prompts appear near the end.Source: Figure 1 · p. 4, §3.2 · p. 4
3.3 The published construction is not a consistent estimator of a fixed-threshold set
- With 292 candidates fixed, the published five-seed rule yields 74 Du prompts versus 228 under the alternative; single-seed sets contain 145–150 (mean 147.8).Disagreement is concentrated near τ = 0.1, and resampling the same checkpoint reproduces approximately 81% of the apparent cross-seed membership effect.Source: Figure 2 · p. 5, §3.3 · pp. 5–6
3.4 How much evaluation does a difficulty-defined set need?
- 0.6322 Jaccard agreement at 8 rollouts rises to 0.7265 at 16; reaching 0.90 requires a projected 242 rollouts per prompt in this setting.The budget depends on the distribution of prompt probabilities near the threshold; small-budget agreement can also show a sawtooth as the effective integer cutoff shifts.Source: Figure 3 · p. 7, §3.4 · p. 7
3.5 Does the gradient gap persist after matching sample count?
- At K = 11 correct rollouts per prompt, matching lowers the easy-to-Du similarity ratio from 2.327 to 1.539, with a 95% interval of [1.26, 1.88].Every tested matched level remains above one, but K = 11 retains only 28 of 71 Du prompts, so this is not a corrected full-cohort estimate.Source: Figure 4 · p. 8, §3.5 · p. 8
- Du prompts average 2.14 contributing steps per run versus 3.13 for learnable and 5.26 for easy prompts, but measured exposure starvation does not explain the slow learning.None of 156 prompts is completely starved across all five runs; content starvation remains unresolved because training-time response strings were not stored. Conditional on an update, the mean 1/σ scale is 2.8360 for Du versus 2.4262 for easy prompts.Source: §3.5 · p. 9
3.6 An unstable difficulty rule does not necessarily imply unstable training
- Two constructions removing approximately 5.0% versus 27.6% of usable training signal produce models differing by less than one held-out accuracy point.The experiment lacks sensitivity to resolve effects of that size, so this is not evidence of equivalence; the result bounds the claim to this setting.Source: §3.6 · p. 9
4 Conclusion
The published “unlearnable” prompts improve, but more slowly than learnable prompts; meanwhile, difficulty-set construction and gradient comparisons are sensitive to measurement choices. The conclusion is that the slow-learning effect persists, not that these prompts literally cannot be learned.
4 Conclusion
- Low-budget difficulty labels are unreliable for classifying individual prompts, and matching gradient-estimator sample counts shrinks—but does not eliminate—the reported similarity gap.The paper’s conclusion distinguishes uncertainty in the prompt set from the persistent slow-learning effect.Source: §4 · p. 9
Appendix — Contents and Index
- The appendices cover set construction, sampling-budget modeling, gradient sensitivity, optimization diagnostics, and experimental configuration.These topics are listed in the appendix contents.Source: Appendix — Contents and Index · p. 14
A.1 The slow-learning effect
- The Du cohort’s reward slope is +0.0422 per 100 optimizer steps, versus +0.1119 for learnable prompts, supporting slow learning rather than literal non-learning.The Du slope interval [+0.0259, +0.0590] excludes zero; at step 120, the learnable–Du difference is +0.0754 [+0.0279, +0.1246].Source: §A.1 · p. 15
A.2 How the published construction changes the selected set
The published multi-seed aggregation changes which prompts are selected rather than simply denoising a fixed-threshold set. Its instability is concentrated near the threshold, while the never-rewarded exclusion does not establish that a prompt is unsolvable.
A.2 How the published construction changes the selected set
- The five-seed construction selects 74 prompts from 292 pooled sub-threshold prompts, compared with 181 under a zero-success rule and 228 under an alternative aggregation.Individual seeds select 145–150 prompts, with a mean of 147.8.Source: §A.2 · p. 15
- The five-seed intersection has 0.976 precision but 0.541 recall against the N = 128 reference, favoring prompts well below the threshold over a neutral recovery of the fixed-threshold set.Disagreement is concentrated near τ = 0.1, and combining more seeds makes the selected set shrink rather than converge to a stable finite-τ set.Source: §A.2 · p. 15
- The training-run exclusion removes 218 prompts, yet a later N = 128 evaluation solves 116 of them at least once; no observed reward is not proof of impossibility.The two constructions overlap on 65 prompts.Source: §A.2 · p. 16
A.3 Evaluation noise versus training-seed variation
Repeated evaluation of the same checkpoint reproduces much of the apparent cross-seed membership disagreement. A binomial threshold model closely matches same-model agreement, and the section also identifies controls for other possible confounds.
A.3 Evaluation noise versus training-seed variation
- At N = 32, same-checkpoint Jaccard similarity is 0.798 versus 0.751 across seeds, and resampling reproduces about 81% of the apparent cross-seed effect.At N = 128, same-model sampling accounts for approximately 74% of the apparent effect; the two percentages are not directly comparable estimates of one constant.Source: §A.3 · p. 16
- The binomial model predicts same-model agreement within 0.004, supporting finite evaluation sampling as a major source of membership instability.Table 1 summarizes controls for evaluation noise, generation-length effects, and rollout-budget allocation.Source: §A.3 · p. 16, Table 1 · p. 16
B.1 From success probability to reproducibility
The sampling-budget analysis models prompt success probabilities and predicts agreement between repeated difficulty assignments. It estimates the evaluation budget needed for a reproducibility target, while accounting for an integer-threshold effect that makes agreement nonmonotonic at small budgets.
B.1 From success probability to reproducibility
- The method estimates a latent pass-rate distribution, integrates expected intersection and union rates for independent evaluations, and uses their ratio as predicted population-level Jaccard agreement.The distribution is estimated with the Kiefer–Wolfowitz maximum-likelihood formulation rather than treating noisy empirical pass rates as noise-free.Source: §B.1 · p. 17
- The integer success-count cutoff makes the effective threshold differ from τ and move downward between jumps as N increases, producing a nonmonotonic agreement curve at small budgets.The sawtooth pattern follows from the changing effective decision boundary, not Monte Carlo noise.Source: §B.1 · p. 17
- For Qwen2.5-0.5B/MATH at τ = 0.1, stable Jaccard agreement targets of 0.80, 0.90, and 0.95 require approximately 40, 242, and 2111 rollouts, respectively.These budgets depend on this threshold and pass-rate distribution; the calculation, not these numbers, is the transferable object.Source: §B.1 · p. 17
B.2 Equal-cost allocation of the rollout budget
At equal response cost, one deep evaluation produces a more reproducible threshold-defined set than four shallow, thresholded blocks. This comparison applies to a fixed checkpoint, and its effective thresholds match only for the selected budget pair.
B.2 Equal-cost allocation of the rollout budget
- The deep-versus-shallow comparison is interpretable because the chosen 128-response and four-by-32 rules have matching effective thresholds.That equality is specific to this budget pair; other comparisons can conflate aggregation with a changed decision threshold.Source: §B.2 · p. 18
- At equal cost, across two independent evaluations on one fixed checkpoint, the deep rule has a paired Jaccard reproducibility advantage of +0.0864 [0.0380, 0.1349] over the four-block intersection.The experiment isolates evaluation noise on a fixed checkpoint, not training variation across seeds.Source: §B.2 · p. 18
- +0.2138 [+0.0905, +0.3354] is the deep rule’s advantage after the never-rewarded exclusion on 759 eligible prompts.Source: §B.2 · p. 18
B.3 Validation of the sampling-budget model
The sampling-budget model captures repeat-set behavior and prospective measurements support its budget guidance, while threshold discreteness and set uncertainty constrain how its results should be interpreted. Its numerical recommendations are setting-specific, and small-budget membership uncertainty can materially affect downstream gradient estimates.
B.3 Validation of the sampling-budget model
- The model recovers the deep-over-shallow ordering but understates the four-block gap: predicted 0.0531 versus observed 0.0864.Its largest discrepancy is 0.8228 predicted against 0.7906 observed; predicted set sizes match within 1.8%.Source: §B.3 · p. 18
- A distributional model misses four-block Jaccards by 0.008–0.032 and deep-pass Jaccards by less than 0.003, unlike a homogeneous model’s 0.19–0.53 errors.Matching the marginal number of flagged prompts is insufficient to predict which prompts change membership.Source: §B.3 · p. 19, Table 2 · p. 19
- At N = 64, measured Jaccard is 0.8742 [0.8369, 0.9091], and both prospective measurements fall inside their frozen prediction intervals.At N = 40, the complete measured interval lies above the 0.80 target; Figure 6 distinguishes the two prospective measurements from other checks.Source: §B.3 · p. 20, Figure 6 · p. 21
B.4 Discrete thresholds and what transfers across settings
- Across N = 8 to 160, discrete threshold boundaries frequently make predicted reproducibility decline as the budget increases by one.The rule can change its effective cutoff at integer boundaries even as the estimate becomes sharper; beyond this range the sawtooth is much less prominent.Source: §B.4 · p. 20
- The 128-versus-four-by-32 threshold match is specific to that pair, and numerical budget values should not transfer directly across models or datasets.Required budgets also depend on threshold and on how much probability mass lies near it; the procedure, rather than the values, transfers.Source: §B.4 · p. 20
C.1 Propagation of set uncertainty to gradient similarity
- At N = 8 and N = 16, set-membership uncertainty adds full 95% interval widths of 0.4128 and 0.3090 to the gradient-similarity ratio.These widths are 83% and 62% of the downstream statistic’s reported sampling interval width; by N = 128 the added width falls to 0.1375.Source: §C.1 · p. 21
- At N = 32, the Du-anchored and label-faithful variants lie on opposite sides of the pre-specified 0.25 materiality threshold, leaving that budget unresolved.Source: §C.1 · p. 21
C.2 Matched-gradient analysis and cohort composition
Matching correct-rollout counts weakens but does not eliminate the easy-to-Du gradient-similarity gap; the matched cohort is selected toward easier Du prompts. The optimization diagnostics find fewer usable update steps for Du prompts, rather than a reduced normalization scale.
C.2 Matched-gradient analysis and cohort composition
- At K = 11, the easy-to-Du similarity ratio is 1.539 times [1.26, 1.88], below the unmatched 2.327 times ratio but above one.Across K ∈ {4, 6, 8, 11}, the matched ratio stays below the unmatched value and above one.Source: §C.2 · pp. 21–22
- Matching removes 48.9% [29.3, 71.2]% of the observed log-scale separation, but is not a causal decomposition because it changes cohort membership.Requiring K = 11 leaves 28 of 71 Du prompts eligible; retained prompts had more successful responses and are easier than many excluded prompts.Source: §C.2 · pp. 21–22
D.1 Exposure, zero-variance filtering, and GRPO normalization
- Du prompts receive 2.14 contributing steps per run versus 3.13 for learnable and 5.26 for easy prompts, but none of 156 prompts is starved across all five runs.Thirty-five prompts (22.4%) receive at least learnable-group mean exposure while improving slowly; the analysis does not rule out content starvation because training responses were not stored.Source: §D.1 · p. 22
- Conditional on non-zero reward variance, mean 1/σ is 2.8360 for Du prompts and 2.4262 for easy prompts, so reduced normalization scale is unsupported as the explanation.The main asymmetry is how often a usable update is formed.Source: §D.1 · p. 23
D.2 Does difficulty construction materially change training?
The two difficulty constructions remove substantially different amounts of usable training signal, yet their held-out accuracy differs by less than one point. The experiment cannot resolve effects of that size, so the result is not evidence of equivalence.
D.2 Does difficulty construction materially change training?
- Despite removing approximately 5.0% versus 27.6% of usable training signal, the resulting models differ by less than one held-out accuracy point.Five seeds per construction were compared with size-matched random-removal controls; registered sensitivity cannot resolve effects of this size.Source: §D.2 · p. 23
E.1 Experimental configuration and statistical procedures
The experiments replicate and extend the original RLVR setup, with audited configuration differences and a fixed 1,023-prompt dataset defining the scope of the reported quantities.
E.1 Experimental configuration and statistical procedures
- Experiments use Qwen2.5-0.5B on 1,023 MATH prompts, GRPO with G = 8 responses per prompt, five seeds, and the original verifier.Set-membership analyses use N = 128 evaluation rollouts per prompt; gradient analyses use N = 200 initial-policy rollouts.Source: §E.1 · p. 23
- The execution-path audit found 47 effective constants, with 30 matching the original configuration and 17 differing across four classes.Table 3 lists the differing constants but does not label each row by class.Source: §E.1 · p. 23, Table 3 · p. 24
- Training and primary evaluations use a 1,024-token response cap rather than the original scripts’ 5,120-token default, so training-dependent results are not exact numerical reproductions.The evaluation-side set-membership analysis is repeated at 5,120 tokens, but training remains at 1,024 tokens.Source: §E.1 · p. 23
- Using the 1,023-prompt subset fixes the prompt set across experiments, making reported set sizes and required sampling budgets specific to this setting.The subset replaces the larger dataset referenced by the original baseline script.Source: §E.1 · pp. 23–24
- Experiments ran on one NVIDIA GeForce RTX 3090 Ti with 24 GB VRAM, using Qwen2.5-0.5B, MATH, and the original implementation under their stated licenses.Most training and evaluation runs took approximately 16–24 hours.Source: §E.1 · pp. 23–24
- Confidence intervals are 95% bootstrap intervals, reward slopes are reported per 100 optimizer steps, and prospective budget predictions were timestamped before evaluation responses were generated.Prompts are resampled at the level appropriate to each statistic.Source: §E.1 · p. 24
E.2 Generation length and answer extraction
The longer response budget reduces truncation and answer-extraction failures, while leaving difficulty-set membership within the variation seen across repeated evaluations. The control applies to evaluation, not training-dependent results.
E.2 Generation length and answer extraction
- Increasing the response budget from 1,024 to 5,120 tokens reduces overall truncation from 0.0961 to 0.0320 and answer-extraction failure from 0.0977 to 0.0329.Almost all extraction failures at the shorter budget are truncations rather than verifier failures.Source: §E.2 · p. 24
- Jaccard similarity between N = 128 sets at 1,024 and 5,120 tokens is 0.8984 [0.8639, 0.9310], versus 0.8770 [0.8397, 0.9127] between evaluations at the same budget.The movement from increasing the token budget is no larger than repeatability variation under a fixed evaluation configuration.Source: §E.2 · p. 24
- The token-budget control addresses evaluation-side set membership only; it does not establish that training curves or gradient quantities would be numerically unchanged at 5,120 tokens.Training trajectories were generated with the shorter limit.Source: §E.2 · p. 24