PaperlayerView exact revision on arXiv

Source-linked AI summary

The Unlearnability Phenomenon in RLVR for Language Models

Yulin Chen, He He, Chen Zhao

arXiv:2605.16787v1cs.LGcs.CL

TL;DR

RLVR can provide correct rollouts for hard examples that nevertheless remain unlearnable, but the reasons are unclear. This paper analyzes their gradients and reasoning traces, finding that unlearnability reflects flawed representations that RL-stage augmentation and curriculum learning do not resolve.

  • Problem

    It remains unclear whether positive rewards suffice to teach initially difficult examples, including why some remain unlearnable despite correct rollouts.

  • Method

    The paper analyzes cross-example rollout gradients and reasoning traces, while testing optimization, sampling, augmentation, and curriculum-based explanations for unlearnability.

  • Results

    Across settings, about half of examples with positive rewards are unlearnable; these examples are gradient outliers with flawed reasoning representations that RL-stage interventions do not improve.

  • Takeaways & Limitations

    Unlearnability exposes a fundamental limitation of RL post-training, while mid-training can improve gradient similarity on hard examples.

  • Takeaways & Limitations

    Experiments are limited to small-to-mid-scale models in mathematical reasoning, and unlearnability is defined using a thresholded convergence criterion.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Reward (RLVR) has proven effective in improving Large Language Model's (LLM) reasoning ability. However, the learning dynamics of RLVR remain underexplored. In this paper, we reveal a counterintuitive phenomenon: among hard examples that the model initially struggles with, a substantial subset remains unlearnable even when correct rollouts are present. To understand the phenomenon, we first demonstrate that existing optimization and sampling techniques fail to resolve unlearnability. With cross-example gradient analysis, we show that unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples and ungeneralizable reasoning patterns. We further show that representation flaws are difficult to mitigate in RL, as data augmentation does not improve gradient similarity. Our study provides the first systematic characterization of unlearnable data in RLVR training and reveals fundamental limitations in current RL approaches for reasoning tasks. Code and data are available at \url{https://github.com/yulinchen99/unlearnability-rlvr}.

1. Introduction

RLVR improves language-model reasoning, but a substantial subset of initially difficult examples remains unlearnable despite positive rollouts. The paper attributes this phenomenon to flawed representations and identifies it as a fundamental limitation of current RLVR training.

  • Background: RLVR, commonly using GRPO, improves language-model reasoning across math, coding, and agentic tasks.Its success relies on outcome-reward variance among grouped rollouts, with both correct and incorrect rollouts for each example.
  • Unlearnability phenomenon: A substantial subset of initially difficult examples remains unlearnable throughout GRPO training, despite the presence of positive reward.Examples are categorized as easy, learnable, or unlearnable according to their learning behavior during training.
  • Optimization-side explanations: Targeted interventions addressing positive-rollout scarcity, clipping, KL penalties, and rollout-gradient interference yield no improvement in unlearnability.The study controls positive-rollout counts and ablates standard regularization mechanisms to test optimization-side explanations.
  • Representation analysis: Unlearnable examples exhibit substantially lower positive-rollout gradient similarity to the rest of the training data than easy and learnable examples.Qualitative reasoning-trace inspection also indicates ungeneralizable reasoning patterns among unlearnable examples.
  • Implication: The study concludes that language models have flawed representations for unlearnable data that are difficult to fix during RL post-training.It characterizes unlearnability as a fundamental limitation of current RLVR training.

2. Related Works

Related work examines whether RLVR enables LLMs to learn new skills, how models generalize after RL, and how training techniques improve GRPO. Prior studies also analyze difficult examples and the roles of exploration, efficiency, and credit assignment.

  • Can LLMs Learn New Skills from RLVR: Studies ask whether LLMs learn new skills through RLVR and report that pass@k can degrade after RL.Follow-up work uses more controlled experiments to investigate this question.
  • Can LLMs Learn New Skills from RLVR: Some studies find that LLMs learn atomic skills through SFT and compose them during RL training.Other work examines generalization after RL and its relationship to the initial policy model.
  • Can LLMs Learn New Skills from RLVR: Wu et al. (2026) discusses why LLMs cannot discover entirely original solutions, while Sun et al. (2025a) studies extremely difficult examples with zero initial pass@k.These works connect RLVR learning questions to solution novelty and difficult-example dynamics.
  • Training Techniques for LLM RLVR: Post-GRPO training techniques primarily target training efficiency, exploration, and credit assignment.DAPO encourages exploration by clipping higher and removing the KL penalty; other methods use entropy to adjust loss weights.

3. Unlearnable Examples in LLM RLVR

This section defines unlearnable examples in GRPO-based RLVR and shows that they comprise roughly half of difficult examples across model and training-data settings, despite correct rollouts during training.

  • Baseline GRPO: GRPO samples k responses per training example, verifies them with binary rewards, and filters prompts with zero reward variance through dynamic sampling.The policy is optimized using a PPO-style objective with GRPO dynamic sampling.
  • Example categorization: Examples with initial success rate ≥10% are easy, while harder examples are learnable if performance improves consistently and unlearnable if reward remains low.Examples that never produce correct rollouts are excluded from the unlearnable group.
  • Working definition: An example is unlearnable when final-policy pass@1 remains below τ = 0.1 after validation performance saturates, despite observing correct training samples.Pass@1 is estimated from N = 32 sampled responses per example.
  • Results: Across all settings, about half of difficult examples are learned smoothly and the other half remain unlearnable after excluding examples with no positive reward.The phenomenon appears across model and training-data configurations.

4. Examining Common Explanations for Unlearnability

The section tests whether unlearnability arises from insufficient positive rollouts or from clipping and KL regularization. Experiments reject both explanations: increasing positive-rollout exposure and relaxing these constraints leave the learnability gap intact.

  • Overview: The analysis examines two hypotheses: too few positive rollouts and gradient suppression from clipping or KL regularization.Experiments primarily use Qwen2.5-0.5B on MATH Easy; corresponding Llama-3.2-3B-Instruct results appear in Appendix A.1.
  • 4.1 Positive-rollout scarcity: Oversampling and replay enforce exactly one positive and seven negative rollouts per prompt during gradient calculation and policy optimization.The method samples 4k rollouts per example, then downsamples to k = 8 with kpos = 1.
  • 4.1 Positive-rollout scarcity: Controlling positive-rollout counts slows learnable-data training but does not resolve unlearnability, leaving the learnable–unlearnable reward gap.The gap also persists under supervised fine-tuning on distilled correct responses and RL with k = 64 on unlearnable examples alone.
  • 4.2 Clipping and KL regularization: Correct rollouts for unlearnable examples have reference log-likelihood distributions comparable to learnable and easy examples, without a systematic shift toward lower probabilities.Thus, low initial reference-policy probability does not distinguish unlearnable examples.
  • 4.2 Clipping and KL regularization: Unlearnable examples are not disproportionately clipped, and clip-higher or removing the KL term leaves their dynamics and the learnability gap essentially unchanged.These findings indicate that resistance to learning is not caused by clipping mechanisms, KL constraints, or low initial reference-policy probabilities.

5. Unlearnability Suggests Representation Issue

Cross-example gradient analysis shows that unlearnable examples are optimization-space outliers with weak transfer from broader training data. Their correct-answer rollouts also contain lower-quality, less generalizable reasoning, and data augmentation fails to reliably improve learning or transfer.

  • Cross-example gradient analysis: Unlearnable examples have much lower gradient similarity with other examples, indicating that learned signals transfer weakly to this group.This explains why their reward gap persists even when the number of positive rollouts is controlled.
  • Cross-example gradient analysis: Easy examples share highly aligned gradients, whereas unlearnable examples are dissimilar to every group and behave as gradient-space outliers.The analysis samples correct rollouts under the initial policy and computes example-level GRPO gradients.
  • Reasoning quality: Correct rollouts for unlearnable examples contain substantially lower-quality reasoning than those for learnable examples at initialization.Reasoning quality correlates with initial success rate even when responses reach the correct final answer.
  • Reasoning quality: Flawed correct reasoning suggests models exploit ungeneralizable shortcuts or heuristics, making outcome-only RLVR signals noisy.The finding motivates concern about reward hacking through “fake reasoning” when intermediate reasoning steps are not validated.
  • Reasoning quality: During RLVR training, reasoning quality improves substantially for learnable examples but remains low for many unlearnable examples, widening their gap.Curriculum-learning experiments in Appendix A.5 further confirm the limited transfer to unlearnable data.
  • Data augmentation: Augmented data provides limited help: evaluation pass@1 saturates early, pass@16 drops after 20 steps, and improvements likely reflect overfitting.Even augmented subproblems with better learnability do not substantially improve unlearnable examples or transfer to the originals.
  • Data augmentation: Unlearnable examples remain distinct in gradient space despite augmentation, showing that semantic similarity does not guarantee optimization-space similarity or useful transfer.Gradient similarity with augmented data is substantially higher for learnable than unlearnable examples.

6. Discussion

The discussion characterizes RLVR unlearnability as a persistent phenomenon rooted in representation flaws rather than missing correct rollouts or inadequate exploration. It emphasizes mid-training as important for improving hard-example representations while noting experimental and definitional limitations.

  • Implications on Understanding of RLVR: Unlearnable examples persist despite correct rollouts during RLVR training, indicating that the phenomenon is not explained by a lack of positive rollouts or imperfect exploration.The study presents unlearnability as a counterintuitive limitation affecting a difficult subset of training data.
  • Implications for Reasoning LLM Training Pipelines: Unlearnability is fundamentally a representation issue that is difficult to address directly during the RL stage.Unlearnable examples appear to have insufficiently structured pre-trained representations for effective and generalizable learning.
  • Implications for Reasoning LLM Training Pipelines: Mid-training appears to improve gradient similarity on hard examples and produce initial policy models that benefit more from subsequent RL fine-tuning.These findings highlight the importance of the mid-training stage in reasoning LLM training pipelines.
  • Limitations: The experiments are limited to small- and mid-scale base and instruct models in mathematical reasoning, and unlearnability is defined using a thresholded convergence criterion.The threshold is presented as an operational tool rather than a sharp categorical distinction, leaving some examples near the boundary.
  • Implications on Understanding of RLVR: Gradient analysis identifies unlearnable examples as optimization-space gradient outliers with likely incoherent reasoning traces.The study also reports that data augmentation and curriculum learning fail to improve gradient similarity or reasoning quality.

A. Additional Results · A.1. Results on Llama-3.2-3B-Instruct

On Llama-3.2-3B-Instruct, positive-rollout scarcity and gradient clipping do not explain unlearnability. Unlearnable examples instead show low gradient alignment, while augmentation fails to improve their learnability; Llama also has flatter, less aligned gradients than Qwen.

  • A.1. Results on Llama-3.2-3B-Instruct: Llama-3.2-3B-Instruct reproduces the finding that positive-rollout scarcity and gradient clipping do not explain unlearnable examples.The paper reports this through corresponding control analyses on the Llama training data.
  • A.1. Results on Llama-3.2-3B-Instruct: Unlearnable examples are more likely to be outliers in Llama’s optimization space.Figure 13 presents the cross-example gradient similarity distribution supporting this characterization.
  • A.1. Results on Llama-3.2-3B-Instruct: Data augmentation does not help Llama learn unlearnable examples effectively.The augmentation analysis is presented alongside the gradient-similarity results.
  • A.1. Results on Llama-3.2-3B-Instruct: The Llama analysis uses the model’s corresponding training data to test the same unlearnability hypotheses.The reported analyses cover rollout controls, gradient similarity, and data augmentation.
  • A.1. Results on Llama-3.2-3B-Instruct: Llama gradients are overall less aligned than Qwen gradients.The paper attributes this comparison to differences in overall gradient alignment between the models.
  • A.1. Results on Llama-3.2-3B-Instruct: For Qwen, easy-example gradients have cosine similarity above 0.7, whereas Llama gradients are largely below 0.6 and have a much flatter distribution.The contrast is offered as a partial explanation for Qwen’s greater suitability for RL training.
  • A.1. Results on Llama-3.2-3B-Instruct: Unlearnable examples have lower gradient similarity with the rest of the examples and with augmented data.This indicates that their gradients remain poorly aligned both with the original training set and its augmentations.

A.2. Further Results on Positive Rollout Scarcity Hypothsis · A.3. Hypothesis 3: Gradient Interference

Additional experiments reject positive rollout scarcity as the cause of unlearnability: unlearnable examples resist both stronger supervision and denser correct rollouts. Gradient analyses likewise find little interference within prompts or across batches, so cancellation by incorrect-rollout gradients does not explain their failure to learn.

  • A.2. Further Results on Positive Rollout Scarcity Hypothsis: Together, stronger supervision and denser positive rollouts fail to resolve unlearnability, providing further evidence against positive rollout scarcity.The additional experiments extend beyond the kpos = 1 setting.
  • A.2. Further Results on Positive Rollout Scarcity Hypothsis: SFT on 800 filtered correct responses leaves unlearnable examples resistant to learning, while learnable and easy examples improve as expected.The dataset contains 200 unlearnable, 200 learnable, and 400 random easy examples.
  • A.2. Further Results on Positive Rollout Scarcity Hypothsis: Even direct supervision on correct responses, a stronger signal than RLVR, does not close the gap between unlearnable examples and the rest of the data.Figure 15 contrasts stronger supervision via SFT with a larger rollout group under RL.
  • A.2. Further Results on Positive Rollout Scarcity Hypothsis: With rollout group size k = 64 and unlearnable-only RL training, average reward does not improve meaningfully despite more correct rollouts per step and many epochs.This directly tests whether denser positive rollouts resolve resistance to learning.
  • A.3. Hypothesis 3: Gradient Interference: At the initial policy, correct and incorrect rollout gradients within each hard example are highly similar, with no distinction between learnable and unlearnable data.Cosine similarity below 0 would indicate potential interference; the reported initial similarities do not distinguish the groups.
  • A.3. Hypothesis 3: Gradient Interference: Cross-prompt analysis finds no clear gradient-interference difference between learnable and unlearnable groups when considering incorrect rollouts across the batch.This extends the analysis beyond rollouts from the same example.
  • A.3. Hypothesis 3: Gradient Interference: Overall, gradient interference is small at both rollout and batch levels, so incorrect-rollout gradients cancelling correct-rollout gradients cannot explain unlearnable examples’ failure to learn.The proposed mechanism involves token-level gradients combined with rollout-level rewards and advantages, which can create imprecise credit assignment.

A.4. Cross-Example Gradient Similarity During Training · A.5. Curriculum Learning Does Not Improve Reasoning Quality · B. Experimental Details

During training, unlearnable examples retain lower cross-example gradient similarity than learnable examples, while curriculum learning fails to improve their reasoning quality or success rate. Although first-stage training on learnable data substantially improves the model, switching to unlearnable examples yields no benefit and can reduce validation accuracy.

  • A.4. Cross-Example Gradient Similarity During Training: Unlearnable examples retain lower gradient similarity than learnable examples during training, even as optimization drastically changes the similarity distribution for easy data.Figure 17 measures gradient similarity after 50 training steps.
  • A.4. Cross-Example Gradient Similarity During Training: The training-time gradient analysis therefore preserves the distinction between unlearnable and learnable examples despite optimization-driven changes in easy-data similarity.The reported distributions show that optimization changes easy data substantially, whereas unlearnable examples remain below learnable counterparts.
  • A.5. Curriculum Learning Does Not Improve Reasoning Quality: Curriculum learning was hypothesized to improve reasoning quality by training sequentially on high-quality rollouts from easier examples.This hypothesis follows prior observations about reasoning quality and curriculum learning’s reported efficiency benefits for RLVR.
  • A.5. Curriculum Learning Does Not Improve Reasoning Quality: The curriculum setup first excludes unlearnable examples until convergence, then adds randomly sampled examples matching the excluded group’s size for continued training.The second-stage training set combines the unlearnable portion with an equally sized random sample from first-stage data.
  • A.5. Curriculum Learning Does Not Improve Reasoning Quality: Curriculum training does not improve reasoning quality or success rate on unlearnable data, despite substantial first-stage model improvement.Continuing into the second stage provides no benefit on unlearnable examples.
  • A.5. Curriculum Learning Does Not Improve Reasoning Quality: Switching from first-stage learnable-data training to second-stage training produces an obvious drop in validation accuracy.Figure 18 analyzes curriculum reward dynamics and post-first-stage reasoning-quality distributions across groups.

B.1. RLVR Training Details … C.1. Prompts

The paper specifies RLVR training configurations, an oversampling-and-replay variant of GRPO, and prompt templates for synthesizing related problems, decomposing problems, and annotating reasoning quality.

  • B.1. RLVR Training Details: RL fine-tuning uses verl 4 with model-dependent sampling batch sizes, rollout groups, gradient batches, and learning rates.Sampling batches are 256 for Qwen2.5-3B and Llama-3.2-3B-Instruct, 1024 for Qwen2.5-0.5B; rollout group size is 8, gradient-step batch size is 64, and rollout-level total batch size is 512.
  • B.2. Algorithm for Oversampling with Rollout Replay: The sampling intervention modifies standard GRPO through oversampling and a rollout replay buffer.Algorithm 1 presents the specific intervention applied to standard GRPO in Section 4.1.
  • B.2. Algorithm for Oversampling with Rollout Replay: The algorithm takes policy models, training data, batch and rollout parameters, positive-sample counts, and a learning rate as inputs.Its required inputs are πθ, πref, Dtrain, batch size N, rollouts per prompt k, positive samples per prompt kpos, and learning rate α.
  • B.2. Algorithm for Oversampling with Rollout Replay: For each iteration, the method initializes an empty replay buffer and samples a batch of training examples.The procedure then gathers positive and negative rollouts for each example and adds rollout information to the buffer.
  • B.2. Algorithm for Oversampling with Rollout Replay: When too few positive rollouts are available, the algorithm samples additional rollouts from the replay buffer and combines them with negative samples.It supplements each prompt’s positive set to reach k rollouts, samples k − kpos negatives, and forms the set used for advantage computation.
  • C.1. Prompts: The related-problem synthesis prompt asks for 5 solvable, well-defined problems preserving core skills, reasoning patterns, difficulty, and structure while varying surface details.It requires mathematical-expression or numerical answers and requests valid JSON containing problem, solution, and answer fields.
  • C.1. Prompts: The subproblem prompt decomposes a mathematical problem into independent, self-contained subproblems with clear answers and a logical progression toward the original solution.Solutions should show complete step-by-step reasoning, use LaTeX for mathematical expressions, and end with a numerical value or single mathematical expression.
  • C.1. Prompts: The reasoning-quality annotation prompt scores a correct-answer reasoning process from 0 to 5 and requires a score line followed by justification.The rubric defines 0 as completely wrong and 5 as perfect, emphasizing logical coherence and consistency.

C.2. Example Data · D. More Case Studies on Low-Quality Reasoning

Augmented problems can closely resemble unlearnable examples yet retain low gradient similarity, reinforcing representation flaws as a source of unlearnability. Additional case studies illustrate reasoning traces with quality scores of 1, 3, and 2.

  • C.2. Example Data: Augmented problems often differ from original unlearnable examples only in numeric values or notation, yet their gradient similarity remains low.The authors interpret this as evidence of inner representation flaws in unlearnable examples.
  • C.2. Example Data: Case Study 1 reports case id 61 with gradient cosine similarity 0.30.This case is presented among augmented examples whose structures closely resemble original unlearnable problems.
  • D. More Case Studies on Low-Quality Reasoning: Case Study 2 reports case id 15 with gradient cosine similarity 0.37.It provides another example of structurally similar augmented data retaining low similarity to the original problem.
  • D. More Case Studies on Low-Quality Reasoning: An example reasoning trace is presented with quality score 1.The trace is included as a case study of low-quality reasoning.
  • D. More Case Studies on Low-Quality Reasoning: Another example reasoning trace is presented with quality score 3.The trace extends the case studies examining reasoning quality.
  • D. More Case Studies on Low-Quality Reasoning: A further example reasoning trace is presented with quality score 2.The trace contributes another case study of low-quality reasoning.
Loading 2605.16787v1…