Demystifying Reinforcement Learning Post-Training of Language Models

Donovan Clay; Saket Gollapudi; Sankar Harilal; Min Jang; Jacob Morrison; Sewoong Oh; Natasha Jaques

Source-linked AI summary

Demystifying Reinforcement Learning Post-Training of Language Models

Donovan Clay, Saket Gollapudi, Sankar Harilal, Min Jang, Jacob Morrison, Sewoong Oh, Natasha Jaques

arXiv:2608.24949v1Aug 24, 2026cs.LGcs.AIcs.CL

TL;DR

RL post-training remains difficult to interpret because the mechanics governing its success are not well understood. Through controlled RLVR experiments, this paper shows that base-model coverage, reward structure, and prompt diversity shape outcomes, including whether sparse RL can learn desired behaviors.

Problem
The mechanics governing how base-model distributions, reward signals, prompt distributions, and model scale affect RL post-training outcomes remain insufficiently understood.
Method
The paper uses controlled sequence-generation and mathematical-reasoning environments to manipulate base distributions, reward sparsity, prompt diversity, and model scale.
Results
Standard sparse RL cannot surpass initialization limits without sufficient target coverage, whereas dense rewards can guide successful learning from low-coverage base models.
Takeaways & Limitations
RL post-training outcomes depend on the interaction between exploration enabled by base-model coverage, reward granularity, and prompt-distribution breadth.

Abstract

from arXiv · show

Reinforcement learning (RL) post-training has emerged as a powerful framework for enhancing the capabilities of large language models (LLMs), enabling impressive reasoning, math, and coding capabilities. Yet for many researchers and practitioners, the principles behind classical RL remain a "black box". In this work, we deconstruct the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface. By isolating the mechanics of RL with Verifiable Rewards in a controlled and simplified environment, we examine how RL outcomes are shaped by the base model's prior distribution, the granularity of the reward signal, the diversity of the prompt distribution, and model scale. We use the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shapes model certainty. Our investigation sheds light on how these choices interact to affect post-training success. For example, we show that the effect of so-called 'spurious rewards' depends on the prompt distribution used for post-training. We also provide insight into why the success of RL post-training depends on whether the base model already places sufficient probability mass on the desired behavior, linking it to the classical concept of exploration in RL. Ultimately, we provide this primer as a resource to those in the NLP community wishing to incorporate RL as a tool in their toolbox.

1 Introduction

This paper demystifies RL post-training by decomposing how sampled responses, scalar rewards, and parameter updates reshape an LLM’s output distribution. It studies these dynamics in controlled RLVR settings by varying base-model knowledge, rewards, prompts, and model scale.

Motivation: RL has become central to LLM post-training for aligning models with human intent, improving reasoning, and ensuring safety, but its mechanisms remain widely misunderstood.Understanding RL post-training requires expertise in both classical RL and NLP.
RL post-training mechanics: The post-training algorithm samples responses, assigns scalar quality rewards, and updates parameters to increase the likelihood of higher-reward outputs, including when rewards are nondifferentiable.Figure 1 presents this iterative process and isolates the base distribution, prompt distribution, and reward function while excluding algorithmic variation.
Distributional perspective: Unlike classical RL, LLM post-training starts from a knowledgeable base model and constrains the learned policy near that reference, redistributing probability mass within its pretrained distribution.The paper tracks policy entropy and probability mass to compare how pretraining, SFT, and DPO shape output distributions and resist corruption.
Experimental setting: Experiments use exactly measurable sequence-generation RLVR tasks—target-string production and math problem solving—to manipulate the base model’s pretrained distribution through SFT+ and SFT− interventions.SFT+ increases target-string probability, whereas SFT− disrupts that knowledge in the base model.
Experimental setting: Across model scales, the study evaluates sparse, dense, and random rewards under narrow reasoning prompts and the broad pretraining distribution used for Tulu-3 (Lambert et al., 2025).These manipulations test how reward signals and prompt distributions affect post-training outcomes.

2 Related Work

Prior work establishes RL post-training’s effectiveness while raising conflicting questions about spurious rewards and whether RL learns genuinely new behaviors. This work reconciles these findings by showing that prompt distribution, base-model probability, and reward density determine what post-training can learn.

2 Related Work: The literature reports strong RL gains for LLM reasoning and alignment, but also contradictory findings that motivate this study (Lambert et al., 2025; Ouyang et al., 2022).
2 Related Work: Spurious-reward gains depend on prompt distribution: narrow training can improve performance within the corresponding domain, rather than implying universal post-training success.This contrasts with the possibility that broad post-training gains are generally attributable to spurious rewards.
2 Related Work: The base model’s probability of sampling a response strongly affects how easily post-training learns it, extending prior analyses of probability-mass redistribution (Zhao et al., 2025; Rajani et al., 2025; Yang et al., 2026; Akgül et al., 2026).
2 Related Work: Sparse rewards limit learning behaviors with negligible base-model support, whereas sufficiently dense and correct rewards can teach such new behaviors, contrary to claims that RL only upweights existing behaviors (Yue et al., 2025; Shao et al., 2025; Wu et al.; Zhao et al., 2025; Wu et al., 20…).Offline methods such as DPO tend to concentrate probability mass on responses already likely under the base model (Ren and Sutherland, 2025), while online RL can explore by generating new data.
2 Related Work: By systematically varying these components, the work reconciles seemingly contradictory findings about RL post-training.A comprehensive literature review is provided in Appendix B.

3 Background: RL for Language Models

RL post-training models language generation as a finite-horizon MDP whose deterministic token transitions and sequence-level verifiable rewards reduce the problem to a contextual-bandit setting. The policy maximizes expected reward while remaining close to a reference SFT policy, and intermediate-step rewards can support exploration beyond sparse sequence-level feedback.

3 Background: RL for Language Models: Language generation is modeled as a finite-horizon MDP in which states are partial token sequences, actions are vocabulary tokens, transitions append tokens deterministically, and the horizon limits generation length.The reward is a deterministic verifier applied to the completed sequence, such as binary mathematical correctness.
3 Background: RL for Language Models: Sequence-level verifiable rewards make text-generation RL a simplified contextual bandit rather than a general sequential-control problem.From an NLP perspective, the task is learning a probability distribution over complete sequences.
3 Background: RL for Language Models: Process Reward Models evaluate intermediate reasoning steps, enabling partial-credit assignment and iterative reshaping toward behaviors that are unlikely under the base model and inaccessible under sparse rewards.This provides a mechanism for exploring low-probability behaviors through step-level credit rather than only completed-sequence rewards.
3 Background: RL for Language Models: The language model acts as a stochastic policy πθ(y|x) trained to maximize expected reward while staying close to a reference policy, typically the initial SFT model, to limit reward hacking and fluency degradation.The objective includes a KL-divergence penalty scaled by c over prompts drawn from dataset D.

4 The Mechanics of RL Post-Training

The section decomposes RL post-training into the base model, reward function, and curated prompt dataset, then tests how their interaction shapes learning in controlled sequence-generation and mathematical-reasoning environments. It emphasizes that initial target coverage, reward sparsity, and prompt distribution govern exploration and the effects of reinforcement.

4 The Mechanics of RL Post-Training: RL post-training combines a base model, reward function, and curated prompt dataset, which the experiments isolate across target-string generation and mathematical reasoning.Two phases examine base-model distributions and reward functions, then prompt-distribution effects.
4.1 Base-Model Coverage and Exploration: Near-zero initial probability of the target creates an exploration bottleneck because the policy may never generate it and therefore receive a learning signal.The study manipulates target-behavior probability through Base, SFT+, and SFT− initialization variants, including a setting that suppresses target sampling to nearly 0%.
4.2 Reward Sparsity and Shaping: Binary sequence-level rewards make RLVR highly sparse, whereas edit-distance and process-reward proxies provide intermediate feedback based on structural proximity or valid reasoning subcomponents.The sequence task uses an exact-string verifier with a length penalty, while the mathematical task uses an exact final-answer verifier or a PRM awarding partial credit for five valid sub-components.
4.3 Prompt Distribution and Spurious Rewards: Random rewards can reinforce correct answers and reduce policy entropy on narrow prompt distributions when the base model already favors them, but broad prompts reward arbitrary responses instead.The prompt distribution determines which states are explored and which highly probable base-model responses are disproportionately rewarded by chance.
4.3 Prompt Distribution and Spurious Rewards: Qwen2.5-7B-Math reproduces the spurious-rewards phenomenon because of its strong math prior, whereas OLMo 3 shows no artificial evaluation improvement under random rewards.The comparison uses random rewards sampled uniformly from [0, 1] and contrasts models with different initial priors.

5 Results: Unpacking Learning Dynamics

RL post-training succeeds when exploration receives either sufficient prior probability mass on complete target behaviors or dense intermediate rewards. Spurious-reward outcomes instead depend on prompt-distribution breadth and the base model’s existing domain bias, with narrow distributions sharpening priors and broad distributions causing degradation.

Coverage and exploration: SFT+ models quickly maximized sparse rewards, whereas SFT- and unmodified Qwen3-8B Base models failed when the target behavior lacked sufficient prior probability mass.Qwen2-7B eventually converged from an approximately 3.5% prior after a 40-step exploration plateau, while Qwen3-1.7B failed from an approximately 0.5% prior.
Coverage and exploration: Approximately 85% exact match was achieved by the injected-reasoning model, while the unmodified Base model stalled near 10% on AIME Problem 4 with sparse rewards.The complete reasoning chain was rarely sampled under the base distribution, leaving almost no positive learning signal.
Dense rewards: Over 90% exact match was reached on AIME with exponentially increasing stepwise rewards, versus approximately 10% with sparse rewards.Dense rewards overcame the exploration bottleneck even though the base model stalled under sparse optimization.
Spurious rewards: Spurious rewards improved Qwen capabilities under a narrow prompt distribution but minimally improved MATH and degraded AMC under a broad distribution.The results show that prompt-distribution breadth determines whether random rewards sharpen useful behavior or cause broader degradation.
Spurious rewards: Under broad prompts, OLMo entropy increased and capabilities fell, whereas narrow prompts reduced entropy and dropped GSM8K from 86% to approximately 32% while largely preserving MMLU and IFEval.Narrow-distribution repetition collapses the policy toward its prior, corrupting the training domain without creating new capabilities.

6 Conclusion · A Resource Details

The paper deconstructs RL post-training for LLMs by isolating the base-model distribution, prompt distribution, and reward structure in a controlled RLVR sequence-generation sandbox. It aims to make RL post-training more interpretable and less of a “black box” for NLP researchers and practitioners.

6 Conclusion: The study examines how the base-model distribution, prompt distribution, and reward structure affect RL post-training optimization outcomes.These components are isolated within a controlled sequence-generation sandbox to explain and empirically verify post-training behavior.
6 Conclusion: The controlled RLVR sandbox explains, empirically verifies, or provides counterpoints to existing post-training results in the literature.
6 Conclusion: Random rewards over a broad prompt distribution increase model entropy while causing a catastrophic decrease in capabilities.
A Resource Details: The authors present the work as a resource for making RL post-training a more controlled and interpretable optimization process rather than a “black box.”The intended audience includes the NLP community and researchers seeking to sharpen their understanding of RL post-training mechanics.

A.1 Computational Specifications · A.2 LLM Usage · B Expanded Related Work

The appendix reports the computational setup, explains how LLMs supported experiment development, and situates the study within related work on RL post-training, reward reliability, base-model coverage, and probability redistribution.

A.1 Computational Specifications: Experiments used NVIDIA A100 80GB or L40s 48GB GPUs, with A100s for main math-reasoning experiments and L40s for the PRM LLM-as-a-Judge.
A.2 LLM Usage: LLMs assisted code-base and experiment development by adapting behavior to resource limitations and simplifying the result-generation pipeline.
B Expanded Related Work: RL post-training improves reasoning and preference alignment, while RLVR can achieve strong performance with smaller models, exemplified by DeepSeek-R1 (Lambert et al., 2025).
B Expanded Related Work: Shao et al. (2025) report mathematical gains from RLVR despite weak, random, or negatively correlated rewards, with spurious rewards shifting code-reasoning behavior in Qwen2.5-Math.
B Expanded Related Work: Yue et al. (2025) find RL-trained models excel at small pass@k while base models catch up at larger k, suggesting RLVR mainly improves sampling efficiency rather than creating novel behaviors.
B Expanded Related Work: Chen et al. (2025) argue that coverage—the base model’s probability mass on high-quality responses—predicts post-training success better than metrics such as cross-entropy loss.
B Expanded Related Work: Ren and Sutherland (2025) explain post-training behavior through probability-mass redistribution after gradient updates, though their analysis is restricted to DPO.

C Reward Function Implementations - Movie Quotes · C.1 Sparse Reward with Length Penalty

The movie-quotes sandbox compares sparse and dense reward implementations to study reward shaping in RL optimization. The sparse variant gives binary success credit while penalizing extra generated tokens to discourage rambling and reward hacking.

C Reward Function Implementations - Movie Quotes: The sandbox evaluates reward shaping using a sparse anti-rambling reward and a dense Levenshtein-based reward.The generated sequence is y, and the exact target string is τ.
C.1 Sparse Reward with Length Penalty: The sparse reward assigns 1 when the model generates the target string τ and 0 when it fails to generate the target entirely.
C.1 Sparse Reward with Length Penalty: A length penalty p reduces the reward for extra tokens generated beyond the target, preventing reward hacking through rambling.Here, n is the maximum token generation limit and s is the number of extra tokens.
C.1 Sparse Reward with Length Penalty: The penalty is defined using the number of extra tokens s relative to the maximum generation limit n.The supplied passage displays the expression only partially, so no additional formula details are inferred.
C.1 Sparse Reward with Length Penalty: The sparse reward function incorporates the length penalty into the target-matching reward.The supplied excerpt introduces the function definition but does not provide its complete displayed equation.

D Exponential Process Reward Model (PRM) Implementation · C.2 Dense Reward (Levenshtein Distance)

The paper contrasts dense structural rewards based on Levenshtein proximity with a PRM that evaluates mathematical reasoning through strategy classification and sequential logical milestones. These reward designs address credit assignment while discouraging imprecise or rambling behavior.

D Exponential Process Reward Model (PRM) Implementation: The PRM evaluates each generated mathematical trajectory after rollout by classifying its strategy and checking sequential logical milestones.This replaces string matching with logical verification across diverse valid solution paths, using Qwen2.5-32B-Instruct.
D Exponential Process Reward Model (PRM) Implementation: The PRM was introduced to scale analysis from structural sequence matching to combinatorial mathematical reasoning, where solutions can follow diverse valid paths.Examples include direct factoring, the quadratic formula, and dehomogenization.
D Exponential Process Reward Model (PRM) Implementation: The sparse optimization baseline assigns reward only when the precise target appears, while penalizing excess-length outputs to discourage rambling.The penalty p may be fixed or depend on the excess length |y| − |τ|.
C.2 Dense Reward (Levenshtein Distance): Dense reward provides continuous intermediate gradient signals based on structural proximity to the target, addressing credit assignment when the model lacks initial probability mass.The construction uses the minimum number of insertions, deletions, or substitutions needed to transform the generated sequence into the target.
C.2 Dense Reward (Levenshtein Distance): Levenshtein distance is defined through dynamic programming over prefixes, accounting for insertion, deletion, and substitution costs.For sequences y and τ, D(i,j) measures the edit distance between their first i and j characters, respectively.
C.2 Dense Reward (Levenshtein Distance): The dense reward normalizes the Levenshtein distance using n = max(len(y), len(τ)) before converting structural similarity into reward.The supplied formulation defines D as the distance between the complete generated and target sequences.

D.1 Logical Milestones and Reward Curve

The method decomposes the multi-step AIME derivation into five milestone-based PRM signals. Because linear partial credit can create local minima, an exponentially scaled reward curve assigns increasing marginal value to advanced reasoning steps.

Logical Milestones: The PRM extracts five milestones covering algebraic setup, root relationships, boundary constraints, branch counting, and origin-overlap subtraction.These milestones operationalize successful completion of the AIME derivation as a boolean vector M(y) ∈ {0, 1}^5.
Reward Curve: Linear partial credit can create local minima when halting after three steps yields a “good enough” reward, weakening the gradient toward the final answer.This advantage compression discourages the policy from risking further token generation.
Reward Curve: The reward curve uses step weights w = [0.05, 0.05, 0.10, 0.15, 0.25] to prevent advantage compression.The weights define the base process reward used to map milestone completion into a reward signal.
Reward Curve: Exponential scaling enforces increasing marginal value, giving advanced reasoning steps a stronger gradient pull than early exploratory steps.This scaling is intended to encourage continuation toward later milestones rather than premature stopping.

D.2 Outcome Overrides and Defensive Penalties · E RLVR Algorithm

The method combines deterministic outcome overrides with penalties for looping and formatting failures, then trains the base model with RL to maximize the resulting reward. This reward structure is intended to prevent reward hacking and bridge exploration gaps for models with near-zero initial priors.

D.2 Outcome Overrides and Defensive Penalties: The final reward anchors the PRM to absolute ground truth through deterministic outcome overrides and active penalties, preventing reward hacking.The unpenalized reward assigns 1.0 to the exact answer, 0.6 to a known nearmiss, and rprocess(y) otherwise.
D.2 Outcome Overrides and Defensive Penalties: The PRM penalizes cyclic reasoning and missing final formatting, applying ploop = −0.3 for loops and pformat = −0.5 without a concluding \boxed{} command.These penalties target redundant token farming and trajectories that fail to produce the required final answer format.
D.2 Outcome Overrides and Defensive Penalties: The final dense reward is bounded at 0 after combining the unpenalized reward with loop and formatting penalties.The bounded formulation prevents the combined penalties from producing a negative training reward.
D.2 Outcome Overrides and Defensive Penalties: By heavily penalizing redundancies and exponentially rewarding progress, the PRM bridges the exploration gap for models with near-zero initial priors.This is the stated effect of the combined reward and penalty design.
E RLVR Algorithm: Algorithm 1 takes a base model πref, prompt dataset D, and reward function r(x, y) as inputs for RLVR training.The algorithm specification identifies these three components as its inputs.
E RLVR Algorithm: When training time arrives, the algorithm updates the model with an RL method of choice to maximize the specified objective.The supplied algorithm excerpt explicitly includes a training-time condition followed by an RL update step.

F Prompt Distribution Experimental Details · G Model Sizes

The experiments use broad-domain random-reward RLVR training and track entropy and baseline-response likelihood over time. Across model sizes, RL success depends on the base model’s initial target-behavior probability and reward granularity.

F Prompt Distribution Experimental Details: Broad-distribution training uses 10,000 OLMo 3 RLVR prompts evenly split across mathematics, instruction following, and code, with uniformly random rewards and fully on-policy updates.Experiments use the standard OlmoRL codebase, random scalars from [0, 1], async steps set to 0, and three 8xH100 nodes.
F Prompt Distribution Experimental Details: Evaluation samples 200 prompts per domain, split between 100 training prompts and 100 held-out out-of-distribution prompts, and measures fixed baseline-response likelihood and per-token entropy every 50 steps.The fixed responses are sampled from the base model before RL training, enabling tracking of entropy and unlearning dynamics during random-reward finetuning.
G Model Sizes: > 98% match rates are reached by both SFT+ and Base Qwen2.5-7B-Instruct and Qwen2-7B models under dense and sparse rewards, whereas SFT- completely fails.Before RL, SFT+ has 28.8% initial mass and Base 3.5% for both 7B models.
G Model Sizes: > 90% match rates are achieved only by SFT+ for Qwen2-1.5B under both reward regimes; Base improves slightly while SFT- remains unsuccessful.Initial target-sequence mass is 8.0% for SFT+, 0.3% for Base, and 0.0% for SFT-.
G Model Sizes: 48.8% versus 10.0% match rates show dense rewards substantially outperform sparse rewards for Base Qwen3-1.7B, while SFT+ succeeds under both signals.Initial coverage is 14.0% for SFT+ and near zero for Base and SFT-.
G Model Sizes: 100.0% versus 0% match rates show Qwen3-8B’s binary outcome: SFT+ succeeds under both sparse and dense rewards, while Base and SFT- fail completely.SFT+ starts with 42.8% target-behavior mass, whereas Base and SFT- start with exactly 0.0%.