Reinforcement Learning for LLM Post-Training: A Survey

Na (Claire) Cheng

• Figure 1 overviews key components in the reinforcement learning-based post-training pipeline for LLMs.The caption does not specify the components or their relationships.Source: Figure 1 · p. 2

1.1 Pretraining and SFT

• Pretraining optimizes next-token prediction, while SFT on curated instruction–response pairs teaches models to answer human queries; neither guarantees alignment with human values.The introduction notes that SFT models can still produce unhelpful, dishonest, or unsafe outputs.Source: §1.1 · p. 4, §1.2 · p. 4

1.2 RLHF

• RLHF trains a reward model from human preferences and optimizes the policy, typically with PPO and a KL penalty; DPO instead enables joint optimization using offline pairwise preference data.The RLHF/PPO pipeline requires four models in memory and on-policy rollouts at each gradient step.Source: §1.2 · p. 4

1.3 RLVR

• RLVR uses objectively verifiable outcomes, such as matching ground truth or passing code tests, and DeepSeek-R1 showed outcome-based GRPO eliciting chain-of-thought reasoning without supervised reasoning data.The cited example concerns mathematics and code generation.Source: §1.3 · p. 4

1.4 Contributions

• The survey identifies a gap in technically detailed comparisons of RLHF and RLVR methods and aims to connect their techniques for readers.Existing surveys predominantly emphasize empirical or qualitative comparisons.Source: §1.4 · p. 4
• The survey derives policy-gradient objectives and gradient coefficients for MLE, REINFORCE, RLHF, PPO, DPO, RLVR, and GRPO as a shared analytical foundation.It presents a unified framework that includes PPO-based RLHF, RLVR, and DPO-based alignment.Source: §1.4 · p. 4
• The survey organizes RLVR methods by prompt sampling, response sampling, and gradient coefficient, and treats offline DPO through response generation, reward modeling, regularization, and SFT integration.It also covers on-policy RLHF, RLAIF, iterative DPO, and Nash learning.Source: §1.4 · p. 5, §1.5 · p. 5
• The survey applies standardized notation to update rules and catalogs algorithmic attributes in appendix tables to support direct cross-method comparison.Catalogued attributes include reward signals, baselines, normalization, and regularization.Source: §1.4 · p. 5

1.5 Paper Organization

• The survey’s six-module pipeline runs from prompts and responses through reward models, reward signals, RL algorithms, and regularization.Its arrows depict prompts entering the LLM, generated responses being scored, and the algorithm updating the LLM under regularization constraints.Source: §1.5 · p. 5

2.1.1 The Unified Gradient Coefficient

• A unified token-level gradient form represents post-training methods through their data source, gradient coefficient, and stabilization mechanism.The data source may be offline, off-policy, or on-policy; stabilization includes mechanisms such as PPO clipping or a KL penalty.Source: §2.1.1 · p. 6

2.2 LLM Pretraining: MLE; 2.2.1 SFT

• Pretraining maximizes next-token likelihood with a constant coefficient that reinforces every token equally, while SFT applies the same constant coefficient to curated prompt-and-demonstration pairs.Pretraining needs no reward model or annotations but cannot distinguish preferred from dispreferred outputs; the curated SFT data acts as an implicit human-selection reward.Source: §2.2 · pp. 6–7, §2.2.1 · p. 7

2.2.2 RFT; 2.2.3 Online RFT

• RFT (Yuan et al., 2023a; Dong et al., 2023) reinforces correct sampled responses and gives incorrect ones zero weight, while online RFT samples from the current policy to explore beyond the fixed SFT distribution.RFT samples once from the SFT policy; online RFT retains the correctness coefficient while updating its data source to the current policy.Source: §2.2.2 · p. 7, §2.2.3 · pp. 7–8

2.3.1 REINFORCE; 2.3.2 Early Applications to Sequence Models

• REINFORCE pairs each action’s log-probability gradient with its future return; MIXER (Ranzato et al., 2016) combined cross-entropy pretraining with sequence-level REINFORCE rewards.MIXER showed sequence-level objectives could surpass token-level cross-entropy on translation and summarization, but was validated only on small recurrent models.Source: §2.3.1 · pp. 8–9, §2.3.2 · p. 9

2.4.1 AC methods; 2.4.2 TRPO and PPO

• Actor-critic methods replace high-variance returns with learned advantages; TRPO (Schulman et al., 2017a) constrains policy updates by KL, while PPO (Schulman et al., 2017b) uses a clipped first-order surrogate.The centered advantage reduces variance without bias, and PPO reuses previous-policy rollouts over multiple epochs; RLHF applies a reverse-KL penalty to a fixed SFT reference policy.Source: §2.4 · p. 9, §2.4.1 · p. 10, §2.4.2 · pp. 10–11

2.4.3 RLHF

• RLHF uses SFT demonstrations, a reward model trained on pairwise human preferences, and PPO constrained by a KL penalty to a reference policy.Anthropic’s study (Bai et al., 2022a) found alignment bonuses for 13B and 52B models on nearly all zero-shot and few-shot evaluations, while smaller models degraded on standard NLP benchmarks. Iterated online RLHF with fresh feedback yielded higher crowdworker Elo ratings; OpenAI’s pipeline improved preference win-rates and reduced hallucination and toxicity (Lin et al., 2022).Source: §2.4.3 · pp. 11–12

2.4.4 DPO

• DPO (Rafailov et al., 2023) trains offline on fixed preference pairs, deriving a classification loss that raises preferred-response likelihood without explicitly training a reward model.It removes the intractable prompt-dependent normalization term by using reward differences, and its gradient emphasizes hard-to-separate pairs.Source: §2.4.4 · pp. 12–13

2.5.1 GRPO and RLOO

• GRPO (Shao et al., 2024) replaces PPO’s learned value baseline with group-normalized response scores and applies the KL term separately from its clipped raw-reward advantage.For each prompt, it samples a group of responses from the previous policy; unlike PPO-based RLHF, its KL term is not folded into the reward before advantage estimation.Source: §2.5.1 · pp. 13–14
• GRPO differs from PPO through group-normalized response-level advantages applied uniformly to each token and a single update per rollout batch; RLOO uses a leave-one-out baseline.RLOO predates GRPO and fixes the standard deviation to 1; GRPO refreshes the old policy after each update (Shao et al., 2024).Source: §2.5.1 · p. 14

2.5.2 DeepSeek R1

• DeepSeek-R1-Zero uses GRPO and rule-based rewards without SFT or neural reward models, and reasoning behaviors including self-verification and reflection emerge during training.The described behaviors include a spontaneous “aha moment” in which the model pauses and re-evaluates its reasoning.Source: §2.5.2 · p. 15
• Full DeepSeek-R1 alternates cold-start SFT, reasoning-focused GRPO, rejection sampling plus SFT, and all-scenario GRPO with preference rewards.The reasoning-focused stage uses rule-based rewards; the later all-scenario stage adds model-based helpfulness and safety rewards after reasoning skills have been established.Source: §2.5.2 · p. 15

2.5.3 GRPO and RLVR Enhancement

• GRPO and PPO use token-level importance ratios and symmetric clipping, while later RLVR work explores prompt filtering, alternative ratios and clipping, reward signals, baselines, and length normalization.When every response in a group is correct or incorrect, advantages are zero; dynamic sampling retains non-trivial prompts to address this degenerate case. Alternatives include sequence-level or no importance ratios, asymmetric clipping, outcome or process rewards, and group-level or omitted length normalization.Source: §2.5.3 · p. 15

2.6 Hybrid Post-Training (HPT)

• Hybrid Post-Training switches per prompt between Dr. GRPO when the group mean reward exceeds threshold κ and SFT otherwise, selecting the signal according to rollout competence.The default κ is 0 for Qwen models and 2/8 for LLaMA; the motivation is that on-policy RL can fail without correct rollouts, while SFT can suppress exploration after surpassing demonstrations (Lv et al., 2025).Source: §2.6 · p. 16

2.7 The Art of Scaling RL: ScaleRL

• The 17B×16 MoE matches and surpasses the 8B dense model using one-sixth of the RL compute, while ScaleRL identifies four orthogonal scaling axes.Longer generation budgets and larger batch sizes raise the performance ceiling A at the cost of lower initial compute efficiency B; joint math-and-code training preserves per-domain sigmoidal scaling.Source: §2.7 · p. 16

2.8 Individual Paper Summaries

• The survey’s RLVR and RLHF/DPO algorithms share a policy-gradient form, and its per-paper tables connect method design choices with the models, data, benchmarks, and comparisons used to validate them.RLVR generates and filters groups of responses scored by verifiable rewards; RLHF/DPO branches into fresh on-policy rollouts or fixed-dataset offline responses, with configurable regularization and optional SFT merging.Source: §2.8 · pp. 16–17

3.1 Prompt Generation

• Human-curated prompts offer high-quality training signal but require annotation effort and may miss parts of the difficulty spectrum; synthetic generation automates prompt creation.Prompt selection then ranges from static curricula to adaptive difficulty and reward-based filtering.Source: §3.1.1 · p. 17, §3.1 · p. 17, §3 · p. 17

3.1.2 Synthetic Data

• Absolute Zero uses one policy to propose and solve program-triplet tasks, rewarding moderately difficult proposals and jointly optimizing both roles with TRR++.Its tasks cover deduction, abduction, and induction; TRR++ uses one update response per prompt and task-role-grouped advantage baselines.Source: §3.1.2 · p. 19

3.2.1 Static Curriculum

• Qwen 3’s four-stage post-training includes reasoning RL with GRPO, where AIME’24 rises from 70.1 to 85.1 over 170 training steps.The stages are Long-CoT cold start, reasoning RL, thinking-mode fusion via SFT, and general RL; lightweight models also use distillation.Source: §3.2.1 · pp. 19–20
• OLMo 3’s OlmoRL builds on GRPO and combines six update changes, including zero-gradient filtering, active replenishment, asymmetric clipping, and truncated importance sampling.It also removes KL loss and normalizes loss by total batch tokens, while omitting standard-deviation normalization of group advantages.Source: §3.2.1 · p. 20
• Magistral modifies GRPO with no KL term, asymmetric clipping, group-length normalization, batch-standard-deviation advantage normalization, and dynamic filtering of zero-advantage prompts.It does not use entropy regularization.Source: §3.2.1 · p. 20

3.2.2 Adaptive Difficulty Curriculum

• AdaRFT updates a global target difficulty from current-policy batch reward, then samples prompts nearest that target for standard RL optimization.Prompt difficulty is estimated using reference-model reward; the target shifts harder or easier to keep success near a specified threshold.Source: §3.2.2 · pp. 20–21

3.2.3 Reward-based Filtering

• Reward-based filtering removes prompts with no gradient signal; SRPO’s history resampling drops all-correct groups while retaining mixed and all-incorrect groups for later epochs.SRPO also uses staged math-then-coding training and zeros advantages for responses exceeding the maximum length.Source: §3.2.3 · p. 21

4.1.1 On-policy Rollouts

• PPO and GRPO typically sample G responses from the current policy and use them for one or more optimization steps, treating slightly earlier policy versions as on-policy.The survey assumes on-policy learning unless a method explicitly specifies otherwise.Source: §4.1.1 · p. 22, §4.1 · p. 22

4.1.2 Off-policy: Replay Buffer

• Recency-based and reward-oriented replay retrieval consistently outperform full-scope and variance-driven strategies in RePO’s replay-buffer sampling.RePO augments on-policy responses with replay samples without additional rollouts, addressing zero-gradient collapse when sampled responses share the same reward.Source: §4.1.2 · pp. 22–23

4.1.3 Off-policy: Distillation

• LUFFY combines on-policy rollouts with stronger-teacher reasoning traces, while GVPO permits off-policy sampling without importance sampling when the behavior policy covers the reference policy’s support.LUFFY’s teacher traces support imitation when the model fails, while successful on-policy solutions preserve exploration; GVPO’s gradient weights avoid importance ratios, clipping, and group standard-deviation normalization.Source: §4.1.3 · pp. 24–25

4.1.4 Asynchronous Multi-node Training

• GEPO addresses stale-policy instability in asynchronous training by replacing each response’s importance-ratio denominator with a shared group-level estimate.High latency can widen the gap between sampling and learning policies, increasing importance-ratio variance.Source: §4.1.4 · p. 25

4.1.5 Prefix-conditioned Rollouts

• Prefix-RFT mixes an expert-prefix/on-policy continuation trajectory into rollouts and masks prefix-token gradients by entropy, while BREAD uses expert anchors to densify cold-start rewards.Prefix-RFT’s cosine decay shortens the expert prefix over training, shifting from imitation toward exploration; BREAD shortens its EAS-selected anchor as the policy improves.Source: §4.1.5 · pp. 25–26

4.1.6 Tree-structured Rollouts

• TreePO shares rollout prefixes in a branching tree and estimates advantages hierarchically across depths, reducing redundant KV-cache computation and enabling finer-grained credit assignment.Subgroup advantages are centered at each depth, then averaged across depths and normalized by global variance.Source: §4.1.6 · pp. 26–27
• VinePPO estimates token-level values by averaging returns from sampled alternative continuations; these auxiliary rollouts do not enter the policy gradient.The resulting gradient is unbiased for any G ≥ 1, with larger G reducing estimator variance at the cost of additional sampling.Source: §4.1.6 · p. 27
• S-GRPO forms a serial group from one reasoning path and early-exit answers, assigning exponentially decreasing credit to successive correct answers to address overthinking.It generates one complete path, then samples positions along it where the model is prompted to stop reasoning and answer.Source: §4.1.6 · p. 27
• Tree-structured rollouts reward earlier correct exits more highly, omit GRPO’s standard-deviation normalization, and remove the KL term while retaining its gradient-coefficient form.Source: §4.1.6 · p. 27

4.2.1 Positive/Negative

• Positive/negative separation treats correct and incorrect rollouts asymmetrically through different loss weights or masking; OREAL (Lyu et al., 2025) and TOPR (Roux et al., 2025) also use this approach.Source: §4.2.1 · p. 28

4.2.2 Reward-based Filtering

• GFPO samples a larger response group, filters to a subset using a target metric, and computes group-normalized advantages only for retained responses to address length inflation.Adaptive Difficulty GFPO varies the retained subset size by estimated question difficulty, keeping more responses for harder problems and pruning easy ones more aggressively.Source: §4.2.2 · p. 28

4.2.3 Advantage-based filtering

• PODS downsamples full rollout groups to maximize reward variance, retaining high- and low-reward responses to preserve contrastive signals while reducing costly policy updates.Its optimal subset consists of the k highest- and m−k lowest-reward responses; in the binary-reward case, it selects m/2 correct and m/2 incorrect responses.Source: §4.2.3 · pp. 28–29

4.2.4 Bootstrapped selection

• PKPO directly optimizes pass@k to preserve response diversity on hard problems, weighting only responses ranked k-th or higher and optionally annealing k toward 1.For binary rewards, incorrect responses receive zero gradient; annealing shifts from exploration toward individual-sample accuracy.Source: §4.2.4 · pp. 29–30

• Methods are compared along five axes: importance-sampling ratio, advantage shaping, advantage normalization, length normalization, and regularization.Source: §5 · p. 30
• PPO and GRPO serve as the two foundational baselines for this comparison.Source: §5 · p. 30

5.1.1 Token-level

• CISPO clips importance ratios inside a stop-gradient rather than zeroing out tokens outside GRPO’s clipping interval, so token gradients are not zeroed by clipping.Its described formulation uses outcome rewards and group mean-and-standard-deviation advantage normalization.Source: §5.1.1 · p. 30
• Beyond 80/20 Rule updates only the top-pρ fraction of highest-entropy tokens, while Clip-Cov and KL-Cov selectively suppress tokens associated with entropy collapse.Clip-Cov zeros selected high-covariance token gradients; KL-Cov instead applies a targeted forward KL penalty to its selected tokens.Source: §5.1.1 · pp. 31–32
• OREAL combines KL-constrained behavior cloning on Best-of-N positives with a co-trained token-level reward model for credit assignment, without a value network.It also incorporates negative samples through a separate penalty term.Source: §5.1.1 · p. 32

5.1.2 Sequence-level

• GSPO replaces per-token importance ratios with a length-normalized sequence-level ratio, aligning the optimization unit with the response-level reward.The sequence-level ratio is described as stable under MoE routing changes, avoiding the need for Routing Replay.Source: §5.1.2 · pp. 32–33
• GMPO replaces arithmetic-mean aggregation of token-level importance-weighted rewards with a geometric mean to damp outlier ratios and support a wider clipping range.The survey describes the geometric mean as yielding a narrower objective range and lower variance than GRPO.Source: §5.1.2 · p. 33

5.1.3 No Importance Sampling Ratio

• GPG removes the importance ratio, clipping, KL penalty, and reference model, using direct REINFORCE with a group-mean reward baseline and no standard-deviation normalization.AGE rescales batches with zero-gradient groups and defers batches when the rescaling factor exceeds a threshold.Source: §5.1.3 · p. 34

5.2.1 Outcome Reward

• Standard outcome rewards score complete responses, then subtract a baseline such as GRPO’s group mean or PPO’s learned critic to form the advantage.The reward can be represented at token level by assigning zero reward to earlier tokens and the outcome reward to the last token.Source: §5.2.1 · p. 34

5.2.2 Length Penalty

• LCPO (Aggarwal & Welleck, 2025) uses a prompt-specified token budget for exact-length or soft maximum-length rewards, while GRPO-λ (Dai et al., 2025a) applies length penalties only to groups with high correctness ratios.LCPO-Max penalizes only over-budget outputs, while LCPO-Exact penalizes deviation from the budget; GRPO-λ retains accuracy-prioritized rewards for other groups to reduce premature collapse.Source: §5.2.2 · p. 35
• GRPO-LEAD (Zhang & Zuo, 2025) penalizes verbose correct responses and reweights group-normalized advantages by problem difficulty, amplifying updates for correct hard-problem and incorrect easy-problem responses.Its reward assigns −1 to incorrect responses, and its modified GRPO coefficient uses the reweighted advantage without KL regularization.Source: §5.2.2 · pp. 35–36
• Ada-GRPO (Wu et al., 2025a) first teaches four reasoning formats with SFT, then inverse-frequency-scales their rewards with a cosine-decayed bias that fades as training stabilizes.The formats are Direct Answer, Short CoT, Code, and Long CoT.Source: §5.2.2 · p. 36

5.2.3 Self-Certainty Reward

• Entropy minimization (Agarwal et al., 2025) offers unsupervised fine-tuning, entropy-reward RL, and inference-time logit optimization; INTUITOR (Zhao et al., 2025b) instead uses online model confidence as a reward without gold-standard solutions.EM-INF updates logits while model parameters remain frozen, and its effectiveness depends on confidence correlating with correctness; INTUITOR uses the evolving policy’s self-certainty to avoid reward hacking.Source: §5.2.3 · pp. 36–37
• SEED-GRPO (Chen et al., 2025) reduces advantages for semantically uncertain prompts, whereas EMPO (Zhang et al., 2025d) replaces outcome rewards with semantic-entropy-derived cluster rewards and filters prompts outside entropy thresholds.SEED-GRPO also removes standard-deviation normalization, length normalization, and KL divergence; EMPO excludes prompts with either too-low or too-high semantic entropy.Source: §5.2.3 · pp. 37–38

5.2.4 Process Reward

• PRIME (Cui et al., 2025a) learns implicit token-level process rewards online from outcome labels, while PURE (Cheng et al., 2025b) concentrates credit on the lowest-reward reasoning step to limit error amplification.PRIME filters for prompts with mixed correct and incorrect rollouts; PURE combines transformed process rewards with outcome rewards using RLOO.Source: §5.2.4 · pp. 38–39

5.2.5 Baseline Estimation

• VC-PPO addresses PPO’s long-CoT length collapse via value pretraining and decoupled GAE (λcritic=1, λactor=0.95); VAPO adds four modifications, including length-adaptive GAE replacing VC-PPO’s fixed λpolicy=0.95.Source: §5.2.5 · pp. 39–40
• VAPO combines PPO’s length-adaptive λ and asymmetric clipping with a positive-example NLL term for correct responses, while retaining KL divergence.Clip-higher separates the clipping bounds to give low-probability tokens more room to increase and mitigate entropy collapse; VAPO also uses global token averaging.Source: §5.2.5 · p. 40
• ORZ (Hu et al., 2025b) replaces GRPO’s group-mean baseline with a learned token-level critic, whose low expected returns for repetitive states produce strongly negative token advantages.It uses vanilla PPO and batch-level advantage normalization; the critic is trained with terminal-only binary outcome rewards and GAE parameters γ = 1 and λ = 1.Source: §5.2.5 · pp. 40–41
• OPO (Hao et al., 2025) enforces exact on-policy updates without importance ratios or clipping and replaces the group-mean baseline with a variance-minimizing baseline.Under approximately orthogonal token gradients with identically distributed norms proportional to response length, its baseline simplifies to a length-weighted reward average; its advantage omits group-standard-deviation and per-response length normalization.Source: §5.2.5 · p. 41
• SPO (Xu & Ding, 2026) samples one response per prompt and uses a persistent Bayesian value tracker with KL-discounted past observations to estimate its baseline.It globally normalizes advantages across the batch, uses PPO-Clip with asymmetric clipping, and enables prioritized sampling of prompts with lower success rates.Source: §5.2.5 · p. 41

5.2.6 Partition Function Estimation

• MDPO (Kimi Team, 2025) refreshes the KL center to the current policy each iteration, while FlowRL (Zhu et al., 2025) matches reward distributions using a learnable partition function.MDPO also uses curriculum and prioritized sampling; FlowRL jointly updates the policy and partition-function network to drive its flow-balance residual toward zero.Source: §5.2.6 · p. 42
• GIFT aligns group-normalized implicit and explicit rewards with a convex MSE objective, and its response-level formulation outperforms the token-level version.It retains on-policy group sampling; group normalization cancels the implicit reward’s β and log Z(x) terms.Source: §5.2.6 · p. 43

5.2.7 Advantage Stability

• AAPO (Xiong et al., 2025a) adds a clipped reward-gap momentum term against a frozen reference response to prevent vanishing gradients when group-relative advantages approach zero.Because the reference policy remains frozen as the current policy improves, the momentum term grows over training; AAPO drops importance ratios, clipping, and the KL penalty.Source: §5.2.7 · p. 43

5.2.8 Hybrid SFT with RL

• SRFT (Fu et al., 2025) combines demonstrations with on-policy rollouts in one stage and uses entropy-aware weights to balance SFT and RL without manual scheduling or an SFT cold start.Its SFT weight suppresses the SFT contribution at high entropy, while its RL weight amplifies positive reinforcement at high entropy.Source: §5.2.8 · p. 44

5.3 Advantage Normalization

• Normalizing advantages before importance-sampling weighting controls effective learning rate and gradient variance; Figure 4 groups the strategies into four approaches.These are group normalization, two-step normalization, multi-objective normalization, and normalization-free methods.Source: §5.3 · p. 44, Figure 4 · p. 45

5.3.1 Beta Normalization

• BNPO replaces GRPO’s fixed group-standard-deviation denominator with an adaptive Beta density re-estimated each batch, yielding lower-variance gradients as correctness shifts.Its parameters are estimated by method of moments.Source: §5.3.1 · pp. 44–45

5.3.2 Multi-objective Normalization

• GDPO independently normalizes each reward before aggregation to prevent reward collapse, then re-normalizes the combined advantage across the batch.This prevents aggregate advantage magnitude from scaling with the number of rewards.Source: §5.3.2 · p. 46

5.3.3 Two-step Normalization

• REINFORCE++ combines local group normalization with batch-level normalization; its baseline variant first subtracts the group mean, then normalizes across the batch.The base variant instead normalizes batch-level advantages after folding a k1-style KL penalty into the outcome reward.Source: §5.3.3 · pp. 46–47

5.3.4 Normalization-free

• Normalization-free methods omit standard-deviation scaling, with Dr. GRPO retaining mean-centering to address length-dependent and variance-induced biases.GPG and AAPO also forgo standard-deviation normalization, relying on importance-sampling clipping, data filtering, or momentum for stability.Source: §5.3.4 · p. 47

5.4.1 Response Length Normalization

• Per-response normalization gives each response equal gradient weight but can bias training toward shorter outputs when combined with per-group advantage normalization.This is the standard choice in GRPO and PPO.Source: §5.4.1 · p. 48

5.4.2 Group Length Normalization

• Group-total normalization gives longer responses proportionally more gradient weight; DAPO and Lite PPO also use token-level aggregation.Lite PPO combines token-level aggregation with mixed advantage normalization; DAPO also uses dynamic sampling, Clip-Higher, and overlong reward shaping.Source: §5.4.2 · pp. 48–49
• Reinforce-Rej filters prompts with mixed-correctness responses and uses binary rewards directly as advantages, without group mean and standard-deviation normalization.Its authors attribute GRPO’s advantage over vanilla REINFORCE primarily to discarding prompts whose sampled responses are all incorrect, rather than to reward normalization.Source: §5.4.2 · pp. 48–49

5.4.3 Length Normalization-free

• Dr. GRPO removes per-response length normalization, advantage standardization, and KL divergence to address optimization biases in standard GRPO.Its advantage retains mean-centering of the group reward.Source: §5.4.3 · p. 49

5.5 Regularization

• KL regularization anchors the policy to a reference model, while entropy regularization maintains output diversity.Methods without explicit regularization rely on other stability mechanisms, including importance-sampling clipping, dynamic sampling, and mixed normalization.Source: §5.5 · p. 49

5.5.1 KL Regularization

• Most surveyed papers use KL divergence as a soft constraint on policy drift from a reference model.ProRL periodically resets its reference policy to a recent policy snapshot and resets optimizer state to prevent cumulative KL from suppressing useful gradient signal and stalling updates.Source: §5.5.1 · p. 50

5.5.2 Entropy Regularization

• Entropy methods differ in how they shape the bonus: entropy shaping uses a self-regulating negative feedback loop, while MAGIC activates its bonus below a target entropy threshold.MAGIC uses an adaptive threshold-based schedule; entropy shaping adjusts the advantage without altering gradient flow.Source: §5.5.2 · pp. 50–51

5.5.3 Regularization-free

• DAPO, Dr. GRPO, and Lite PPO remove both KL and entropy regularization, relying on clipping, prompt filtering, or mixed normalization for stability.Figure 6 presents a comprehensive taxonomy of RLHF and DPO methods.Source: §5.5.3 · p. 51, Figure 6 · p. 52

6.1 RLHF

• RLHF trains a reward model from pairwise human preferences and optimizes the policy with reinforcement learning; RLAIF uses an LLM judge instead of explicit human annotation for its reward signal.The described RLHF pipeline uses PPO to optimize reward subject to a KL penalty.Source: §6 · p. 52, §6.1 · p. 53

6 RLHF and Iterative DPO: On-Policy Methods

• The section’s unified algorithm organizes on-policy and offline methods around prompt sampling, response generation, reward signals, and gradient coefficients.Offline DPO uses a fixed preference dataset, while iterative DPO refreshes the training distribution through current-policy generations.Source: §6 · pp. 52–53

6.2 RLAIF

• RLAIF achieved performance comparable to RLHF in summarization and helpful dialogue, outperformed it on harmlessness, and Direct RLAIF surpassed Distilled RLAIF in alignment.The survey also reports that RLAIF could improve an SFT policy when the LLM labeler was the same size as the policy; OpenAI’s rule-based reward separated prompt categories that the help-only reward model struggled to distinguish.Source: §6.2 · pp. 54–55

6.3 Iterative DPO

• Iterative DPO generates responses from the current policy, forms preference pairs, and fine-tunes on them, making it an on-policy approach.Self-Rewarding Language Models use the same LLM to generate and judge candidate responses, then apply DPO to self-generated preference pairs.Source: §6.3 · pp. 55–56

6.4 Nash Learning based Methods

• Nash learning compares policies directly in preference space rather than optimizing a scalar reward from a Bradley-Terry model.Nash-MD is reported to converge with last-iterate policies; DNO uses batched on-policy learning and single-timescale updates, and its authors suggest it could approximate Nash equilibrium for general preferences.Source: §6.4 · pp. 56–58

7 DPO: Offline Methods

• Offline preference methods are grouped by response structure—pairwise comparisons, single responses with feedback, or ranked lists—and by their training-signal and SFT-integration choices.The survey also organizes methods by reward form and regularization strategy.Source: §7 · p. 58, §7.1 · p. 58

7.1.1 Pairwise Responses

• Other pairwise approaches include SLiC-HF, which calibrates offline from existing preference data, and DPOP, which penalizes drops in preferred-response likelihood below the reference.SLiC-HF uses a max-margin ranking objective and an SFT-anchoring regularizer; its sample-rank variant was reported to learn more stably and converge more robustly.Source: §7.1.1 · pp. 58–59
• Pairwise methods include variants that adjust the reference, β, or objective to address weak reference alignment, data-quality sensitivity, deterministic preferences, and overlooked reward scores.sDPO updates its reference across stages; β-DPO calibrates β by batch; IPO retains KL regularization under deterministic preferences; RPO uses explicit reward differences; GPO places several losses in one family.Source: §7.1.1 · pp. 59–61

7.1.2 Single Response

• Single-response methods use unpaired binary feedback or pointwise rewards: KTO uses a prospect-theoretic utility, DRO jointly optimizes policy and value, and UNA accommodates pairwise, binary, and scalar feedback.UNA is equivalent to DPO on pairwise data, improves over KTO on binary feedback, and outperforms DPO and KTO for scalar-reward distillation.Source: §7.1.2 · pp. 61–62

7.1.3 List Responses

• Listwise methods optimize rankings over multiple responses: RRHF matches rankings, PRO scales contrasts by reward gaps, and LiPO uses listwise preferences and relative label values.RRHF can integrate alignment into SFT; PRO uses an InfoNCE-based objective with dynamic temperature and combines alignment with SFT.Source: §7.1.3 · p. 63
• Llambda-loss ranked above the tested alternatives, followed by Llist_mle ≈ Lpair_logistic ≈ Lpair_hinge, Lsoftmax, Lpoint_sigmoid, and Lpoint_mse.The comparison reports the ordering of loss functions evaluated for LiPO.Source: §7.1.3 · p. 64

7.2 Reward

• Pairwise reward compares responses, pointwise reward assigns an absolute scalar score, and listwise reward ranks multiple responses simultaneously.The survey also distinguishes token-level pairwise rewards and negative rewards focused on suppressing undesired outputs.Source: §7.2 · p. 64, §7.2.2 · p. 64, §7.2.3 · p. 66

7.2.2 Token-level Pairwise Reward

• DPO’s response-level bandit formulation can be extended to token-level credit assignment by expressing each generated token’s reward in a token-level MDP.The survey describes this extension as making token-level credit assignment explicit and tractable.Source: §7.2.2 · pp. 64–65
• TDPO adds sequential forward-KL constraints at the token level, defining token-wise rewards whose sum gives the response reward.This replaces response-level regularization with token-level constraints in the autoregressive generation setting.Source: §7.2.2 · p. 65

7.2.5 Negative Reward

• D2O replaces human-labeled positive examples with on-policy LLM-generated responses paired with human-labeled negatives; NPO keeps only the negative component of DPO.D2O uses generated responses as anchors, while NPO is reported to significantly slow catastrophic collapse.Source: §7.2.5 · pp. 66–67

7.3 Regularization

• Offline alignment regularization addresses preference overfitting through entropy, alternative divergences, changes to reference-model use, and explicit length penalties.The survey notes that unconstrained optimization can lead to reward hacking or degenerate outputs.Source: §7.3 · p. 67, §7.3.1 · p. 67, §7.3.2 · p. 67

7.3.2 Divergence

• JSD matched RKL’s rewards with higher diversity, while FKL and α divergence showed better entropy with lower rewards.The experiments report a reward-diversity trade-off across the tested divergences.Source: §7.3.2 · p. 68

7.3.3 Reference Model

• SimPO removes the reference model with a length-normalized reward and reward margin, while R-DPO explicitly penalizes response length.SimPO aligns its training reward with the average log-likelihood metric used during generation; R-DPO’s α|y| term controls the length penalty.Source: §7.3.3 · p. 68, §7.3.4 · p. 69

7.4 Merge SFT

• SFT and alignment can be combined by reformatting instruction data as preference data and training jointly, or by training separately and merging model weights.UFT assigns maximal reward to high-quality instruction-tuning pairs and mixes them with alignment data; PAFT trains SFT and DPO in parallel, then merges adapters, using ℓ1 regularization to promote SFT-adapter sparsity.Source: §7.4.1 · pp. 69–70, §7.4.2 · p. 70

8.1 Foundations: Gradient Coefficient Theory and Convergence

• Convergence guarantees for GRPO remain open given its shifting group baseline, large action space, and response-level rewards; the optimal combination of five gradient-coefficient axes is also unknown.The survey also asks when to update the reference model and how fixed versus moving references affect KL-penalty stability and final-policy quality.Source: §8.1 · p. 71

8.2 Prompt: Curriculum Design and Self-Play

• Curriculum research should test joint domain-and-difficulty scheduling, while self-play must address proposer bias toward tasks the solver already handles.For tasks without automatic checking, the survey asks how to validate self-generated prompts without human annotation.Source: §8.2 · p. 71

8.3 Response: Sampling, Diversity, and Length

• Self-play responses lack an external correctness oracle, and it remains unclear whether learned strategies transfer to held-out benchmarks or drift from real task requirements.The survey also asks how to preserve response diversity while reusing old-policy rollouts only as long as their importance-sampling ratios remain reliable.Source: §8.3 · pp. 71–72
• The survey asks whether models can predict appropriate response length from prompt difficulty and optimize concision alongside correctness without distorting advantages.This response-design cluster also concerns rollout generation, response diversity, and reasoning-length control.Source: §8.3 · pp. 71–72

8.4 Feedback: Supervision Quality and Evaluation

• Feedback can reward verbosity or formatting over content, while static benchmarks may improve even as real-world utility falls or new failure modes go undetected.The survey asks how to allocate human and AI feedback, detect stale judge reliability, and combine benchmark and adversarial testing across failure dimensions.Source: §8.4 · p. 72

8.5 Reward: Models, Granularity, and Safety

• RLVR remains concentrated in automatically checkable domains such as math and code, and the optimal reward granularity—from whole-response to token-level—under fixed compute is unresolved.The survey asks how to construct verifiable rewards for factual, scientific, and long-form tasks, and whether finer rewards improve final performance or only convergence speed.Source: §8.5 · pp. 72–73

8.6 Algorithm: Optimization and Pipeline Integration

• Open algorithm questions include when iterative offline DPO converges to online RLVR, how advanced objectives behave at scale, and whether joint SFT-RL losses can limit forgetting while retaining exploration.The comparison of Nash and listwise methods with PPO and DPO includes reward hacking, verbosity bias, and diversity collapse.Source: §8.6 · p. 73

8.7 Extensions: Frontiers Beyond Current Scope

• Future extensions ask how RLVR handles multi-turn tool use and sparse rewards, whether reasoning gains transfer across languages, and which network components produce RLVR self-verification behaviors.The proposed interpretability question is whether identifying those components could allow the rest of the network to be frozen to reduce training cost.Source: §8.7 · p. 73

9 Conclusion

• The survey places pretraining, SFT, actor-critic RLHF, RLVR, and offline and iterative DPO within a single gradient-coefficient framework.Its RLVR coverage includes prompt generation and selection, response generation and selection, and five gradient-coefficient sub-axes.Source: §9 · p. 73
• The framework organizes prompt sampling, response sampling, and gradient-coefficient design, while extending to on-policy RLHF and iterative DPO, including RLAIF and Nash learning.Offline DPO is organized by response generation, reward modeling, regularization, and SFT integration.Source: §9 · pp. 73–74
• Seven open-problem clusters cover foundations, prompts, responses, feedback, rewards, algorithms, and extensions.The framework decomposes methods into prompt sampling, response sampling, and gradient-coefficient design.Source: §9 · pp. 73–74

A RLVR - detailed characterization of all papers

• RLVR prompt-source tables cover training-data sourcing, curriculum design, and prompt selection; response-design tables cover generation and selection strategies.The listed scopes include seven prompt-source papers and 15 response-design papers.Source: Table 2 · p. 92, Table 3 · p. 93, Table 7 · p. 97
• Table 2 lists 7 prompt-source papers and Table 7 lists 6; Tables 3 and 8 each list 15 response-design papers, while Tables 6 and 11 each list 13 gradient-coefficient-part-3 papers.Source: Table 2 · p. 92, Table 3 · p. 93, Table 6 · p. 96
• Gradient-coefficient tables group RLVR methods by importance ratios and advantage computation, reward design and baselines, and normalization, length scaling, and regularization.The listed groups include 14 papers in the first two parts and 13 in the third part.Source: Table 4 · p. 94, Table 5 · p. 95, Table 6 · p. 96

B RLHF - detailed characterization of all papers

• The methodology summary table covers papers 1–20, from the DPO baseline through Nash and self-play variants.Source: Table 12 · p. 103
• The listed method entries include pairwise, list, and single-response formats across preference-based approaches.Examples include f-DPO and SimPO as pairwise entries, RRHF and PRO as list entries, and NPO as a single-response entry.Source: B RLHF - detailed characterization of… · p. 104
• The experimental summary table covers papers 21–35, from token-level reward methods through SFT-merge variants.Source: Table 15 · p. 106