PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Post-Training: A Deep Dive into Reasoning Large Language Models

Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip H. S. Torr, Fahad Shahbaz Khan, Salman Khan

arXiv:2502.21321v2cs.CLcs.CV

TL;DR

LLMs can produce hallucinations, lose logical consistency, and fail to align with user expectations or ethical standards, motivating specialized post-training strategies. This survey systematically reviews fine-tuning, reinforcement learning, and scaling methods, finding that recent reinforcement learning and test-time scaling have improved reasoning capabilities while identifying open challenges for future work.

  • Problem

    LLMs remain vulnerable to hallucinations, inconsistent extended discourse, and misalignment with user expectations or ethical standards.

  • Method

    The survey systematically reviews fine-tuning, reinforcement learning, and scaling, including reinforcement-learning approaches for alignment and reasoning.

  • Results

    Recent advancements in reinforcement learning and test-time scaling have significantly improved LLM reasoning capabilities for increasingly complex tasks.

  • Takeaways & Limitations

    The survey consolidates current research and identifies open challenges to guide future efforts in optimizing LLMs for real-world applications.

  • Takeaways & Limitations

    Post-training remains constrained by sensitive or proprietary fine-tuning data that can create privacy risks through memorization and regurgitation.

Abstract

from arXiv · show

Large Language Models (LLMs) have transformed the natural language processing landscape and brought to life diverse applications. Pretraining on vast web-scale data has laid the foundation for these models, yet the research community is now increasingly shifting focus toward post-training techniques to achieve further breakthroughs. While pretraining provides a broad linguistic foundation, post-training methods enable LLMs to refine their knowledge, improve reasoning, enhance factual accuracy, and align more effectively with user intents and ethical considerations. Fine-tuning, reinforcement learning, and test-time scaling have emerged as critical strategies for optimizing LLMs performance, ensuring robustness, and improving adaptability across various real-world tasks. This survey provides a systematic exploration of post-training methodologies, analyzing their role in refining LLMs beyond pretraining, addressing key challenges such as catastrophic forgetting, reward hacking, and inference-time trade-offs. We highlight emerging directions in model alignment, scalable adaptation, and inference-time reasoning, and outline future research directions. We also provide a public repository to continually track developments in this fast-evolving field: https://github.com/mbzuai-oryx/Awesome-LLM-Post-training.

1 Introduction

LLM post-training refines pretrained models for reasoning, factuality, adaptability, and alignment, addressing limitations that next-token training alone does not resolve. The survey organizes this landscape around fine-tuning, reinforcement learning, and test-time scaling, while identifying benchmarks and open challenges.

  • Motivation: LLMs support broad applications but can hallucinate, lose logical consistency over long discourse, and produce statistically coherent rather than explicitly logical reasoning.These limitations motivate specialized post-training strategies for reliability, bias, and context sensitivity.
  • Post-training scope: Post-training comprises fine-tuning and alignment stages that refine behavior, mitigate biases or inaccuracies, and better align outputs with human intent.
  • Fine-tuning: Fine-tuning adapts LLMs to specific tasks but risks overfitting, high compute costs, and reduced generalization.
  • Reinforcement learning: Reinforcement learning improves adaptability through dynamic feedback and sequential decision-making, but LLMs face vast action spaces, sparse or delayed subjective rewards, and conflicting objectives.
  • Test-time scaling: Test-time scaling adjusts computation during inference without changing the core architecture, improving generalization while introducing computational challenges.Chain-of-thought and Tree-of-Thought frameworks support multi-step reasoning at inference time.
  • Survey contributions: The survey contributes a systematic taxonomy, practical benchmarks and evaluation metrics, and discussion of open challenges across fine-tuning, reinforcement learning, and scaling.

2 Background

The background frames autoregressive language generation as sequential decision-making: supervised likelihood training predicts tokens from static data, whereas reinforcement learning optimizes expected cumulative reward over reasoning trajectories. Early policy-gradient and actor-critic methods provide foundations for modern reasoning and alignment techniques.

  • Likelihood-based training: Maximum Likelihood Estimation trains LLMs by maximizing the probability of target tokens conditioned on the input and preceding output tokens.X denotes the input, while Y is the target sequence.
  • Limits of autoregressive training: Token-wise training can preserve fluency but may produce cascading inference errors, while long-sequence generation also challenges coherence and contextual relevance.
  • LLM reasoning as RL: In the MDP formulation, the generated-token sequence is the state, the next token is the action, and the reward evaluates output quality.
  • RL objective: RL optimizes expected return through dynamic interactions, balancing exploration and exploitation to improve reasoning, coherence, and alignment.The discount factor γ controls how strongly future rewards influence current decisions.
  • Early RL methods: REINFORCE updates policy parameters using received rewards, while actor-critic methods add value estimation to stabilize learning and improve sample efficiency.A baseline reduces variance in policy-gradient updates.
  • Connection with modern methods: Modern RLHF with PPO and methods such as GRPO build on early policy-gradient, sequence-training, and actor-critic foundations to address exposure bias and high variance.

3 Reinforced LLMs

Reinforced LLM training is presented as a staged process that begins with supervised behavior shaping, learns a reward model from preferences, and then optimizes the policy. The survey also situates DPO and GRPO as scalable alternatives to constrained policy-optimization methods.

  • Training pipeline: RL integration typically begins with supervised fine-tuning on high-quality human examples to establish baseline compliance with format and style guidelines.
  • Training pipeline: A reward model is trained from preference labels on outputs generated by the fine-tuned model, then guides reinforcement-learning fine-tuning of the target model.
  • Scalable alignment: DPO and GRPO reformulate preference-based alignment as scalable alternatives to policy-optimization approaches using constrained updates and KL-divergence regularization.

3.1 Reward modeling

Reward modeling converts human or other preference signals into training rewards for LLM policy optimization. The section distinguishes preference-data construction, ranking-based reward learning, explicit and implicit rewards, and outcome/process supervision.

  • Preference data: Reward-model datasets pair queries with candidate responses and collect pairwise or ranked human preferences.Candidates may be sampled from a base model, generated by beam search, or produced under different prompting conditions.
  • Preference modeling: Bradley–Terry and Plackett–Luce losses learn scalar reward functions from pairwise preferences or full and partial rankings.The learned reward model maps a query-response pair to a scalar score reflecting preferences.
  • Reward types: Explicit reward modeling uses predefined rules, heuristics, human annotations, or specialized AI modules to produce direct numeric signals.It can yield precise estimates but may be costly or time-consuming at scale.
  • Reward types: Implicit reward modeling infers latent rewards from interactions such as upvotes, acceptance rates, click-through patterns, or engagement time, but may encourage heuristic exploitation.Large datasets can be accumulated with minimal overhead, while engagement optimization can displace content quality.
  • Supervision granularity: Outcome rewards evaluate final-answer success, whereas process rewards evaluate intermediate reasoning steps and can be combined with outcome rewards.Outcome rewards offer limited insight into how conclusions were reached and create credit-assignment problems for long responses; process rewards require more complex annotations.
  • Reward-model refinement: Policy Reward Modeling with last-step aggregation outperforms Outcome Reward Modeling by using final-step evaluations for policy updates.Adaptive Reward Models instead iteratively update reward and policy models using current behavior and human feedback to address reward hacking and drift.

3.2 Policy Optimization

Policy optimization adjusts an LLM’s generation policy using reward or preference signals. The described approaches include reward-model-based RL and direct preference optimization, with different mechanisms for credit assignment, regularization, and reward integration.

  • Reward-based policy optimization: A trained reward model Rθ(x, y) replaces or augments the environment reward so policy πϕ produces responses preferred by humans.The objective maximizes expected reward across queries and responses sampled from the policy.
  • Reward-based policy optimization: In language-model RL, states are partial generations and actions are next tokens or text chunks, creating vast action spaces and delayed credit assignment.The advantage function compares each generation step with baseline performance.
  • Direct preference optimization: ORPO directly optimizes a policy from pairwise preferences, increasing the preferred response probability while decreasing the dispreferred response probability.It avoids first training a separate reward model and then running standard RL.
  • Direct preference optimization: Pairwise preference objectives use logistic comparisons and negative log-likelihood losses to align policy probabilities with human-labeled preferences.The preferred response is represented as yj and the less-preferred response as yk for query x.
  • Direct preference optimization: ORPO may be less flexible when multiple reward signals must be combined.This limitation follows from its direct pairwise-preference optimization formulation.

3.2.2 Proximal Policy Optimization (PPO) in LLMs

PPO adapts clipped policy optimization and KL regularization to LLM alignment, balancing reward improvement against stable, bounded policy updates. RLHF combines supervised fine-tuning, reward modeling from human rankings, and PPO, while RLAIF substitutes AI-generated feedback for human annotation.

  • PPO mechanism: PPO updates an LLM policy with a clipped objective based on probability ratios, advantage estimates, and an allowable deviation hyperparameter ϵ.Advantages are computed with Generalized Advantage Estimation using rewards and a learned value function.
  • PPO mechanism: Clipping restricts policy-distribution changes, moderating updates to avert catastrophic shifts in language generation and preserve training stability.The constraint is particularly relevant to LLMs’ complex, high-dimensional action spaces.
  • KL regularization: PPO can maximize reward while keeping the policy close to the base model through a KL-divergence penalty weighted by β.The KL term limits over-optimization of proxy rewards and helps retain linguistic coherence while avoiding degenerate outputs.
  • Alignment pipelines: RLHF performs SFT, trains a reward function from human-ranked responses, and applies PPO for reinforcement-learning fine-tuning.The reward model predicts preferences and supplies a scalar training signal.
  • Alignment pipelines: RLAIF replaces human preference annotation with labels generated by a capable secondary language model, reducing data-collection cost and time.Those labels train a reward model that guides reinforcement-learning fine-tuning of the target model.
  • Trust-region alternatives: TRPO directly enforces a KL-based trust-region constraint, but its second-order optimization is computationally expensive for large-scale LLMs.PPO is preferred in practice for simplicity, implementation ease, and comparable large-scale performance.

3.2.6 Direct Preference Optimization (DPO)

DPO directly fits policy probabilities to preference pairs instead of using the traditional RL loop. The section contrasts its trajectory-level treatment with stepwise methods such as OREO and GRPO, which support finer credit assignment or lower-memory optimization.

  • DPO formulation: DPO integrates human preference signals directly into the training objective, increasing the probability of preferred responses without a separate reward model or policy-gradient loop.It adjusts preferred versus dispreferred response log-likelihoods and avoids advantage functions and explicit clipping.
  • DPO formulation: DPO uses a learnable policy πθ, reference policy πref, sigmoid σ, scaling parameter β, and triplets containing x, y+, and y−.The dataset records a preferred output y+ over a dispreferred output y− for each query x.
  • Credit assignment: DPO loses the per-step granularity of advantage-based methods because it treats entire trajectories uniformly.The advantage function is described as quantifying per-step contributions and identifying key reasoning errors.
  • Alternative offline RL: OREO jointly trains a policy and value function with sparse final-outcome rewards, optimizing a soft Bellman equation for fine-grained credit assignment.Its explicit value function supports test-time beam search and iterative refinement from failed trajectories.
  • Alternative offline RL: OREO’s cost scales with trajectory length and value-model training, while broader-domain generalization and iterative-training data curation require validation.The passage specifically identifies coding generalization and overfitting to failure modes as open concerns.
  • GRPO comparison: GRPO removes the separate value function by estimating baselines from groups of sampled outputs for the same question.This reduces memory usage and supports outcome or process supervision through normalized group-level advantages.
  • GRPO comparison: Fine-grained per-step rewards enable models to identify and reinforce high-quality responses in complex, multi-step reasoning tasks.GRPO’s process-supervision mode assigns rewards at reasoning steps, whereas outcome supervision rewards only the final output.

3.2.9 Multi-Sample Comparison Optimization

Multi-sample comparison optimization evaluates several responses jointly rather than relying only on isolated pairwise comparisons, promoting diversity and mitigating bias.

  • The method compares multiple responses simultaneously for a query instead of using only single-pair comparisons.This frames response evaluation as a broader ranking problem.
  • The ranking framework is designed to promote diversity and reduce comparison bias.
  • Jointly evaluating each response against all others captures more nuanced preferences than isolated pairwise events.

3.3 Pure RL Based LLM Refinement

Pure-RL refinement and its extensions use staged training, rejection sampling, reward design, and distillation to improve reasoning quality, alignment, and deployment efficiency. The described pipeline ranges from DeepSeek-R1-Zero’s RL-only approach to DeepSeek-R1’s multi-stage process.

  • DeepSeek-R1-Zero uses a purely reinforcement-learning approach without supervised fine-tuning.
  • DeepSeek-R1 combines cold-start data, RL training, supervised fine-tuning, distillation, and targeted handling of language mixing and readability.The pipeline is presented as supporting robustness and alignment with human preferences.
  • Providing chain-of-thought traces before RL training establishes a stronger foundation for reasoning tasks and improves output robustness and interpretability.
  • Rejection sampling filters generated responses for correctness and clarity before blending them with additional data for supervised fine-tuning.
  • Group Relative Policy Optimization samples output groups and computes rewards and advantages using checks such as mathematical correctness, code correctness, structured CoT tags, and language-mixing penalties.
  • Distillation transfers refined reasoning capabilities from a large model to smaller architectures, reducing deployment overhead while retaining competitive benchmark performance.

4 Supervised Finetuning in LLMs

Supervised fine-tuning adapts LLMs through instruction, dialogue, reasoning, domain-specific, preference, and distillation-based training, while PEFT reduces adaptation costs. These methods span task behavior, multi-turn coherence, reasoning transparency, specialized knowledge, alignment, and efficient deployment.

  • Instruction Finetuning: Instruction fine-tuning trains models on curated instruction-response pairs to follow user requests accurately and helpfully across task domains.Datasets commonly include summarization, question answering, classification, and creative writing.
  • Dialogue-style Finetuning: Dialogue-style fine-tuning uses multi-turn transcripts to teach models to preserve context and produce coherent replies across conversations.
  • CoT Reasoning Finetuning: Chain-of-thought fine-tuning exposes step-by-step reasoning traces, improving interpretability and accuracy on complex tasks.It typically uses supervised reasoning annotations showing how solutions unfold.
  • Domain-specific Finetuning: Domain-specific fine-tuning uses curated domain text and labeled examples for areas such as biomedicine, finance, climate, and code understanding.
  • Knowledge Distillation: Knowledge distillation trains smaller student models on teacher-generated labels or rationales, producing lighter models that retain much of the teacher’s performance.The passage notes effectiveness in zero-shot and few-shot tasks.
  • Efficient Finetuning: Parameter-efficient fine-tuning freezes most model weights while training a small set of parameters or learnable prompts, reducing memory and computation demands.LoRA, Prefix Tuning, and Adapters are presented as examples.
  • Efficient Finetuning: Combining LoRA and QLoRA with system and data optimizations enables cost-effective domain-specific adaptation without expensive full fine-tuning.

5 Test-time Scaling Methods

Test-time scaling improves LLM reasoning by allocating inference computation across structured search, sampling, refinement, and branching strategies. These methods can improve accuracy or efficiency, but their benefits depend on task difficulty, compute budget, and search design.

  • 5.1 Beam Search: Beam search retains the highest-probability partial sequences at each decoding step, enabling parallel exploration while pruning less promising reasoning paths.In Tree-of-Thoughts, beam search maintains the b most promising states at each reasoning step.
  • 5.2 Best-of-N Search: Best-of-N samples complete outputs independently and selects the highest-rated candidate, offering diversity without retraining but increasing computational cost.Selection can use a reward model, checker, or model likelihood; imperfect proxy rewards can cause reward hacking.
  • 5.2 Best-of-N Search: Beam search outperforms best-of-N sampling at low compute budgets for harder questions, whereas best-of-N scales better for easier tasks.The choice between methods depends on question difficulty and available compute.
  • 5.3 Compute-Optimal Scaling: Compute-Optimal Scaling Strategy assigns easier prompts to sequential refinement and harder prompts to parallel sampling or beam search.COS achieves four times lower compute usage than traditional best-of-N sampling while maintaining equivalent performance.
  • 5.6 Tree-of-Thoughts: Additional inference computation can outperform larger models on challenging reasoning tasks, while Tree-of-Thoughts improves complex puzzles, planning, and games through systematic solution-space search.These gains come with higher computational demands, and Graph-of-Thoughts is described as more efficient and adaptable than Tree-of-Thoughts for complex reasoning.
  • 5.11 Monte Carlo Tree Search: Monte Carlo Tree Search uses simulated rollouts to explore reasoning paths and select high-reward answers, outperforming naive sampling in scientific question answering.MCTS also supports code generation and multi-model exploration, but is slower than straightforward sampling or beam search.
  • 5.13 Pretraining vs. Test-Time Scaling: Test-time scaling can match a model 14× larger on easy-to-intermediate tasks while using four times fewer FLOPs in compute-intensive scenarios.Pretraining remains superior for the hardest tasks or when inference compute is constrained.

6 Benchmarks for LLM Post-training Evaluation

The survey organizes post-training evaluation around diverse benchmark groups spanning reasoning, alignment, multilinguality, comprehension, dialogue, and search. Table 3 provides an overview of reasoning, RL alignment, and multilingual datasets, distinguishing pointwise and pairwise evaluation.

  • Post-training benchmarks cover reasoning, alignment, multilinguality, general comprehension, dialogue, and search tasks.
  • Table 3 overviews datasets for reasoning, RL alignment, and multilingual evaluation.
  • Pointwise and pairwise denote different methods for evaluating model performance across tasks.
  • A structured evaluation framework assesses LLM strengths and limitations across multiple task domains.

7 Future Directions

Future work centers on improving efficiency, reliability, personalization, privacy, reward design, and evaluation for increasingly capable post-training systems. The survey highlights both empirical trends and concrete directions for addressing computational costs, alignment risks, and reasoning inefficiencies.

  • RL applications for refining LLMs have increased in prominence since 2020, alongside interest in human-in-the-loop reinforcement and scalability.
  • Reward modeling interest has risen with self-rewarding language models, while reward hacking remains an unresolved challenge.
  • Safe Adaptation: Privacy-preserving adaptation, differential privacy, federated learning, and parameter-efficient methods remain important responses to data leakage and catastrophic forgetting risks.
  • Test-Time Scaling: Test-time scaling must determine how much computation each query requires while avoiding both mistakes from under-allocation and inefficiency from over-allocation.
  • Reasoning Efficiency: 4× efficiency gains over baseline methods with 94% solution accuracy on MATH result from combining PRM-guided tree search with online distillation.
  • Reasoning Efficiency: 22% of computation is wasted in reasoning chains exceeding optimal reasoning length, motivating length penalties and compression-based approaches.

8 Conclusion

The survey systematically reviews fine-tuning, reinforcement learning, and scaling methods for LLM post-training. It synthesizes advances in reasoning, planning, multitask generalization, efficiency, and alignment while identifying open challenges for real-world deployment.

  • The survey provides a systematic review of post-training methodologies focused on fine-tuning, reinforcement learning, and scaling.
  • It analyzes how reinforcement learning and test-time scaling support reasoning, planning, and multitask generalization.
  • The survey consolidates recent advances and identifies open challenges for optimizing LLMs for real-world applications.
Loading 2502.21321v2…