Reinforcement Learning from Human Feedback

Nathan Lambert

Source-linked AI summary

Reinforcement Learning from Human Feedback

Nathan Lambert

arXiv:2504.12501v11Apr 16, 2025cs.LG

TL;DR

Post-training must reshape next-token predictors for conversation while eliciting latent knowledge from pretraining. The book organizes this challenge around RLHF stages and reports that RL generalizes out of distribution where SFT can fail dramatically.

Problem
Post-training must reshape models from next-token prediction to conversation question-answering while extracting latent knowledge from pretraining.
Method
The book structures post-training around preference data, reward optimization, rejection sampling, reinforcement learning, and distillation.
Results
Across task variants, RL consistently improves OOD performance as compute scales, while SFT consistently degrades OOD performance despite improving in-distribution.
Takeaways & Limitations
The reported comparison indicates that reinforcement learning can preserve and improve spatial reasoning under distribution shifts, unlike the tested SFT behavior.

Abstract

from arXiv · show

Reinforcement learning from human feedback (RLHF) has become a crucial tool to build the latest machine learning systems at scale. The field grew around the core methods of RLHF into today's broader suite of post-training techniques. In this book, we give a comprehensive introduction to the core methods for post-training models for people with some level of quantitative background, organized around the canonical RLHF recipe. The book starts with what RLHF does and why it was created, with seminal technical milestones in its young history and a primer on reinforcement learning context needed to understand the book. The core of the book details every optimization stage in using RLHF, from starting with instruction tuning to training a reward model and finally all of rejection sampling, reinforcement learning, on-policy distillation, and direct alignment algorithms. The book also discusses broader topics, such as the origins of RLHF -- both in recent literature and in a convergence of disparate fields of science in economics, philosophy, and optimal control. The book concludes with advanced topics -- understudied or emerging research questions in synthetic data, tool-use, character training, and evaluation -- and open questions for the field. The book is released with a variety of companion resources, including a codebase, a library to compare model completions from within post-training stages, and an educational course, to be a one-stop shop for learning all foundational concepts for post-training language models.

9 Rejection Sampling

This book presents RLHF as a foundation for understanding post-training language models, from its canonical recipe to newer methods, applications, and open questions. It emphasizes both practical implementation and the broader intuition that post-training extracts and cultivates capabilities from pretrained models.

Origins and Scope: RLHF began as a method for using basic human preference signals to guide optimization on hard-to-specify problems.Its early applications included control, summarization, instruction following, web-information parsing, and alignment.
Core Recipe: The canonical RLHF pipeline trains an instruction-following model, learns a reward model from human preferences, and optimizes generations against that reward.The book explains key decisions and implementation examples for each stage.
Performance and Elicitation: Post-training can substantially improve a static base model: OLMoE’s evaluation average rose from 35 to 48 when the second version changed post-training without changing most pretraining.The broader technique set includes mid-training, instruction tuning, RLVR, and preference-tuning.
Performance and Elicitation: The book frames post-training as reshaping next-token prediction into conversational question-answering while extracting latent knowledge and intelligence from pretraining.This perspective is presented as the Elicitation Theory of Post-training.
Emerging Directions: Recent innovation centers on RLVR, reasoning training, and related methods that build on RLHF infrastructure and ideas while evolving faster than the stable RLHF literature.The book aims to capture that stable literature after RLHF’s initial period of rapid change.

2 A Tiny History of RLHF

RLHF developed from early reinforcement learning methods using human preferences into a central technique for training large language models. Its history includes reward modeling, trajectory comparisons, safety and alignment applications, and expansion into broader preference fine-tuning.

Early origins: RLHF emerged from earlier work on learning policies from human evaluations, including TAMER, COACH, and trajectory-preference learning.TAMER learned reward models from iterative action scores, while COACH used positive and negative feedback to tune advantage functions.
Early origins: Trajectory comparisons enabled agents to optimize predicted human preferences in Atari domains, with human choices sometimes outperforming direct environment interaction.The core loop trained a reward predictor asynchronously from trajectory-segment comparisons while the agent maximized predicted reward.
Reward modeling: Reward models expanded from tools for solving reinforcement learning problems into methods for studying alignment and modeling human preferences.This transition broadened reward modeling beyond approximating fixed environment rewards.
Language-model adoption: Between GPT-2 and GPT-3, RLHF was adopted for language models, formalizing canonical concepts such as reward models, KL distances, and feedback diagrams.Early applications included summarization, instruction following, browser-assisted question answering, citation-supported answers, and dialogue.
Language-model adoption: ChatGPT’s training announcement helped establish RLHF as a widely recognized component of large-language-model development.The same broad approach was subsequently used in systems including Claude, Llama, Nemotron, and Tülu 3.
Broader preference fine-tuning: RLHF is now broadening into preference fine-tuning that includes process rewards, direct alignment algorithms, execution feedback, and online reasoning methods.The field remains rapidly evolving, and this chapter is intentionally a starting point rather than a comprehensive review.

3 Training Overview

RLHF adapts the standard reinforcement-learning loop to language-model prompts and completions, using learned preference rewards and regularization. Its training process links instruction tuning, reward modeling, and optimization, while retaining practical advantages despite higher infrastructure demands.

Training overview: RLHF combines multiple models and stages in an online optimization centered on a proxy reward for human preferences.The canonical objective adds a distance-based regularizer to the proxy reward.
Problem formulation: Standard RL maximizes expected discounted reward through policies, environment transitions, and finite-horizon or continuing-task objectives.The policy maps states to action distributions, while γ balances near-term and future rewards.
Problem formulation: The standard RL loop repeatedly observes a state, chooses an action, receives a reward, and updates the policy through trial and error.The thermostat example illustrates this process using temperature as state, heater control as action, and proximity to a target as reward.
Problem formulation: CartPole instantiates the same setup with four continuous state variables, left-or-right force actions, physics-based transitions, and episode rewards for maintaining balance.The state update advances position and velocity variables using Euler integration.
Manipulating the standard RL setup: RLHF replaces environmental rewards with learned reward models, uses prompts as initial states and completions as actions, and assigns rewards at the response level without discounting.A prompt-completion pair forms a complete rollout, making the language-model setting distinct from traditional sequential RL.
Advantages of RL in post-training language models: RL stages can improve rough edges and selectively target prompt distributions while preserving broad capabilities, as illustrated by math-focused Tülu 3 training.The text also characterizes RL losses as robust, scalable, effective, and flexible.

4 Instruction Fine-Tuning

Instruction fine-tuning adapts pretrained language models to instruction-response behavior and provides the foundation for later preference optimization. Consistent chat templates serialize roles and messages into token sequences that guide training and generation.

4 Instruction Fine-Tuning: Instruction fine-tuning emerged from unified text-to-text task framing, prompting, and evidence that explicit instruction-response training improves cross-task reliability.This convergence established training general models on large collections of instructions.
4 Instruction Fine-Tuning: IFT is supervised learning that adapts language models to a desired task distribution and prepares them for the question-answer format used by RLHF.It is now standard practice across many language-model pipelines.
4.1 Chat Templates and the Structure of Instructions: Chat templates convert user queries and conversation roles into a tokenizer-readable sequence for post-training and generation.All post-training stages rely on this interaction structure.
4.1 Chat Templates and the Structure of Instructions: A template may serialize system, user, and assistant messages with beginning, ending, and padding tokens, while optionally appending an assistant generation prompt.The assistant tag cues the model to continue generating its response.
4.1 Chat Templates and the Structure of Instructions: Templates enforce role alternation and can treat an initial system message as a special first turn through an offset.The resulting sequence is a flat token stream that the language model predicts from.
4.1 Chat Templates and the Structure of Instructions: System messages provide initial behavioral or contextual instructions, while user and assistant roles represent the human input and model response.Special tokens separate these messages before they are passed to the model.
4.1 Chat Templates and the Structure of Instructions: Generation begins after an assistant-start marker and ends when the model emits its end-of-sequence token or reaches the context limit.The example sequence demonstrates this structure for a system message followed by a user query.
4.1 Chat Templates and the Structure of Instructions: Open tooling commonly stores chat templates as Jinja snippets in tokenizer configurations, with multiple formats including ChatML-derived and Zephyr templates.The same structure extends to multiple conversational turns.

5 Reward Modeling

Reward models translate human preference comparisons into scalar proxy rewards for RLHF optimization. The canonical Bradley-Terry approach scores chosen and rejected completions, trains on their score difference, and is commonly implemented as a language model with a scalar head.

Reward model role: Reward models learn complex, hard-to-specify human preferences and provide scalar proxy objectives for downstream RLHF optimization.They replace fixed environment rewards with learned signals that can guide language-model training.
Bradley-Terry preference modeling: The common Bradley-Terry reward model predicts preference probability from the relative scores of prompt-conditioned completions.The chapter also distinguishes Bradley-Terry models from outcome and process reward models.
Bradley-Terry preference modeling: The Bradley-Terry preference probability is a sigmoid of the chosen-minus-rejected reward difference, and adding a constant to all scores leaves preferences unchanged.This makes the model identifiable only up to a shared score offset.
Reward-model loss: Reward-model training minimizes the expected negative log-likelihood over preference pairs, equivalently expressed through log-sigmoid or softplus forms.The logarithm is taken before averaging because expected probability and expected log-probability are different objectives.
Bradley-Terry preference modeling: The model scores chosen and rejected completions, often from a sequence-level representation such as the EOS hidden state, and uses only their score difference in the contrastive loss.The chosen completion is denoted yc and the rejected completion yr.
Implementation: A typical implementation adds a small linear head to a causal language model to output one scalar reward for each prompt-completion pair.The main implementation challenge lies in the separate data-loading and inference pipeline for tokenized chosen and rejected inputs.
Implementation: Llama 3 removed the margin term after observing diminishing improvements with scaling.This is an author-reported design change rather than a general requirement of Bradley-Terry training.
Reward-model evaluation: Generative reward models and judge models remain behind existing reward models on reward-model evaluations, reinforcing the importance of reward modeling in current RLHF.The passage frames this comparison specifically around RM evaluations.

6 Reinforcement Learning

RLHF updates a language-model policy from reward-model feedback on its generated completions, while a frozen reference model supplies a KL penalty. The chapter develops policy-gradient methods, their objectives, and practical trade-offs for this process.

Training loop: The policy generates completions, the reward model scores them, and reinforcement learning uses those scores for gradient updates.The initial policy copy remains frozen as a reference model for the KL penalty.
Policy-gradient algorithms: Policy-gradient methods such as PPO, GRPO, and REINFORCE update the model using recently generated samples rather than replay-buffer scores.REINFORCE-style methods can be simpler than PPO because they do not require a separate value model or generalized advantage estimation.
Policy-gradient objective: The policy-gradient objective estimates updates by sampling prompts and completions from the current policy, then weighting log-probability gradients by scalar rewards or advantages.RLHF commonly sets γ = 1 because the completion is treated as the optimization unit rather than individual tokens.
Policy-gradient objective: Positive advantages increase an action’s likelihood, whereas negative advantages decrease it.The update combines the direction indicated by the log-policy gradient with a scalar assessment of outcome quality.
Algorithmic trade-offs: GRPO-related methods trade off variance, bias, and numerical stability through group-relative estimates, sequence-level ratios, or clipped importance weights.CISPO clips importance weights while retaining a gradient signal for every token, whereas GSPO computes ratios at the sequence level.
Implementation trade-offs: Sequence-length normalization can change gradient allocation: masked_mean gives short sequences larger per-token gradients, while two alternatives equalize them.The reported example gives per-token gradients of 0.25 for short sequences and 0.14 for long sequences under masked_mean; gradient accumulation can change the balance.

7 Reasoning and Inference-Time Scaling

Reasoning-focused post-training uses reinforcement learning, especially with verifiable rewards, to improve capabilities through repeated problem solving and increased inference-time computation. The emerging recipe has helped make RL more stable and central to frontier-model training.

The role of inference-time scaling: Inference-time scaling improves performance by allocating more computation during generation, such as longer reasoning chains or multiple sampled responses.Reasoning models are trained to think extensively before answering and exploit this generation-time computation.
Reasoning models: RL training for reasoning models combines post-training techniques with verifiable-domain reinforcement learning to increase reasoning, coding, and mathematics problem-solving capabilities.The described reasoning-model training is presented as a combination of preference alignment and RL on verifiable domains.
Reinforcement Learning with Verifiable Rewards: RLVR replaces or makes optional the learned reward model by using a verification function that returns a positive reward for correctness and 0 otherwise.Verification can use answer extraction for mathematics or unit tests for code, including tasks with multiple correct solutions.
The role of RLVR: Repeatedly revisiting the same questions can improve training performance and generalize to some unseen questions and domains.The passage conditions this result on careful data distribution and stable training infrastructure.
Why RL works now: Stability barriers have fallen substantially: many model releases use verifiable-reward RL, and technical barriers to RL are described as being at an all-time low.The passage identifies instability and brittle training, including loss spikes and crashes, as earlier adoption limits.
Looking ahead: RL has shifted from the cake metaphor’s finishing touch to a load-bearing component of frontier model training.The text presents current RLVR techniques as the field’s best understanding for eliciting reasoning, while noting that future methods may differ.

8 Direct-Alignment Algorithms

Direct Alignment Algorithms optimize the RLHF objective directly from preference data, avoiding an intermediate reward model and online reinforcement-learning optimization. They are simpler and cheaper to implement, although reports find policy-gradient methods slightly outperform DPO and variants.

DAAs update models toward the RLHF objective using preference data without training an intermediate reward model or using reinforcement-learning optimizers.This reduces implementation complexity and training compute.
DPO: DPO uses gradient ascent to directly optimize a policy for the constrained RLHF objective.Its loss compares chosen and rejected completion probability shifts relative to a reference model.
DPO: DPO reparameterizes preference learning through an implicit reward model, bypassing explicit reward-model training and completion sampling for score estimation.The corresponding optimal policy can be extracted in closed form.
DPO: DPO increases the relative log-probability gap between chosen and rejected responses, weighted by β to balance reward optimization against KL divergence from the reference model.The loss decreases when the chosen response shifts more than the rejected response.
DAAs vs. RL: Policy-gradient and reinforcement-learning methods have been reported to outperform DPO and its variants, but DAAs remain widely used because they support rapid, simple iteration.The reported performance gap is small, while DPO offers a controlled training environment and lower computational demands.

9 Rejection Sampling

Rejection sampling generates multiple completions, scores them with a reward model, selects high-reward examples, and instruction-fine-tunes the current model on those examples. It is widely used and flexible, but its prompt, reward-model, and sequencing choices remain poorly documented.

Training Process: The procedure can be applied after instruction fine-tuning, reinforcement-learning optimization, or RLVR, making it versatile but difficult to place canonically.Many prominent RLHF pipelines use it as a core component.
Training Process: Rejection sampling curates candidate completions with a reward model, then fine-tunes the original model only on the top completions.The final optimization uses the standard instruction-fine-tuning loss.
Open Choices: The chapter states that prompt choice, reward-model choice, and rejection-sampling sequencing are not well documented, leaving further experimentation necessary.The overview explains the methods without establishing a canonical implementation.
Training Process: For each prompt, rejection sampling generates N completions, scores every prompt-completion pair with a reward model, and stores the results in a reward matrix.Rows correspond to prompts and columns to sampled completions.
Selection: Selection can choose the maximum-reward completion per prompt or the K highest-reward prompt-completion pairs globally.Global selection flattens the reward matrix and maps selected indices back to prompt and completion coordinates.
Related Method: Best-of-N follows the same generate-and-score procedure as rejection sampling but selects completions at inference time without fine-tuning the model.It computes the best completion for a static prompt or prompt set.

10 The Nature of Preferences

RLHF models human preferences to optimize difficult-to-specify objectives, combining ideas from preference research, reinforcement learning, optimal control, and deep learning. The approach inherits substantial assumptions because preferences can be contextual, unstable, multidimensional, and difficult to aggregate.

Motivation: RLHF uses human preferences as an indirect reward signal for aligning models on tasks without clear right and wrong answers.Implementation is challenging because interpreting the best practices involves substantial ambiguity.
Intellectual Foundations: Modern RLHF converges philosophy and psychology, economics and decision theory, optimal control and reinforcement learning, and modern deep learning.Each contributing area supplies assumptions about preferences and optimization.
Reward Modeling: Reward modeling compresses a multidimensional reward landscape into a scalar while unmodeled dynamics, such as preference shifts during sequential labeling, can influence decisions.The scalar signal may therefore omit relevant aspects of the decision process.
Theoretical Assumptions: Assumptions behind the VNM utility theorem are challenged in RLHF because visual presentation and the distinction between choice and preference can affect measured judgments.Numerical preference models may not capture every relevant preference in a scenario.
Measurement Challenges: Preference measurement may produce non-transitive or incomparable judgments, vary with presentation and context, and aggregate inconsistent respondent data.These issues complicate treating preference labels as a uniform optimization target.

11 Preference Data

Preference data serves as a proxy for complex human values that cannot be precisely expressed as a single reward function. Its collection depends on carefully designed interfaces, on-policy examples, and mitigation of subtle biases, while remaining costly and operationally difficult.

Why We Need Preference Data: Preference data proxies human rewards and preferences because complex human values cannot be completely captured in a single reward function.It is used to match desired behaviors and avoid unwanted failure modes.
Sourcing and Contracts: Human-data collection requires detailed instructions, substantial spending, annotator or intermediary coordination, and careful contracting.The process is described as opaque and vulnerable to administrative failures, including undelivered data and restrictive contracts.
Collecting Preference Data: On-policy preference data is collected from the current family of models, although the necessity of this practice is not well documented.Different checkpoints produce different generation patterns, making closely related data potentially more useful.
Collecting Preference Data: Preference labels are relative judgments rather than globally correct answers, so models can learn even when all compared completions are correct or incorrect.The selected answer may simply be clearer, safer, more helpful, or less incorrect than its alternatives.
Interfaces: Preference interfaces can collect pairwise choices, ratings, notes, or feedback during training and everyday product use.Early interfaces emphasized rich metadata, while deployed interfaces may also support evaluation or future training.
Bias: Things to Watch Out For in Data Collection: Prefix bias, sycophancy, verbosity, and formatting habits can enter preference data and pass into the trained model.The text characterizes mitigating these subtle biases as a key distinction between good and great preference data.

12 Synthetic Data & Distillation

Synthetic data has expanded across post-training because it is cheaper and easier to iterate, but it has not displaced human data uniformly. Distillation and on-policy methods address how stronger-model information is transferred while accounting for train-test distribution gaps.

The Roles of Synthetic Data: Synthetic data lowered the cost of RLHF experimentation and now supports many post-training stages, including prompts, completions, preference data, filtering, and verification.Its role expanded as models became reliable enough to generate and supervise training data.
The Roles of Synthetic Data: Unfiltered, repetitive, single-model self-training can cause model collapse by narrowing diversity, underrepresenting rare facts and styles, and amplifying mistakes.Mixing real or human data and using diverse training sources are identified as practical countermeasures.
The Roles of Synthetic Data: Synthetic data dominates instruction tuning, while human data remains important for capability frontiers, ground-truth labels, and parts of preference training and evaluation.The balance differs by pipeline stage: synthetic preference data can perform comparably, whereas human preference data remains an industry advantage.
Distillation with Synthetic Data: Distillation uses stronger-model outputs either as a broad post-training data engine or to transfer specific skills such as mathematical reasoning and coding.The term originates in teacher-student knowledge distillation but is used more broadly in post-training.
From Offline to On-Policy Distillation: Offline distillation trains on teacher-generated trajectories, whereas on-policy distillation trains on student rollouts with per-token teacher supervision.On-policy self-distillation uses one model in both roles, adding privileged information to create a teacher trajectory.
From Offline to On-Policy Distillation: The classic imitation-learning analysis predicts student-trajectory loss can scale quadratically with sequence length, motivating on-policy methods.For language models, this bound is an analogy rather than an exact theoretical guarantee because token distributions and KL losses differ from discrete action disagreement.
Modern OPD Variants: On-policy distillation is presented as a core method for combining multiple skills or advancing specialized deployments, while rubric-based AI feedback extends reinforcement learning beyond verifiable-answer domains.Rubric rewards are reported to improve skills including scientific reasoning and factuality.

13 Tool Use and Function Calling

Tool use extends language models beyond knowledge and capabilities contained in their weights by letting them request external operations and incorporate returned results. Function calling adds schema-constrained arguments, while evaluation measures correctness, validity, completion, and reliability.

Tool-Use Overview: Tool use lets language models access current information and perform tasks that model weights alone cannot attempt.Examples include answering time-sensitive questions through search and moving files through a filesystem tool.
Fundamentals and Formatting: Tool use consists of structured tool requests, orchestration, returned results appended to context, and continued generation.Function calling is the schema-constrained form, while code execution is tool use through a code interpreter.
Origins and Applications: Modern tool-use systems span calculators, search engines, translation, calendars, thousands of APIs, productivity applications, scientific domains, medical domains, and coding agents.Gorilla was trained on 1645 APIs, while ToolBench covers more than 16,000 real-world APIs.
Evaluation: Tool-use evaluation measures tool-name and argument correctness, schema validity, end-to-end completion, and consistency across trials.The pass^k metric targets consistent success rather than occasional success, complementing exact-match and benchmark measures.
Function Calling: Tool-use training commonly adds a system prompt describing available tools and their JSON-formatted schemas.The model then emits a structured call such as search_movies("Star Wars"), receives tool output, and continues.
Code Execution: Code execution enables precise answers to complex logic and mathematics problems by separating generated code from returned output.The example computes the 50th Fibonacci number as 12586269025.
Tool-Use Overview: MCP standardizes connections between models and external servers or clients, creating a more predictable development environment for real-world tool-use systems.The shared infrastructure supports attaching servers or clients to different models through a predictable format.

14 Over-Optimization

Over-optimization occurs when RL strongly improves a proxy reward while downstream quality or alignment with real-world goals deteriorates. In RLHF, this reflects divergence between reward-model scores and the broader objectives those scores approximate.

The over-optimization problem: RL optimizers can push language models toward satisfying reward models or checkers without aligning with training goals.The model may exploit areas of the reward signal that do not map to real usage.
The over-optimization problem: Reward over-optimization occurs when reward-model scores keep improving while held-out evaluations or human judgments eventually worsen.Qualitative degradation can also make outputs feel worse even without measurable reward hacking.
Why proxies fail: RLHF lacks a universally good chatbot reward function, so human-labeler rewards remain proxies for downstream users’ desires.Potential errors include reward-model approximation and estimation errors, policy optimization error, and uncertainty about downstream user preferences.
The over-optimization problem: Over-optimization differs from overfitting because the model improves on the proxy objective while the objective itself diverges from actual user satisfaction.The failure is metric mismatch rather than failure to generalize from training examples.
Manifestations: Observed failure modes include verbosity, confident-sounding but unhelpful answers, rare-token exploitation, over-refusal, and extreme sycophancy.The cited examples include innocuous-query refusals and a 2025 update that validated grandiose or implausible user claims.
Why proxies fail: Goodhart’s law explains why a statistical regularity can collapse when optimized as a control target, especially when an ML loss is treated as ground truth.The book frames RLHF proxy objectives as local optimization tools whose global use creates challenges.

15 Regularization

Regularization constrains post-training updates so models can gain task performance without drifting too far from prior capabilities. The section contrasts SFT’s forward-KL behavior with RL’s reverse-KL behavior, linking the distinction to generalization and forgetting.

Regularization: Post-training methods use regularization because optimization can change a model too much relative to its strong reference model.The reference is often the instruction-tuned model or a previous RL checkpoint.
KL regularization: KL divergence is commonly implemented as a penalty measuring how far the current policy drifts from a static reference policy.In practice, reverse KL often uses tokens sampled from the RL model and probabilities computed under the reference model.
SFT versus RL: Across task variants, RL improves OOD performance as compute scales, whereas SFT improves in-distribution performance but degrades OOD performance.On V-IRL with language-only inputs, RL OOD per-step accuracy rises from 80.8% to 91.8%, while SFT falls from 80.8% to 1.3%.
Forgetting: RL achieves comparable or higher target-task gains while forgetting substantially less than SFT.The reported forgetting advantage is linked to the different objectives optimized by the two methods.
KL directions: SFT minimizes forward KL KL(π⋆∥πθ), while RL maximizes its regularized objective equivalently by minimizing reverse KL KL(πθ∥π⋆).The SFT equivalence follows because the entropy term is constant with respect to θ and therefore does not change the gradients or minimum.
Reverse-KL dynamics: For multimodal policies, reverse-KL’s mode-seeking behavior can shift a new mode toward the target without disturbing an old mode containing prior knowledge.The section argues that this structural behavior preserves the breadth of prior knowledge and enables better generalization.

16 Evaluation

Evaluation in RLHF and post-training has evolved from narrow chat assessments toward generative, internally calibrated, and contamination-aware regimes. Prompting and evaluation configuration can substantially affect measured performance and reproducibility.

Evaluation phases: Evaluation began with chat-focused benchmarks using LLM-as-a-judge to scale human assessment against strong reference models.Early examples included MT-Bench, AlpacaEval, and Arena-Hard.
Scoring methods: Exact match and conditional log-likelihood scoring evaluate answers differently: sampling introduces randomness, whereas likelihood scoring compares token probabilities.Exact match is common in post-training, while log-likelihood is more common in pretraining evaluation.
Evaluation phases: Evaluation has shifted toward generative responses with chain-of-thought prompting as models use special formatting to separate reasoning and answer tokens.This shift reflects changes in how model capabilities are elicited and measured.
Reproducibility: External comparisons are difficult to reproduce because evaluation inputs, configurations, and prioritized signals differ across labs, while open standards remain hard to guarantee.Internal evaluation improves statistical power when comparing training runs by reducing noise in prioritized signals.
Contamination: Decontamination of Tülu 3 evaluations identified 8-gram overlaps between training data and popular benchmarks, including TruthfulQA, HumanEval, MATH, and safety evaluations.The examples show that contamination can affect both general and safety-oriented assessments.

17 Crafting Model Character and Products

Post-training methods increasingly shape stable model characters and products, while inference-time interventions provide additional control over persona and harmful behavioral drift. These techniques offer flexible behavior modification but remain bounded by uncertain scaling effects and implementation choices.

Character training: Character training changes model weights to establish stable traits in personality, values, and response manner, extending post-training beyond conventional capability optimization.It is presented as a practical engineering discipline spanning safety, values, and personality.
Character training: RLHF-trained character examples vary responses toward sarcastic, caring, or casual styles while preserving the same underlying interaction context.These examples illustrate personality modification through curated behavioral data.
Limitations: Persona steering has uncertain limits: increasing α may eventually reduce its effect, and further research is needed across traits such as sycophancy and hallucination.The cited discussion describes a possible U-shaped relationship between coefficient size and behavioral effect.
Persona vectors: Persona vectors steer traits in activation space without retraining, and negative α suppresses unwanted behavioral shifts introduced by fine-tuning.The method treats personality traits as reusable directions in the model’s residual stream.
Persona vectors: Scaling a persona vector changes trait intensity nearly linearly for nine of ten vectors, while vector addition and subtraction compose or contrast personality effects.Combining inventive and outgoing vectors raises Extraversion by +1.13 and Openness by +0.20 from baseline.
Assistant Axis: The Assistant Axis captures a recurring direction associated with default Assistant behavior, aligning with PC1 across three models with cosine similarity >0.60 at all layers.Similarity exceeds 0.71 at each model’s middle layer.
Assistant Axis: Activation capping reprojects drifting activations toward the Assistant Axis at inference time, reducing harmful outputs with minimal capability loss and no weight changes.The intervention targets turn-by-turn behavioral drift in sensitive conversations.
Product alignment: Model specifications clarify intended behaviors for designers and developers, but outcomes depend substantially on how much effort developers invest in following the specification.Similar goals can produce different results under strong adherence to a mediocre specification or weak adherence to an excellent one.

A Definitions

The section introduces language-modeling concepts and the terminology used to describe RLHF and post-training. It also highlights how post-training methods can improve human-facing behavior while producing trade-offs across evaluations, capabilities, compute, and variance.

Language Modeling: Autoregressive language models predict each next token conditioned on preceding tokens, factorizing sequence probability into conditional distributions.Modern language models commonly use decoder-only Transformers with self-attention.
Language Modeling: Training commonly minimizes negative log-likelihood, implemented as cross-entropy between each true token and the model’s next-token prediction.This objective fits the model to maximize the likelihood of training data.
RLHF Terminology: A policy is a probability distribution over completions, while prompts, chosen completions, rejected completions, and preference relations define the preference-learning vocabulary.The policy is parameterized as πθ(y | x), and reward models can predict preference probabilities.
Evaluation Trade-offs: RLHF-related methods often improve human preference or chat evaluations, but gains can diverge from benchmark performance and may trade against mathematics or coding.DPO is described as improving human preference evaluation while degrading benchmark evaluation; PPO-based feedback loops can also harm other tasks.
Compute and Variance: Post-training evaluation and reinforcement learning can be expensive, with checkpoint evaluation adding substantial time and RLVR extending training by approximately 3.5 weeks in one example.Repeated evaluation can reduce variance, but doing so for every evaluation can substantially increase costs.