Source-linked AI summary
Direct Language Model Alignment from Online AI Feedback
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, Mathieu Blondel
TL;DR
DAP methods avoid separate reward models but rely on fixed preference data, creating offline feedback and distribution shifts as the aligned policy changes. OAIF instead has an LLM annotate response pairs sampled from the current policy and uses those preferences with standard DAP losses. Human evaluations show online DAP outperforming offline counterparts and competing alignment methods, while prompting can control response length with a quality trade-off.
Problem
DAP methods use fixed preference datasets, so they lack feedback on the aligned policy’s own generations and train on distributions that differ from the evolving policy.
Method
OAIF samples two responses from the current policy, uses an LLM to select the preferred response, and updates the policy with standard DAP losses.
Results
Online DAP methods achieved a 66% average human-evaluation win rate over their offline counterparts and online DPO outperformed RLAIF and RLHF across various tasks.
Takeaways & Limitations
OAIF provides a general way to make DAP methods online while retaining direct alignment and enabling prompt-controlled alignment goals.
Takeaways & Limitations
The study evaluates in-distribution prompts and only the PaLM 2-XS model, leaving out-of-distribution performance and scaling effects uninvestigated.
Abstract
from arXiv · showhide
Direct alignment from preferences (DAP) methods, such as DPO, have recently emerged as efficient alternatives to reinforcement learning from human feedback (RLHF), that do not require a separate reward model. However, the preference datasets used in DAP methods are usually collected ahead of training and never updated, thus the feedback is purely offline. Moreover, responses in these datasets are often sampled from a language model distinct from the one being aligned, and since the model evolves over training, the alignment phase is inevitably off-policy. In this study, we posit that online feedback is key and improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as annotator: on each training iteration, we sample two responses from the current model and prompt the LLM annotator to choose which one is preferred, thus providing online feedback. Despite its simplicity, we demonstrate via human evaluation in several tasks that OAIF outperforms both offline DAP and RLHF methods. We further show that the feedback leveraged in OAIF is easily controllable, via instruction prompts to the LLM annotator.
1. Introduction
DAP methods simplify alignment by directly updating language models from preference pairs, but fixed, mismatched datasets make feedback offline and training off-policy. OAIF addresses this by having an LLM annotate responses sampled from the current model, with evaluations showing improved performance and controllable response length.
- Motivation: DAP methods directly update language models from pairwise preferences, avoiding the separate reward model required by RLHF.This makes DAP alignment simpler, more efficient, and more stable than RLHF.
- Motivation: Fixed preference datasets provide offline feedback because the aligned policy cannot receive feedback on its own generations during training.Dataset responses are often generated by different language models, creating a distribution shift between training and evaluation.
- Proposed approach: OAIF samples two responses from the current policy, asks an LLM to choose the preferred response, and updates the policy with standard DAP losses.The method combines DAP’s direct updates with online feedback without training a separate reward model.
- Results: 66% was the average human-evaluation win rate of online DAP methods over offline versions of the same methods.The comparison covered online and offline DPO, IPO, and SLiC.
- Results: 58.00% of TL;DR four-way comparisons favored online DPO over SFT, RLHF, and RLAIF.This result came from human evaluation.
- Controllability: Instruction prompts shortened aligned responses from approximately 120 to approximately 40 while quality remained above the SFT baseline.Response length served as the test bed for annotator controllability.
2. Background
DAP methods train directly on preference pairs, but pre-collected data creates offline feedback and distribution shifts as the aligned policy evolves. The background contrasts this limitation with online RLHF and motivates making DAP online without sacrificing its simpler loss-based updates.
- Preference data: Preference datasets sample two responses from an existing language model and store an annotator-ranked preferred and less-preferred pair.Repeating this process over prompts constructs the fixed preference dataset used by DAP methods.
- DAP methods: DAP losses directly update the target policy from preference pairs sampled from the data-generating policy ρ.The losses include DPO, IPO, and SLiC formulations.
- DAP methods: DAP gradients can be computed exactly and efficiently, unlike RLHF, which typically uses policy-gradient estimates and an additional value function.The value function reduces gradient variance but requires another model in memory.
- Offline versus online: Offline DAP prevents the policy from receiving feedback on its own generations during alignment, while RLHF and RLAIF acquire training data interactively.The fixed dataset is typically collected before alignment because online human preferences are expensive and complex to obtain.
- Distribution shift: Pre-collected DAP data makes alignment off-policy because the current policy πθt differs from the policy ρ that generated the dataset and keeps evolving.DPO reduces the initial shift through supervised fine-tuning, but the off-policy issue remains during alignment.
- Bridging the gap: OAIF is positioned as a bridge between simple DAP losses and the online interaction of RLHF, while RM-based approaches still inherit shifts from pre-collected data.The proposed approach uses an LLM annotator rather than a separately trained reward model.
3. Direct alignment from online AI feedback
OAIF makes DAP training online and on-policy by collecting preferences from the current policy’s sampled responses through an LLM annotator. The procedure supports differentiable DAP losses and allows the feedback criterion to be changed through prompting.
- Online AI feedback: OAIF collects preferences on-the-fly from response pairs sampled from the policy being aligned, using an LLM as the annotator.AI feedback is used because collecting human preferences online would be prohibitively expensive.
- Algorithm: Sampling responses from the current policy ensures on-policy learning, while annotating those responses provides online learning.The approach works with any differentiable DAP loss function.
- Algorithm: Each training iteration samples a prompt and response pair, obtains an LLM preference, and updates θ using the DAP-loss gradient.The algorithm takes a prompt dataset, an SFT baseline, an LLM annotator, and a DAP loss as inputs.
- Gradient computation: OAIF stops gradients through response sampling and LLM annotation while computing the DAP gradient with respect to the model parameters.This reflects that θ affects both sampling and the DAP loss in online DAP training.
- Prompting: Pairwise prompting asks the annotator to choose between two responses, averages both response orders, and reduces position bias.Changing the prompt can adapt the reward signals or preference function.
4. Experiments
Experiments evaluate OAIF across alignment tasks, DAP objectives, annotator sizes, and comparisons with offline DAP, RLHF, and RLAIF. Results generally favor online, on-policy feedback, while prompt-controlled shortening improves over SFT but reduces helpfulness.
- Experimental setup: Experiments use TL;DR, Helpfulness, and Harmlessness tasks with PaLM 2 models and evaluate win rates using AI and human judgments.Human raters score quality from 1 to 5 and select the best response; automatic evaluations use Gemini Pro.
- Online versus offline DPO: Offline DPO rapidly overfits pre-collected preferences, whereas online DPO’s win rate keeps increasing and surpasses offline DPO after 4,000 steps.The pattern is consistent under Gemini Pro and PaLM 2-L automatic evaluation.
- Online versus offline DPO: Human evaluation shows that OAIF significantly improves DPO across TL;DR, Helpfulness, and Harmlessness, with substantial superiority over offline DPO.The comparison uses side-by-side evaluations of selected online and offline DPO models.
- Other DAP methods: OAIF generalizes across DPO, IPO, and SLiC, whose online variants achieve win rates of approximately 64% to 71% against offline counterparts.The results support OAIF as a framework compatible with arbitrary DAP loss functions.
- Comparison with RLHF and RLAIF: Online DPO is preferred 58% of the time in four-way human comparisons against offline DPO, RLAIF, and RLHF on TL;DR.Quality remains higher for OAIF responses than other methods when responses are compared at fixed lengths, addressing length bias.
- Annotator size: Smaller annotators still improve alignment over SFT, although larger annotators generally perform better and PaLM 2-XS achieves performance comparable to RLHF.Annotator size therefore affects performance without eliminating the benefit of OAIF.
- Prompt controllability: Prompting annotators to prefer shorter responses reduces average length from approximately 120 tokens to 90 or 40 tokens, while quality remains above SFT but helpfulness declines.Human quality scores decrease from 4.08 to 3.72 and 3.26, compared with 3.19 for SFT.
5. Discussion
The discussion highlights OAIF’s controllability and possible extensions, while identifying prompt-distribution, scaling, and response-length limitations. It also considers real-time human feedback but notes its sample-efficiency challenge.
- Limitations: The study assumes evaluation prompts are in-distribution because prompts come from the preference dataset, leaving out-of-distribution prompt evaluation.Prompt-distribution shift and shifts in the ground-truth human value function are not fully studied.
- Limitations: Scaling remains unexamined because the aligned model in Section 4 is always PaLM 2-XS, while higher-quality responses may be harder to distinguish.The paper identifies feedback quality for responses from larger LLMs as requiring further study.
- Qualitative preference annotation from LLMs: OAIF can align language models to qualitative objectives by changing LLM-annotator prompts, reducing reliance on human labellers.The paper uses response length as a test-bed and argues that the approach can extend to objectives such as helpfulness and impartiality.
- Preference from real-time human feedback: Replacing LLM annotators with real-time users could personalize alignment, but roughly 256,000 samples for 2,000 steps is excessive for one user.The paper identifies sample efficiency as the bottleneck and leaves single-user personalization to future research.
6. Conclusion
The conclusion presents OAIF as a way to make DAP methods online, improving them over offline counterparts and competing favorably with RLHF and RLAIF. It also reports generality across three DAP methods and controllability through prompts.
- 6. Conclusion: OAIF makes DAP methods online through AI feedback and outperforms their offline counterparts across empirical evaluations.The conclusion also reports that offline DAP methods tend to overfit, whereas OAIF can mitigate reward overoptimization.
- 6. Conclusion: OAIF generalizes across DPO, IPO, and SLiC, demonstrating effectiveness beyond a single DAP method.The conclusion explicitly identifies these three methods as prominent DAP methods evaluated with OAIF.
- 6. Conclusion: Online DPO outperforms RLAIF and RLHF across various tasks when both methods receive feedback from identical models.This comparison isolates the learning procedures because the AI feedback quality is held constant.
- 6. Conclusion: Instruction prompts can control the LLM annotator, allowing OAIF to pursue desirable alignment goals.Response length serves as the test bed for demonstrating this controllability.
- 6. Conclusion: OAIF may support more scalable alignment strategies requiring less human annotation effort.This is presented as the broader direction enabled by the demonstrated effectiveness of OAIF.
Impact statements
The paper frames OAIF as a method for improving alignment while supporting more scalable alignment with reduced human effort, within responsible and safe AI considerations.
- Impact statements: OAIF aims to improve alignment with human values while reducing human effort through AI feedback.The paper places this approach within the broader context of responsible and safe AI because it relies on AI feedback.
Author contribution statement
The statement lists individual contributions spanning project conception, coding, experimentation, prompt development, human evaluation, supervision, and paper writing.
- Guo proposed the project, wrote the initial codebase and experimental prompts, ran initial experiments, and wrote the paper.
- Zhang wrote the codebase, ran the main experiments, further developed the prompts, and wrote the paper.
- Tianlin Liu and Yao Zhao participated in discussions, with Zhao also contributing to the initial codebase.
- Khalman performed human evaluation and helped write the experiment section, while Llinares helped implement the initial codebase and set up experiments.
- Piot, Liu, and Ramé contributed to codebases, discussions, and paper feedback in varying combinations.
- Blondel, Ferret, and Mesnard contributed supervision, paper writing, implementation, or paper comments.
A.3. Distribution shift between RM training and inference
The reward model is trained on preference data associated with one policy but labels responses generated by the evolving alignment policy, creating a distribution shift.
- The reward model is trained on D associated with ρ, while the aligned policy at step t is πθt.
- At inference, the reward model labels responses from πθt rather than the distribution used for its training data.
- This mismatch creates a shift between the reward model’s training distribution and inference distribution.
B. Distribution Shift in Preference Data Curation
The paper examines distribution shift by comparing GPT-2 Large’s probabilities for on-policy preference responses with an off-policy response generated by PaLM 2-S.
- Figure 8 compares GPT-2 Large log-probabilities for preferred, less-preferred, and off-policy responses.
- GPT-2 Large assigns significantly lower probabilities to PaLM 2-S generations than to on-policy responses.
- The observed margin verifies a distribution shift between on-policy and off-policy preference data.
C. Alignment Accuracy of Gemini Pro
The paper evaluates how closely Gemini Pro’s preference judgments match human judgments and compares its accuracy with PaLM 2 L.
- The evaluation measures alignment accuracy as the accuracy of LLM-labelled preferences relative to human preferences.
- 70.21% average alignment accuracy was achieved by Gemini Pro against human preferences.
- Gemini Pro’s 70.21% accuracy was comparable to PaLM 2 L’s 70.72%.
- The results support using Gemini Pro for judgment, although its judgment quality had not been well studied beforehand.
D. Win Rate of Online DPO and Offline DPO against SFT over Training on TL;DR by PaLM 2 L
Figure 9 tracks online and offline DPO win rates against the initial SFT baseline over training, judged by PaLM 2 L. The evaluation uses pairwise LLM preferences, with position-bias correction by averaging both response orders.
- Figure 9 measures online DPO and offline DPO win rates against the initial SFT baseline over training using PaLM 2 L judgments.
- The LLM compares two candidate responses and converts the log-probabilities of generating “1” versus “2” into a preference score.These scores provide online AI feedback and support win-rate computation.
- The method averages preference distributions from both response orders to mitigate non-negligible positional bias.
- For TL;DR summarization, the prompt presents unlabeled input text and two policy-generated summary candidates for pairwise selection.
- For Helpfulness, prompt instructions disable conversational extensions and support controlling the LLM reward signal through text.The highlighted instruction was not used for evaluation.
- Harmlessness ratings reverse the preference distribution to obtain the AI rating for harmless responses.