PaperlayerView exact revision on arXiv

Source-linked AI summary

AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto

arXiv:2305.14387v4cs.LGcs.AIcs.CL

TL;DR

Learning from human feedback for instruction-following LLMs is costly, difficult to evaluate reliably, and missing validated reference implementations. AlpacaFarm addresses these gaps with simulated feedback, automatic evaluation, and reference methods; its simulated method rankings closely match human-feedback rankings, while PPO improves win-rate against Davinci003.

  • Problem

    Research on instruction-following models lacks affordable annotation, trustworthy automated evaluation, and validated implementations of learning-from-feedback methods.

  • Method

    AlpacaFarm combines oracle-LLM simulated pairwise feedback, an evaluation protocol validated against real user interactions, and reference implementations for learning from pairwise feedback.

  • Results

    Method rankings from AlpacaFarm closely agree with rankings from human-feedback training, with Spearman correlation 0.98; PPO raises LLaMA 7B's win-rate against Davinci003 from 44% to 55%.

  • Takeaways & Limitations

    AlpacaFarm supports rapid study and development of human-feedback methods by enabling simulation-based iteration before training with human data.

  • Takeaways & Limitations

    Validation uses relatively simple single-turn instructions, LLaMA 7B as the starting point, and feedback from 13 crowdworkers that may not represent broader preferences.

Abstract

from arXiv · show

Large language models (LLMs) such as ChatGPT have seen widespread adoption due to their strong instruction-following abilities. Developing these LLMs involves a complex yet poorly understood workflow requiring training with human feedback. Replicating and understanding this instruction-following requires tackling three major challenges: the high cost of data collection, the lack of trustworthy evaluation, and the absence of reference method implementations. We address these challenges with AlpacaFarm, a simulator that enables research and development for learning from feedback at a low cost. First, we design LLM prompts to simulate human feedback that are 50x cheaper than crowdworkers and display high agreement with humans. Second, we propose an automatic evaluation and validate it against human instructions obtained on real-world interactions. Third, we contribute reference implementations for several methods (PPO, DPO, best-of-n, expert iteration, and more) that learn from pairwise feedback. Finally, as an end-to-end validation of AlpacaFarm, we train and evaluate eleven models on 10k pairs of real human feedback and show that rankings of models trained in AlpacaFarm match rankings of models trained on human data. As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a +10% improvement in win-rate against Davinci003. We release all components of AlpacaFarm at https://github.com/tatsu-lab/alpaca_farm.

1 Introduction

AlpacaFarm addresses the cost, evaluation, and implementation barriers that make learning from human feedback difficult to study. It combines simulated feedback, automatic evaluation, and reference methods, then validates simulation-based rankings against human-feedback training.

  • AlpacaFarm targets high annotation costs, unreliable automated evaluation, and missing validated implementations for instruction-following research.
  • 45x cheaper simulated pairwise feedback uses oracle API LLM prompts tuned to capture human judgments, variability, and stylistic preferences.
  • The automatic protocol estimates win-rates against a baseline LLM on realistic instructions and produces rankings highly correlated with rankings from real user interactions.
  • Reference implementations include PPO, expert iteration, and Quark; PPO with a surrogate reward model raises LLaMA 7B's win-rate against Davinci003 from 44% to 55%.
  • Across eleven methods, AlpacaFarm rankings closely agree with rankings from actual human-feedback training, with a Spearman correlation of 0.98.

2 Background & problem statement

The paper studies instruction-following models trained through learning from pairwise feedback, where human comparisons guide model improvement and pairwise win-rates evaluate models. AlpacaFarm provides simulated feedback, realistic automated evaluation, and reference implementations to support this workflow and compare simulated with human-trained methods.

  • 2 Background & problem statement: Instruction following seeks a model that generates high-quality responses to open-ended user instructions according to an unobserved human reward function.
  • 2 Background & problem statement: The study focuses on pairwise feedback because it plays a central role in recent instruction-following LLMs, while algorithms differ in whether they learn a surrogate reward or operate directly on comparisons.
  • 2 Background & problem statement: Learning from pairwise feedback samples two responses per instruction, obtains human preferences, and learns from those binary comparisons.
  • 2 Background & problem statement: Pairwise model evaluation compares a trained model with a reference model and aggregates preferences as the model's average win-rate.
  • 2 Background & problem statement: AlpacaFarm combines low-cost feedback generation, automated evaluation, and reference implementations so researchers can develop methods in simulation and transfer insights to human-feedback training.
  • 2 Background & problem statement: Its end-to-end criterion is agreement between method rankings from simulated preferences and rankings from human preferences, rather than directly training a good model on simulated data.

3 Constructing the AlpacaFarm

AlpacaFarm is a low-cost simulation sandbox for developing learning-from-pairwise-feedback methods, addressing data, evaluation, and implementation challenges. It combines simulated annotators, automatic evaluation, and validated reference methods for instruction-following models.

  • Data construction: AlpacaFarm repurposes Alpaca data into 42k examples across supervised fine-tuning, pairwise preference, unlabeled, and validation splits.The splits contain 10k SFT, 10k preference, 20k unlabeled, and 2k validation examples.
  • Simulated pairwise preference: Simulated annotators use prompted API LLMs to generate pairwise preferences while targeting agreement with humans and human-like variability.The design varies models, prompts, batch sizes, in-context examples, and preference noise to emulate inter- and intra-annotator variability.
  • Simulated pairwise preference: $6 for 1000 outputs makes simulated preference annotation 50x cheaper than human annotation.The simulator is intended to preserve agreement and variability while reducing annotation cost.
  • Automatic evaluation: The automatic evaluation measures a model’s win-rate against Davinci003 using 13 simulated annotators and a diverse 805-instruction evaluation set.The evaluation set combines Self-Instruct, OASST, Anthropic helpfulness, Vicuna, and Koala test instructions, guided by real-world demo interactions.
  • Reference methods: AlpacaFarm provides reference implementations spanning direct preference methods, reward-model methods, best-of-n, expert iteration, PPO, and Quark.The methods begin with supervised fine-tuning; reward-model methods optimize continuous classifier-logit rewards, while PPO applies a KL penalty relative to the SFT initialization.

4 Validating the AlpacaFarm simulator

AlpacaFarm’s simulated annotators and evaluation protocol closely reproduce key human-feedback properties, including method rankings and reward-model overoptimization. Its validation combines controlled comparisons with real human annotations and Alpaca Demo interactions.

  • End-to-end validation: Spearman correlation of 0.98 shows that simulated and human-feedback workflows produce closely agreeing method rankings.The comparison covers eleven methods trained and evaluated in the two workflows.
  • End-to-end validation: Two rank mismatches occur for SFT10k versus SFT52k and ChatGPT versus PPO, but neither is considered a major error.Human preferences were 44.3% versus 40.7% and 55.1% versus 52.9%, respectively, while the simulator reversed both comparisons.
  • Simulated annotator agreement: 65% evaluator agreement with the human majority matches the 66% held-out human agreement rate, while evaluation costs fall from $300 to $12 per 1000 examples.The training annotator adds unbiased label-flip noise without changing the underlying preference function.
  • Simulated annotator variability: Human and AlpacaFarm preferences reproduce reward overoptimization, whereas simple GPT-4 preferences do not.As surrogate rewards increase, win-rates eventually decrease; added annotator variability is needed to reproduce this behavior.
  • Evaluation protocol: Simulated win-rates on AlpacaFarm evaluation strongly correlate with win-rates on 200 real Alpaca Demo instructions, with r2 = 0.97.The Demo instructions exclude personal identifying information, toxic or unsafe questions, and chatbot-directed questions.

5 Benchmarking reference methods on the AlpacaFarm

Benchmarking shows that supervised fine-tuning supplies most gains, while reward-model-based methods—especially PPO—provide the strongest further improvements. The results also expose computational and deployment trade-offs in training and simulator design.

  • Comparing LPF methods: SFT raises LLaMA’s simulator win-rate from 11% to 37% and human win-rate from 7% to 44%, with little or no gain from SFT 10k to SFT 52k.SFT therefore provides the majority of the observed win-rate gains before preference-learning methods are applied.
  • Comparing LPF methods: Best-of-n outperforms every studied preference-learning method except PPO, indicating that the reward model captures useful learning signals.The comparison concerns training-time methods evaluated through AlpacaFarm’s simulated and human-feedback workflows.
  • Comparing LPF methods: Expert Iteration trails best-of-n by 3–6% in both workflows, while Quark and direct pairwise-feedback methods do not outperform the SFT 10k baseline.The results suggest that learning a surrogate reward model may be important for these methods.
  • Computational cost: Most reference-method training takes under 2 hours on one 8×A100 machine, with decoding many samples dominating the largest costs.Best-of-n and Expert Iteration incur the largest computational costs because they decode many candidate samples.
  • Output analysis: PPO and best-of-n make outputs substantially longer, increasing average length from 278 characters for SFT 10k to 570 for Best-of-16 and 637 tokens for PPO.The analysis examines output changes to understand factors contributing to win-rate improvements.
  • Model transfer: Simulation-only PPO reaches 43% human-evaluation win-rate versus 50% for PPO trained on single-annotator GPT-4 data and 55% for human-trained PPO.Simulation-only training remains competitive with the 44% initial SFT model but has a 5% gap relative to real human annotations.
  • Model transfer: Simulator design presents a faithfulness–performance trade-off: variability improves behavioral fidelity, while a consistent GPT-4 annotator can yield better directly deployable models.Standard AlpacaFarm evaluators are intended for method development and selection, whereas consistent annotation supports direct deployment.

6 Related work

AlpacaFarm relates to work on instruction following, AI-simulated feedback, simulated human behavior, and low-cost simulator environments. It focuses these ideas on pairwise feedback for training and method-level validation, while leaving alternative feedback modalities and some RL algorithms for future work.

  • AlpacaFarm studies instruction following through real-world human interaction patterns rather than collections of existing NLP benchmarks.
  • Unlike Constitutional AI, AlpacaFarm simulates human feedback to reflect outcomes of experiments performed with real human feedback.
  • AlpacaFarm distinguishes itself by using pairwise feedback for training and validating beyond per-example agreement metrics.
  • The work complements persona-based and behavioral human simulation by showing that simulated LLM annotators can replicate qualitative features of pairwise human-feedback training.
  • AlpacaFarm applies simulator-based experimentation to human preference feedback rather than physical systems.
  • The study focuses on methods that learn from pairwise feedback, while alternative sources include natural language, numeric ratings, and execution traces.
  • The evaluated RL algorithm set is not comprehensive, leaving broader RL-for-NLP exploration to future work.

7 Limitations and future directions

AlpacaFarm’s validation and simulated annotators have important scope boundaries, while its hyperparameters differ between simulated and human feedback. The authors propose extending simulation to other domains and feedback forms.

  • Validation uses relatively simple single-turn instructions, LLaMA 7B starting models, and feedback from 13 crowdworkers that may not reflect broader human preferences.
  • AlpacaFarm assumes access to an oracle LLM substantially more powerful than the investigated models, an assumption that may fail when building state-of-the-art models.
  • Suitable learning hyperparameters differ between simulated and human feedback, including the range of RLHF KL regularization coefficients.
  • Simulated annotators can prefer first-position outputs and outputs from their own model, despite controls for these biases.
  • Future work aims to extend AlpacaFarm beyond pairwise feedback to other domains and forms of human supervision.

A Reference LPF methods on AlpacaFarm

The reference methods cover direct preference learning, supervised baselines, and algorithms that optimize a surrogate reward model. These implementations support comparisons across training-time and inference-time approaches.

  • The methods are organized by whether they fit a surrogate reward model during learning, alongside baseline methods from the OpenAI API.
  • The OpenAI API baselines include GPT-4, ChatGPT, Davinci001, and Davinci003 as the comparison baseline.
  • Binary FeedME adapts FeedME to pairwise feedback by fine-tuning the SFT model on the chosen response from each preference pair.
  • Binary reward conditioning fine-tunes the SFT model using positive or negative control tokens attached to preference examples.
  • DPO maximizes preference likelihood under a Bradley–Terry model using a reward model implicitly defined by the current KL-regularized policy.
  • Best-of-n sampling returns the highest-reward response among n independent SFT-model samples, improving generation quality at high inference cost.
  • Expert iteration performs best-of-n sampling, stores the samples, and fine-tunes the SFT model on them in one round.
  • PPO alternates sampling from the current policy with clipped, importance-sampled gradient updates under a KL-regularized objective.

B Details on methods implementation and hyperparameters

The implementation uses customized PPO and Quark procedures, specified optimization settings, and prompt-engineering choices for simulated annotators. Evaluation relies on a varied annotator pool, while training adds controlled noise and cost-saving labeling.

  • PPO implementation: PPO normalizes advantages across each rollout batch and initializes the value model from the reward model to improve small-batch stability.
  • PPO hyperparameters: PPO uses batch size 512 per step, two gradient epochs with batches of 256, a peak learning rate of 10^-5, and gradient clipping at 1.
  • Quark implementation: Quark discards previous rollouts after generating a new batch to avoid growing pool overhead during training.
  • Quark hyperparameters: The reported Quark run uses KL coefficient 0.05, peak learning rate 3 × 10^-6, rollout batch size 512, and 390 training steps.
  • DPO hyperparameters: DPO on noisy simulated preferences uses β = 0.1, one training epoch, batch size 64, and a peak learning rate of 1e^-5.
  • Annotator setup: Simulated annotators use OpenAI API-generated outputs and prompts released with the project.
  • Evaluation annotators: The evaluation pool contains 13 simulated annotators varied by model, in-context examples, prompt format, and preferences.
  • Training annotators: Training annotators flip evaluation outputs with probability 0.25, requiring labels for only half the outputs and making collection 2× faster and cheaper.

C.2 Additional results

Simulated annotators reproduce several human preference properties, including low bias, high variance, and stylistic preferences for longer outputs and lists. Most variability within the annotator pool comes from the underlying model.

  • Annotator agreement and variability: Simulated annotators show low bias and high variance, more closely matching human evaluation variability than single evaluators.Single evaluators have variance below 0.2, compared with human variance of 0.34; simulated evaluator bias values are 0.38 and 0.35, versus 0.33 for a strong GPT-4 annotator.
  • Annotator agreement and variability: The underlying model is the largest source of variability between simulated annotators.Agreement heatmaps form clusters associated with GPT-4, ChatGPT, and Davinci003 annotators.
  • Stylistic preferences: Humans prefer longer outputs 62% of the time, compared with 64% for simulated annotators.These results indicate close alignment in a measured stylistic preference.
  • Stylistic preferences: Humans prefer outputs with lists 69% of the time, compared with 63% for simulated annotators.The paper reports that simulated annotators therefore match this human stylistic preference reasonably well.

D Details on human data collection

The human-data collection process qualifies crowd annotators, provides structured pairwise-response guidance, and removes a clear outlier. The interface accommodates marginal differences through slightly-better or slightly-worse labels.

  • Qualification: The study selected the top 16 annotators from five student authors’ agreement on 25 qualification examples.The qualification examples were generated with an OPT 6B model, and the selected annotators had the highest agreement with the authors.
  • Quality control: One annotator with approximately 50% agreement with GPT-4 was removed as a clear outlier.The annotator was discontinued during the project and their annotations were removed.
  • Annotation outcomes: Around 18% of collected preferences selected a slightly-better option.The paper notes that these labels arise when response pairs differ only marginally, including punctuation or minimal edits.
  • Annotation task: Annotators compare two AI responses for helpfulness, naturalness, factual accuracy, and overall quality.The instructions frame these dimensions as questions to consider when choosing which response is better.
  • Annotation interface: The annotation interface includes the instruction, optional input, and two model responses for comparison.The task asks workers to decide which response is better using the provided evaluation dimensions.
  • Annotation task: The guideline offers four pairwise choices: better or slightly better for either response.Slightly-better options are reserved for marginal differences, while annotators are asked to use them sparingly when responses are similar.

E.1 Label noise ablations for simulated annotators

Ablations identify label noise as the main ingredient producing reward overoptimization in simulated annotators. Without label noise, win-rates continue increasing as best-of-n sample counts grow.

  • Ablation design: The simulator models variability through randomized annotators and label noise.Randomization varies the simulated annotator, while label noise flips simulated preferences before reward-model training.
  • Ablation design: The study uses best-of-n sampling to compare reward models trained on the ablated preference distributions.The ablation isolates how annotator randomization and label noise affect overoptimization under increasing sample counts.
  • Ablation results: Added label noise provides the majority of the overoptimization effect in the ablation.The experiment trains reward models on varied preference distributions and compares their best-of-n sampling performance.
  • Ablation results: Without label noise, simulated annotators’ win-rates keep increasing with more best-of-n samples.The two no-label-noise options continue improving as sample counts increase.
  • Computational considerations: Best-of-n has no training cost but incurs inference cost that can reach 16x at the optimal n of around 16.The reported cost concerns batched inference rather than model training.
  • Computational considerations: PPO performance peaks after 40–80 steps, requiring less than two hours on the full training setup.The timing is reported for the authors’ implementation and a single 8×A100 machine.

E.3 Additional analysis of model outputs

The output analysis examines length changes after learning from pairwise feedback and compares training and evaluation instruction distributions. LPF training generally produces longer outputs, while the two instruction distributions match at a high level.

  • Output length: LPF training generally leads to longer model outputs under both human and simulated preferences.Tables 6 and 7 report average output lengths, while Table 5 provides qualitative before-and-after examples.
  • Output length: The qualitative examples compare SFT 10k, PPOhuman, and PPOsim outputs before and after preference-based training.The analysis uses these examples to illustrate the reported increase in output length.
  • Instruction distributions: Both Alpaca training and AlpacaFarm evaluation instructions cover diverse instruction types.The inner wheel shows root verbs and the outer wheel shows direct subjects.
  • Instruction distributions: The training and evaluation instruction distributions match at a high level.The comparison is based on the root-verb and direct-subject breakdowns shown in Figures 14 and 15.
Loading 2305.14387v4…