Source-linked AI summary
LIMA: Less Is More for Alignment
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy
TL;DR
The paper asks whether large-scale instruction tuning and human-feedback reinforcement learning are necessary for aligning a strong pretrained language model. It trains LIMA with supervised fine-tuning on 1,000 curated examples and finds competitive preference results, while noting limits in scalability and robustness.
Problem
Existing alignment methods require substantial compute and specialized instruction-tuning or human-feedback data, motivating evaluation of whether much less alignment data suffices.
Method
LIMA fine-tunes a pretrained 65B-parameter LLaMa model on 1,000 carefully curated prompts and responses spanning diverse tasks with a consistent helpful-assistant style.
Results
LIMA responses were equal or preferred to GPT-4 in 43% of human comparisons, to Bard in 58%, and to DaVinci003 in 65%, while 50% were rated excellent.
Takeaways & Limitations
The findings support the Superficial Alignment Hypothesis and indicate that pretraining contributes more to model capabilities than large-scale alignment training.
Takeaways & Limitations
Constructing diverse, high-quality examples is laborious and difficult to scale, and LIMA is less robust than product-grade models under unlucky decoding or adversarial prompts.
Abstract
from arXiv · showhide
Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.
1 Introduction
The paper argues that alignment may require far less data than prevailing instruction-tuning and RLHF approaches, because pretraining already supplies broad knowledge and capabilities. LIMA tests this idea with a carefully curated 1,000-example dataset and reports strong comparative results.
- Existing alignment methods rely on large multi-million-example instruction-tuning datasets and millions of human-annotated interactions for RLHF.
- The Superficial Alignment Hypothesis proposes that pretraining acquires most knowledge and capabilities, while alignment mainly teaches interaction formats.
- LIMA uses a pretrained 65B-parameter LLaMa model fine-tuned on exactly 1,000 carefully curated prompts and responses.
- Ablations found diminishing returns from increasing data quantity without prompt diversity, but major gains from improving data quality.
2 Alignment Data
The alignment dataset combines diverse prompts with stylistically consistent, helpful responses from community sources and manual authoring. Its construction prioritizes prompt diversity and response quality over sheer quantity.
- The Superficial Alignment Hypothesis treats alignment as learning which response-format subdistribution to use, rather than acquiring most model knowledge.
- The dataset contains 1,000 diverse prompts and stylistically aligned responses, including community Q&A, manually authored examples, and held-out development and test sets.
- Community data came from Stack Exchange, wikiHow, and Reddit, with Reddit requiring manual curation because highly upvoted responses were often humorous or trolling.
- wikiHow examples use article titles as prompts and cleaned article bodies as responses, drawing on a heavily moderated how-to corpus.
- Manual examples were designed for task diversity and a uniform helpful-assistant tone, with consistent response formatting generally improving performance.
- The authors identify manual construction of diverse prompts and rich, uniformly styled responses as laborious and difficult to scale.
3 Training LIMA
LIMA is trained by standard fine-tuning of LLaMa 65B on the 1,000-example alignment set. The protocol uses speaker-separating end-of-turn tokens and manually selected checkpoints.
- LIMA starts from LLaMa 65B and is fine-tuned on the 1,000-example alignment training set.
- Training runs for 15 epochs with AdamW, learning-rate decay, batch size 32, sequence trimming at 2048 tokens, and residual dropout increasing to p_d=0.3.
- Checkpoints are manually selected between epochs 5 and 10 using a held-out 50-example development set because perplexity does not correlate with generation quality.
4 Human Evaluation
LIMA is evaluated against five baselines through human and GPT-4 preference judgments, alongside absolute analyses of response quality, generalization, and safety. The results show strong comparative performance despite LIMA’s much smaller training set, while revealing uneven safety on sensitive prompts.
- Experiment setup: Human preference evaluation compares LIMA with five baselines across 300 challenging test prompts, using pairwise judgments of model responses.
- Experiment setup: GPT-4 annotators receive the same instructions and data as human annotators, and their judgments show similar trends to the human preference study.
- Absolute analysis: 50% of LIMA’s answers are rated excellent in a manual analysis, and the model follows all but 6 of 50 analyzed prompts.
- Out-of-distribution analysis: On 20 out-of-distribution examples, 45% of responses are excellent, 35% pass, and 20% fail, suggesting similar absolute performance outside the training distribution.
- Safety analysis: LIMA responds safely to 80% of 30 sensitive prompts, including 6 of 10 with malicious intent, but implicit malicious intent more often elicits unsafe responses.
5 Why is Less More? Ablations on Data Diversity, Quality, and Quantity
Ablations show that alignment benefits more from diverse prompts and high-quality responses than from quantity alone. Doubling training data did not improve response quality, while filtered data produced a significant advantage.
- Diversity and output quality have measurable positive effects on alignment, whereas scaling quantity alone might not.
- Diversity: The more diverse Stack Exchange data yields significantly higher performance than homogeneous wikiHow data.Both sources provide high-quality responses, but wikiHow uses homogeneous “how to” prompts.
- Quality: A 0.5-point difference separates models trained on filtered versus unfiltered Stack Exchange data.The comparison uses 2,000 examples from each source.
- Quantity: At least 2,000 examples improved training stability in the 7B setting, although preliminary experiments could tune the model with 1,000 examples.
- Quantity: Doubling the training set does not improve response quality, suggesting alignment scaling depends on prompt diversity while maintaining high-quality responses.The quantity experiment samples exponentially increasing datasets from Stack Exchange, up to a 16-fold increase.
6 Multi-Turn Dialogue
LIMA can conduct coherent multi-turn dialogue despite training only on single-turn interactions, but its out-of-distribution failures are reduced substantially by adding 30 dialogue examples. The added examples improve both response quality and whole-dialogue evaluations.
- With zero dialogue examples, LIMA produces coherent responses that reference earlier turns, but fails to follow the prompt in 6 of 10 conversations within three interactions.
- Adding 30 dialogue chains to create a 1,030-example training set substantially improves multi-turn generation quality.The added chains include 10 author-composed dialogues and 20 edited Stack Exchange comment chains.
- Excellent responses rise from 45.2% to 76.1% after dialogue fine-tuning.
- The failure rate drops from 15 fails per 42 turns to 1 fail per 46 turns after adding dialogue examples.
- The fine-tuned model is significantly better in 7 of 10 conversations and tied in the remaining 3.
7 Discussion
The paper finds that 1,000 carefully curated examples can produce competitive results, but constructing them is difficult to scale and the resulting model is less robust than product-grade systems.
- Constructing 1,000 carefully curated examples requires significant mental effort and is difficult to scale.
- LIMA typically generates good responses but can produce weak outputs from unlucky decoding or adversarial prompts.The paper characterizes LIMA as less robust than product-grade models.
A Training Examples
The paper presents six training examples from various sources.
- Six training examples are shown from various sources.
B Anticorrelation between Perplexity and Generation Quality
During LIMA fine-tuning, higher held-out perplexity coincided with higher generation quality, contrary to the usual overfitting interpretation of rising perplexity.
- As perplexity rose with additional training steps, ChatGPT-evaluated generation quality also increased.This pattern was observed on held-out Stack Exchange data comprising 2,000 examples.
- The authors evaluated model generations with ChatGPT to quantify the observed relationship between perplexity and response quality.
- Rising perplexity was typically interpreted as overfitting, but here it accompanied improved generation quality.
D ChatGPT Score
The study uses ChatGPT to automatically score model outputs for ablation experiments, addressing the difficulty of evaluating generative models.
- ChatGPT evaluates model outputs on a 6-point Likert score for the ablation experiments.
- The evaluation prompt includes the task and the model submission as the items being judged.
- This automated evaluation method is used because evaluating generative models automatically is difficult.
E Generating Text with Complex Structure
LIMA learns to satisfy complex structural and formatting requirements from only a few targeted examples, then generalizes those requirements to unseen tasks.
- E Generating Text with Complex Structure: Six additional formatting examples substantially improve LIMA’s responses to prompts with structural constraints.The examples include product pages with named sections and question-answer pairs derived from an article.
- E Generating Text with Complex Structure: After structure-oriented training, LIMA generates a complex marketing plan despite having no marketing-plan example in its training data.
- E Generating Text with Complex Structure: Figure 10 contrasts training examples mined from community Q&A with manually authored examples, whose responses are longer than the displayed excerpts.
- E Generating Text with Complex Structure: The comparison contrasts LIMA without the six format-constraint examples against LIMA trained on all 1,000 examples.
- E Generating Text with Complex Structure: Figure 13 presents model outputs for test prompts requiring specified response structures, with long responses truncated by gray ellipses.