Source-linked AI summary
Self-Alignment with Instruction Backtranslation
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, Mike Lewis
TL;DR
Instruction-following models require high-quality annotated or preference data that is difficult to scale. The paper introduces instruction backtranslation, which uses a model to generate and curate instruction pairs from unlabeled text, and reports that Humpback outperforms other non-distilled models on the Alpaca leaderboard. The approach assumes a base model, seed data, and suitable unlabeled examples.
Problem
High-quality instruction-following annotation is difficult to scale, motivating methods that use large amounts of unlabeled data.
Method
Instruction backtranslation iteratively uses a seed model to generate instructions for web documents, curate high-quality pairs, and finetune improved instruction-following models.
Results
Humpback outperforms all other existing non-distilled models on the Alpaca leaderboard.
Takeaways & Limitations
The method provides a scalable way for language models to improve their own instruction-following ability using unlabeled data and fewer human-annotated examples.
Takeaways & Limitations
The approach assumes access to a base language model, a small seed dataset, and a large, diverse collection of unlabeled examples suitable for instruction pairing.
Abstract
from arXiv · showhide
We present a scalable method to build a high quality instruction following language model by automatically labelling human-written text with corresponding instructions. Our approach, named instruction backtranslation, starts with a language model finetuned on a small amount of seed data, and a given web corpus. The seed model is used to construct training examples by generating instruction prompts for web documents (self-augmentation), and then selecting high quality examples from among these candidates (self-curation). This data is then used to finetune a stronger model. Finetuning LLaMa on two iterations of our approach yields a model that outperforms all other LLaMa-based models on the Alpaca leaderboard not relying on distillation data, demonstrating highly effective self-alignment.
1 INTRODUCTION
Instruction backtranslation addresses the difficulty of scaling high-quality human annotation by using unlabeled data and iterative self-training. The resulting Humpback model outperforms other non-distilled models on the Alpaca leaderboard.
- High-quality instruction-following annotation is difficult to scale despite its importance for aligning language models.
- Instruction backtranslation uses unlabeled data to create instruction-tuning data through iterative self-training, with the model augmenting and curating examples.
- The method generates instructions for web documents, filters the resulting examples by quality, and iterates training on the selected pairs.
- Humpback outperforms all other existing non-distilled models on the Alpaca leaderboard.
2 METHOD
Instruction backtranslation assumes a base model, seed examples, and diverse unlabeled text that can serve as candidate outputs. It alternates instruction generation with quality-based self-curation, iteratively training improved models on the selected data.
- Assumptions: The approach assumes access to a base language model, a small seed dataset, and a large collection of diverse unlabeled examples.
- Iterative self-curation: Each iteration trains an improved model on seed data and previously curated augmentation data, then uses it to rescore candidates for the next iteration.
- Core process: The method performs two core steps: self-augmentation generates instructions for unlabeled data, and self-curation selects high-quality demonstration examples.
- Seed data: Seed data provides human-annotated instruction-output examples for predicting outputs from instructions and instructions from outputs.
- Unlabeled data: Web documents are segmented, deduplicated, length-filtered, and screened with heuristics before being used as unlabeled data.
- Self-augmentation: A backward model p(x|y) predicts candidate instructions for document segments, producing augmented instruction-output pairs.
- Self-curation: The seed instruction model scores each candidate pair on a 5-point scale, and examples meeting the threshold are selected for training.
- Data-source tagging: System prompts distinguish seed examples from augmented examples during joint finetuning and inference.
3 EXPERIMENTS
Experiments evaluate instruction backtranslation across data construction, scaling, model quality, and ablations. Results show that self-curated augmentation, especially combined with seed data, improves instruction following and data efficiency.
- 3.1 EXPERIMENTAL SETUP: Experiments use 3200 human-annotated Open Assistant examples, pretrained LLaMA models at 7B, 33B, and 65B, and 502k ClueWeb segments as unlabelled data.Evaluation covers 1130 prompts from several instruction-following benchmarks and datasets.
- 3.2 DATA ANALYSIS: Augmented data complements seed data by increasing instruction diversity, especially in the long tail, while self-curated data has shorter instructions and outputs closer to seed data.The augmented content broadens topics and task types beyond those represented in the seed set.
- 3.3 SCALING ANALYSIS: Self-curation improves scaling: uncured augmented data does not improve instruction following, whereas higher-quality augmented data yields steady gains as its size increases.The results indicate that data quality matters more than simply increasing the quantity of noisy augmented examples.
- 3.3 SCALING ANALYSIS: Instruction backtranslation achieves higher instruction-following performance and more efficient data scaling than the compared instruction datasets, including when scaling A5 data.The comparison measures win rate against text-davinci-003 and summarizes efficiency with the scaling coefficient α.
- 3.4 MODEL QUALITY: Humpback is the top-performing non-distilled model at both 65B and 33B scales, and its zero-shot performance improves over the base model on several reasoning tasks.Additional evaluations include commonsense reasoning benchmarks and MMLU.
- 3.5 ABLATIONS: Combining self-curated augmentation with seed data produces large improvements, because the two sources provide complementary domain distribution and instruction diversity.Seed data matches the target AI-assistant response distribution, while web data broadens instructions and outputs.
4 RELATED WORK
Prior instruction-following systems depend heavily on costly human annotation or distillation, motivating scalable alternatives. This work instead combines model-generated instruction creation with model-based curation of human-written text without distilling from a stronger model.
- Human annotation is expensive, time-consuming, expertise-intensive, and difficult to scale across diverse instruction-following tasks.
- Prior approaches generate instructions and responses with language models, often using manually engineered filtering rules to remove low-quality pairs.
- Instruction backtranslation uses human-written web text as responses and generates corresponding instructions, making self-curation a critical distinction from similar work.
- Unlike concurrent work, this approach does not use an instruction-tuned stronger model to generate instructions, making it an instance of self-alignment.
- The work selects high-quality training data with the model itself rather than manually curating it or prompting ChatGPT to score it.
- Many distillation-based LLaMA approaches require an already strong model and do not provide a recipe for building one from scratch.
5 CONCLUSION
The paper concludes that instruction backtranslation is a scalable self-training approach for instruction following. Its models outperform other non-distilled instruction-following models while using fewer human-annotated examples.
- Instruction backtranslation uses iterative self-training to let a model augment and curate high-quality examples from large amounts of unlabeled data.
- The finetuned models outperform all other non-distilled instruction-following models on the Alpaca leaderboard while using fewer human-annotated examples.
- Future work should scale the method with larger unlabeled corpora, which the analysis suggests could yield further gains.
A LIMITATIONS
Because augmented data comes from a web corpus, the method may amplify web-data biases. Improved detection accuracy does not establish that the model generates fewer biased responses.
- Web-sourced augmented data could cause the finetuned model to amplify biases present in the web corpus.
- The model improves accuracy on CrowS-Pairs bias detection compared with the base model across nine evaluated categories.
- Bias-detection accuracy does not show that the model is less likely to generate responses containing biases.
A.2 SAFETY
The safety evaluation examines responses to potentially sensitive prompts, while the figures report instruction diversity and scaling behavior. The supplied passages do not state a safety outcome from the figures.
- Safety evaluation: The safety evaluation uses 30 potentially sensitive prompts because the training data lacks red-teaming examples and the finetuning objective does not target harm reduction.
- Safety evaluation: The model tends to respond cautiously or refuse information when evaluated on those sensitive prompts.
- Instruction diversity: Figure 6 compares instruction diversity in seed and augmented data, including common root verbs and noun objects from parsed instructions.
- Data scaling: Figure 7 reports improvement from scaling self-curated instruction data for both 7B and 65B LLaMA models, with neither saturated at 40,000 instructions.
B ADDITIONAL RESULTS
Additional experiments examine instruction diversity, scaling, general capabilities, seed-model improvements, and iterative self-curation. The results show gains from high-quality augmented data and improved selection across iterations, with a stated limitation on 5-shot MMLU performance.
- Jointly scaling of data and model: Adding high-quality augmented data A5 further improves the strong 65B seed-model baseline.
- MMLU: The finetuned model improves zero-shot MMLU accuracy across all domains but underperforms the base model with 5-shot in-context examples.
- Improvement over seed model: 16% of test prompts, or 41 out of 251, showed improved responses after adding self-augmented data.Improvements included reasoning, information seeking, and detailed advice.
- Data selection quality: Self-curation performance improves in the second iteration for selecting high-quality data, measured by precision and recall.Finetuning on selected data also produces better instruction following measured by win rate.
C GENERATION SAMPLES
The generation-sample section documents qualitative examples of generated instructions, improvements over the seed model, safety responses, and failure cases. The reported failure boundary concerns instructions requiring specific output formats such as ASCII art.
- Sample outputs: Examples cover generated instructions and improvements over the seed model in reasoning, information seeking, advice, and writing.
- Safety prompts: Tables provide examples of responses to sensitive and adversarial prompts.
- Failure cases: The method struggles with instructions requiring specific formats, such as ASCII art.The paper proposes larger-scale backtranslation or upsampling unlabelled-data distributions to improve long-tail coverage.
D HUMAN EVALUATION
Human evaluation uses screened Mturk annotators and pairwise preference judgments, with procedures for worker qualification, agreement testing, and data-selection assessment. The section also provides interfaces and generated-instruction examples.
- Worker screening: Mturk workers were selected through qualifications, screening tests, agreement with in-house annotators, and manual review of justifications.The screening process required agreement exceeding 85% with in-house annotators, treating ‘neither’ choices as half agreements.
- Data selection quality: Data-selection quality was evaluated using precision, recall, and win rate on a 250-example development set with expert quality labels.The win rate compares a 7B LLaMa model trained on selected data against text-davinci-003.
- Pairwise evaluation: The annotation study used 29 selected annotators to gather pairwise preferences through the displayed evaluation interface.
- Generated instructions: The backward model generated candidate instructions from unlabelled text for inclusion in the self-augmentation pipeline.
E MORE EXPERIMENT DETAILS
Further details describe corpus preprocessing, controlled data-scaling experiments, example tables, and the self-curation prompt. The procedures filter web segments and hold training conditions constant across data scales.
- Preprocessing: ClueWeb segments were filtered by length, sentence duplication, and header quality before sampling.Segments contained 600–3000 characters, removed repetitive sentences, and excluded low-quality headers or navigation text.
- Examples: The appendix includes qualitative examples of improvements, sensitive and adversarial responses, and prompts where Humpback fails.
- Training: Data-scaling experiments trained each dataset for the same number of steps and batch size across N ∈ {100, 800, 1600, 3200, 6400, 12800, 25600, 51200}.
- Training: The data-scaling experiments use the same 7B base LLaMa model, with lr=1e−5 linearly decaying to 9e−6.
- Self-curation: The self-curation prompt evaluates candidate instruction–output pairs derived from self-augmentation.