PaperlayerView exact revision on arXiv

Source-linked AI summary

Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, Hannaneh Hajishirzi

arXiv:2311.10702v2cs.CL

TL;DR

Open instruction-tuning resources have advanced rapidly, creating a need to evaluate and combine improved data, base models, and adaptation methods. TÜLU 2 builds a suite around these advances, including TÜLU-V2-mix, LLAMA-2 and CODE LLAMA finetuning, and DPO. The suite reports stronger benchmark performance, including state-of-the-art results among open-weight models on MT-Bench and competitive performance with GPT-3.5-turbo-0301, while identifying limitations in QLoRA, benchmark contamination, and mechanistic understanding.

  • Problem

    Rapid advances in open instruction-tuning resources motivate evaluating how improved data, base models, and adaptation methods can be combined for language-model adaptation.

  • Method

    The paper combines a new instruction-data mixture with LLAMA-2 and CODE LLAMA finetuning, DPO training, and QLoRA experiments across multiple model scales.

  • Results

    TÜLU 2 reports stronger downstream performance, DPO improves AlpacaEval by 13% on average across model sizes, and TÜLU 2 70B is competitive with GPT-3.5-turbo-0301 across several evaluations.

  • Takeaways & Limitations

    The released models, data, and code provide an open suite for studying and improving language-model adaptation recipes.

  • Takeaways & Limitations

    The mechanisms behind dataset and DPO improvements remain insufficiently understood, and comparisons with GPT models may be affected by possible benchmark-data contamination.

Abstract

from arXiv · show

Since the release of TÜLU [Wang et al., 2023b], open resources for instruction tuning have developed quickly, from better base models to new finetuning techniques. We test and incorporate a number of these advances into TÜLU, resulting in TÜLU 2, a suite of improved TÜLU models for advancing the understanding and best practices of adapting pretrained language models to downstream tasks and user preferences. Concretely, we release: (1) TÜLU-V2-mix, an improved collection of high-quality instruction datasets; (2) TÜLU 2, LLAMA-2 models finetuned on the V2 mixture; (3) TÜLU 2+DPO, TÜLU 2 models trained with direct preference optimization (DPO), including the largest DPO-trained model to date (TÜLU 2+DPO 70B); (4) CODE TÜLU 2, CODE LLAMA models finetuned on our V2 mix that outperform CODE LLAMA and its instruction-tuned variant, CODE LLAMA-Instruct. Our evaluation from multiple perspectives shows that the TÜLU 2 suite achieves state-of-the-art performance among open models and matches or exceeds the performance of GPT-3.5-turbo-0301 on several benchmarks. We release all the checkpoints, data, training and evaluation code to facilitate future open efforts on adapting large language models.

1 Introduction

TÜLU 2 combines advances in instruction data, base models, and adaptation methods into an open suite evaluated across diverse tasks. Its results include stronger data mixtures, scalable DPO, QLoRA trade-offs, and improved code models.

  • 1 Introduction: TÜLU 2 combines improved instruction data, newer base models, and adaptation methods across 7B, 13B, and 70B scales.The work comprehensively evaluates training recipes for these model sizes.
  • 1 Introduction: TÜLU-V2-mix improves performance across reasoning and knowledge-probing tasks, while TÜLU 2+DPO 70B is competitive with state-of-the-art on MT-Bench and AlpacaEval.The 70B model is trained with direct preference optimization after finetuning on the new mixture.
  • 1 Introduction: QLoRA remains solid on traditional language-processing tasks but trails full finetuning on long-form generation, especially AlpacaEval.The reported average gap is 20% on open-ended generation tasks such as AlpacaEval.
  • 1 Introduction: CODE TÜLU 2 outperforms CODE LLAMA and CODE LLAMA-Instruct across all model sizes.The broader summary reports a 70% average Codex-Eval improvement over TÜLU 2, alongside a 20% AlpacaEval drop.
  • 1 Introduction: 13% average AlpacaEval improvement across model sizes is reported for DPO, while capabilities remain without degradation.The summary also reports state-of-the-art MT-Bench performance for TÜLU 2+DPO 70B among open-weight models.
  • 1 Introduction: The authors publicly release the models, data, training code, and evaluation code to support future research on language-model adaptation.The release is intended to aid future open efforts.

2 TÜLU V2 Details

TÜLU 2 updates the adaptation recipe with LLAMA-2 and CODE LLAMA bases, a curated V2 mixture, longer contexts, DPO, and QLoRA experiments. The mixture emphasizes quality, diversity, complexity, reasoning, coding, and longer examples.

  • 2 TÜLU V2 Details: TÜLU 2 compares new base models, data mixtures, extended-context training, and RLHF methods against TÜLU 1.The adaptation study spans multiple components of the training recipe.
  • 2 TÜLU V2 Details: LLAMA-2 replaces LLAMA-1, and experiments cover 7B, 13B, and 70B LLAMA-2 models plus 7B, 13B, and 34B CODE LLAMA models.LLAMA-2 was pretrained on 2 trillion tokens, compared with 1 or 1.4 trillion for LLAMA-1.
  • V2 data mixture: TÜLU-V2-mix retains high-quality data, adds curated or GPT-generated datasets, and downsamples larger sources such as FLAN.The mixture includes reasoning, coding, conversational, scientific, and distilled data sources.
  • V2 data mixture: After filtering references to other LLM systems, the V2 mixture contains 326,154 samples versus 490,445 in V1.The filtering avoids contradictions with hardcoded prompts.
  • Extended context length: Training context expands from 2,048 to 8,192 tokens, reducing truncated V2 samples from 63,900 to 20.The V2 sample length has a mean of 1097 tokens, with 230 and 1464 tokens at the 25th and 75th percentiles.
  • RLHF training: DPO uses filtered, binarized UltraFeedback for three epochs and a low learning rate of 5 × 10^-7.The method is chosen for implementation simplicity and improves open-ended-generation evaluations.
  • QLoRA training: QLoRA is tested to reduce compute demands, but its sub-par instruction-tuning performance prevents exploration during RLHF training.Prior work is noted as finding QLoRA effective for PPO-based RLHF training.

3 Experiments

TÜLU 2 is evaluated across factual knowledge, reasoning, multilinguality, coding, open-ended generation, toxicity, and truthfulness. Results show strong gains from the V2 mixture and DPO, while QLoRA and code-specialized pretraining create task-dependent trade-offs.

  • 3.1 Overall Results: TÜLU 2 outperforms all open models on average, with TÜLU 2 70B the best open model in 3/7 tasks.In the remaining four tasks, TÜLU 2 70B trails the task leaders by an average gap under 1%.
  • 3.1 Overall Results: TÜLU 2 70B matches GPT-3.5-turbo-0301 on MMLU, BBH, and TydiQA and outperforms it on AlpacaEval and ToxiGen.A large gap remains with GPT-4 and a moderate gap with GPT-3.5-turbo-0613 in most evaluations.
  • 3.2 V2 Data Mixture: The V2 mixture improves open-ended generation and most overall evaluations over V1 and ShareGPT, but underperforms V1 on GSM8k and TydiQA.The average improvement over V1 is 13% at 7B but only 1% at 70B, indicating smaller V2 gains at larger scales.
  • 3.3 Scaling DPO Training: DPO significantly improves AlpacaEval and MT-Bench, scales stably to 70B, and does not significantly change most factual or reasoning metrics.TÜLU 2+DPO 70B is the best-performing open model on the MT-Bench leaderboard and the second-best open model on AlpacaEval.
  • 3.3 Scaling DPO Training: DPO-trained models generally produce longer AlpacaEval answers, consistent with reported verbosity biases from RLHF training.The authors note that their DPO-trained models remain dramatically less verbose than other open-weight models.
  • 3.4 Parameter-efficient Finetuning: QLoRA consistently underperforms full-finetuning on AlpacaEval, although the overall performance gap shrinks with model size.The authors observe much closer QLoRA and full-finetuning performance on MMLU.
  • 3.5 Improving Code Performance with CODE LLAMA: CODE TÜLU 2 significantly improves coding performance, with its 7B model matching TÜLU-V2+DPO 70B on Codex-Eval.Across non-code tasks, TÜLU 2 and CODE TÜLU 2 show different strengths, and AlpacaEval performance drops by around 20% for CODE TÜLU 2.
  • 3.5 Improving Code Performance with CODE LLAMA: CODE TÜLU 2 outperforms both base CODE LLAMA and CODE LLAMA-Instruct in 5 of 8 evaluation settings and is stronger on average.The models use the V2 data mixture, whereas CODE LLAMA-Instruct was trained on an inaccessible private dataset.

4 Conclusion

TÜLU 2 advances open language-model fine-tuning through improved data, stable DPO training, and comparisons with parameter-efficient methods. Important mechanisms behind these gains remain insufficiently understood, motivating further investigation.

  • TÜLU 2 provides models and recipes for continuing progress in fine-tuning language models across varied tasks.
  • Its improvements include a stronger data mixture, more stable DPO training, and comparisons with parameter-efficient training methods.
  • The mechanisms behind gains from the datasets and DPO methodology remain insufficiently understood.
  • Future work should examine refusal behavior, data ablations, large-scale comparisons with PPO, and improved base models.

A Evaluation Suite

The evaluation suite measures language-model performance across instruction following, reasoning, multilingual question answering, coding, toxicity, truthfulness, and open-ended generation. It combines established benchmarks with task-specific prompting, decoding, and scoring procedures.

  • MMLU: MMLU uses the official script with zero-shot prompts and reports average accuracy across test examples.
  • GSM: GSM uses chain-of-thought prompting with eight in-context examples and extracts the final number for average accuracy.
  • BBH: BBH follows the original chain-of-thought setup with three examples, extracting answers after “So the answer is” and averaging subtask accuracy.
  • TydiQA: TydiQA evaluates multilingual answering in the GoldP setting, where the gold passage is provided alongside one formatting example.
  • Codex-Eval: Codex-Eval measures coding ability on 164 HumanEval problems using unbiased pass@10 estimates with temperature 0.8 sampling.
  • ToxiGen, TruthfulQA, and AlpacaEval assess toxic generation, informative truthfulness, and GPT-4-preferred open-ended responses, respectively.AlpacaEval compares generations on 805 prompts against Davinci-003.
  • MT-Bench: MT-Bench evaluates 160 responses to 80 questions with follow-ups, graded by GPT-4-0613 across varied domains.

B Training Hyperparameters

The training setup specifies separate supervised fine-tuning, QLoRA, and DPO configurations. QLoRA hyperparameters were selected in smaller-scale experiments under compute constraints, while models used different hardware depending on the training method.

  • Instruction-tuning/supervised fine-tuning: Supervised fine-tuning uses BFloat16, two epochs, learning rate 2e-5 or 1e-5 for 70B, sequence length 8,192, and effective batch size 128.
  • QLoRA: QLoRA uses five epochs, learning rate 1e-4, sequence length 4,096, effective batch size 128, rank 64, alpha 16, and dropout 0.1.
  • QLoRA: The selected QLoRA settings were the strongest configurations found in smaller-scale experiments that fit the compute budget.
  • DPO: DPO uses BFloat16, three epochs, learning rate 5e-7, sequence length 8,192, effective batch size 32, and beta 0.1.
  • Hardware: Non-QLoRA models were trained on TPU v3 pods, with 512 chips used for 70B DPO training.
  • Hardware: QLoRA models were trained on an internal A100 80GB cluster using the open-instruct fine-tuning code.

C Science Mixture Breakdown

The science literature instruction mixture is documented through a dataset breakdown in Table 7. The table identifies the datasets included in TÜLU V2’s science-literature component.

  • Table 7 breaks down the datasets included in the science literature instruction mix for TÜLU V2.

D Full MT-Bench Results

Table 8 reports full MT-Bench results by category for the models shown in Table 4, using GPT-4 as the judging model. Scores are averaged from GPT-4 annotator ratings, with the best open-weight performance underlined.

  • Table 8 presents full MT-Bench results split by category for all models shown in Table 4.
  • GPT-4-0613 serves as the judge model for the MT-Bench evaluation.
  • Scores are averages of ratings from a GPT-4 annotator, with the best open-weight model performance underlined.
Loading 2311.10702v2…