PaperlayerView exact revision on arXiv

Source-linked AI summary

How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, Hannaneh Hajishirzi

arXiv:2306.04751v2cs.CL

TL;DR

Open instruction-tuned models are difficult to compare because prior claims often rely on limited tasks and model-based evaluation. The paper systematically evaluates models trained on diverse public datasets across capabilities and evaluation types, finding dataset-specific strengths, strong effects from base-model quality, and persistent gaps with proprietary models. It concludes that comprehensive evaluation and continued development of base models and diverse instruction data remain necessary.

  • Problem

    Existing comparisons of open instruction-tuned models often cover few tasks and rely mainly on model-based metrics, limiting systematic assessment across capabilities.

  • Method

    The paper evaluates models from 6.7B to 65B trained on public instruction datasets across factual knowledge, reasoning, multilinguality, coding, safety, and open-ended following using automatic, model-based, and human metrics.

  • Results

    Different datasets improve specific skills, no dataset is best across all evaluations, stronger base models perform better, and preference-based winners differ from benchmark-based winners.

  • Takeaways & Limitations

    Comprehensive evaluation, stronger base models, and broader diverse datasets are needed because the strongest open models do not yet match ChatGPT or GPT-4.

  • Takeaways & Limitations

    Model preference evaluation is not holistic and should be used only as part of a larger comprehensive evaluation setup.

Abstract

from arXiv · show

In this work we explore recent advances in instruction-tuning language models on a range of open instruction-following datasets. Despite recent claims that open models can be on par with state-of-the-art proprietary models, these claims are often accompanied by limited evaluation, making it difficult to compare models across the board and determine the utility of various resources. We provide a large set of instruction-tuned models from 6.7B to 65B parameters in size, trained on 12 instruction datasets ranging from manually curated (e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and systematically evaluate them on their factual knowledge, reasoning, multilinguality, coding, and open-ended instruction following abilities through a collection of automatic, model-based, and human-based metrics. We further introduce Tülu, our best performing instruction-tuned model suite finetuned on a combination of high-quality open resources. Our experiments show that different instruction-tuning datasets can uncover or enhance specific skills, while no single dataset (or combination) provides the best performance across all evaluations. Interestingly, we find that model and human preference-based evaluations fail to reflect differences in model capabilities exposed by benchmark-based evaluations, suggesting the need for the type of systemic evaluation performed in this work. Our evaluations show that the best model in any given evaluation reaches on average 87% of ChatGPT performance, and 73% of GPT-4 performance, suggesting that further investment in building better base models and instruction-tuning data is required to close the gap. We release our instruction-tuned models, including a fully finetuned 65B Tülu, along with our code, data, and evaluation framework at https://github.com/allenai/open-instruct to facilitate future research.

1 Introduction

The paper argues that open instruction-tuned models need systematic, multifaceted evaluation because existing comparisons cover limited tasks and rely heavily on model-based metrics. It evaluates diverse resources and introduces TÜLU, finding dataset- and capability-specific gains but persistent gaps with proprietary models.

  • Evaluation scope: The study evaluates instruction-tuned models across factual knowledge, reasoning, multilinguality, coding, safety, and open-ended instruction following.It spans models from 6.7B to 65B parameters and uses automatic, model-based, and human-based metrics.
  • Key findings: Different instruction datasets promote specific skills, while no single dataset performs best across all evaluations.The results indicate that dataset choice exposes or enhances different capabilities.
  • Key findings: TÜLU combines existing instruction datasets and achieves the best average performance across benchmarks, although it is not best in every evaluation setting.The suite includes fine-tuned LLaMA models based on a combination of open resources.
  • Key findings: Even 65B TÜLU does not outperform ChatGPT, though it substantially exceeds similar smaller models.This result leaves a performance gap between the strongest open models and proprietary systems.
  • Evaluation implications: Model-based preference scores can favor longer, more diverse generations and may hide capability differences revealed by benchmark evaluations.The paper therefore compares model-based and benchmark-based assessments rather than relying on preference evaluation alone.

2 Background: Instruction Tuning and Resources

Instruction tuning adapts pretrained language models to natural-language requests, and its success depends on both the base model and the instruction data. The paper surveys diverse public datasets and pretrained-model families as the resources for its comparisons.

  • 2.1 Instruction Tuning: Instruction tuning finetunes pretrained language models to understand and respond to varied natural-language requests.This work focuses on supervised learning from demonstrations rather than reinforcement learning from feedback.
  • 2.1 Instruction Tuning: Effective instruction tuning requires a powerful pretrained model and an instruction dataset that is diverse and representative of downstream use.The paper studies both factors.
  • 2.2 Instruction Datasets: The dataset inventory reports conversation turns and average prompt and completion lengths, while sampling CoT and FLAN V2 to 100K examples.These statistics characterize the investigated instruction resources.
  • 2.2 Instruction Datasets: The studied datasets include researcher-constructed benchmarks, human-authored data, proprietary-model generations, user-shared prompts, and skill-specific resources.Examples include SuperNI, FLAN V2, Dolly, Open Assistant, Self-Instruct, Alpaca, Baize, GPT4-Alpaca, ShareGPT, and CoT.
  • 2.3 Pretrained Models: The pretrained-model table lists the base models finetuned in the study.The table supports comparisons among the model families used for instruction tuning.
  • 2.3 Pretrained Models: The experiments primarily use LLaMA models ranging from 6.7B to 65B parameters and also consider OPT and Pythia.LLaMA-2 models were added in the camera-ready version after initial LLaMA-1 experiments.

3 Training Models with Various Datasets

The training pipeline unifies varied instruction datasets into a chatbot format, encodes conversations as single sequences, and applies loss masking to assistant outputs. It also constructs human-only and Human+GPT mixtures to study dataset diversity.

  • 3.1 Unifying the Format: All datasets are converted to a chatbot-style schema with user, assistant, and end-of-text markers.This format supports arbitrary interaction rounds in one causal-language-model input sequence.
  • 3.1 Unifying the Format: ShareGPT conversations are reproduced from a public version and split into blocks of at most 2048 tokens.The exact original ShareGPT dataset was not released.
  • 3.2 Model Training Details: Decoder-only models use teacher forcing with loss masking, masking input tokens while training on target outputs.The target outputs correspond to desired responses following user prompts and prior conversation turns.
  • 3.2 Model Training Details: The entire conversation is encoded together, while training loss is computed only on assistant parts between role boundaries.Figure 1 illustrates assistant tokens as the supervised targets.
  • 3.3 Dataset Mixtures: The paper creates dataset mixtures because prior work links greater instruction diversity with improved instruction-tuning performance.It forms a human-authored mixture and a Human+GPT mixture.
  • 3.3 Dataset Mixtures: The human mixture combines FLAN V2, CoT, Dolly, and Open Assistant 1, excluding SuperNI because FLAN V2 covers most of its tasks.This mixture is intended to represent strong human-authored resources.
  • 3.3 Dataset Mixtures: The Human+GPT mixture adds GPT4-Alpaca, Code-Alpaca, and ShareGPT to the human mixture.Both mixtures concatenate datasets, while more complex sampling strategies are left for future work.
  • 3.3 Dataset Mixtures: LLaMA models trained on the Human+GPT mixture are named TÜLU, with LLAMA-2-based versions designated TÜLU-1.1.The name refers to a hybrid camel produced by interbreeding different species.

4 Evaluation Setup

The evaluation suite measures core capabilities and open-ended instruction following using benchmark-based, model-based, and human-based methods. It covers factual knowledge, reasoning, multilinguality, coding, safety, and diverse user instructions.

  • Core capabilities: MMLU measures factual knowledge across 57 subjects using multiple-choice questions spanning elementary to professional difficulty.Its format probes knowledge without the open-endedness of generated responses.
  • Core capabilities: GSM and Big-Bench-Hard evaluate mathematical and general reasoning through a math test split and 23 challenging tasks.The two benchmarks target complementary forms of reasoning ability.
  • Core capabilities: TyDiQA tests multilingual question answering across 11 typologically diverse languages in a gold-passage setting.The setup provides a passage containing the reference answer.
  • Core capabilities: Codex-Eval uses HumanEval to assess whether models generate functionally correct programs from docstrings.The paper uses Codex-Eval to distinguish this coding benchmark from its human evaluation.
  • Open-ended and safety evaluation: The suite also evaluates safety and open-ended instruction following, combining benchmarks for toxicity and truthfulness with diverse user requests.Open-ended evaluation addresses requests that specific-skill benchmarks may not capture.

5 Results

Instruction-tuning outcomes depend on both dataset specialization and base-model quality: different resources improve different capabilities, while mixtures perform best on average. However, preference-based evaluations can diverge from benchmark and human assessments, and open models still lag proprietary systems.

  • Dataset and base-model effects: Different instruction datasets enable different capabilities, with CoT especially helpful for GSM and Code-Alpaca helpful for Codex-Eval.No single dataset is best across all tasks.
  • Dataset and base-model effects: Combining datasets gives the best average benchmark performance, despite being best in only 2 of 6 individual evaluation settings.The result motivates better dataset mixing or modular instruction-tuning models.
  • Dataset and base-model effects: Better base models perform better across evaluation settings, with LLAMA outperforming comparable OPT and Pythia models.The authors associate this advantage with substantially more pretraining tokens and observe further gains from upgrading to LLAMA-2.
  • Dataset and base-model effects: Most evaluated datasets degrade vanilla performance on GSM and TydiQA, consistent with possible forgetting of chain-of-thought or multilingual abilities.The authors hypothesize that limited reasoning-style and multilingual training data contribute to this degradation.
  • Scaling and proprietary comparisons: Instruction tuning improves LLAMA models at every size, but relative gains are largest for smaller models.The 65B base LLAMA is comparable to or better than 65B TÜLU on MMLU, BBH, and TydiQA.
  • Scaling and proprietary comparisons: TÜLU 65B remains behind ChatGPT and GPT-4 in all evaluation settings, despite outperforming similar smaller models.The authors cannot exclude contamination of the evaluation suite in proprietary-model training, but the gap also appears in model- and human-based evaluations.
  • Risks and open-ended evaluation: GPT-distilled datasets yield strong risk-evaluation results, while TruthfulQA does not improve with model size.Larger models produce more correct facts but also hedge or refuse more often, leaving overall TruthfulQA performance largely unchanged.
  • Risks and open-ended evaluation: CoT, FLAN, and SuperNI improve benchmark capabilities yet perform extremely poorly in open-ended instruction following.By contrast, datasets encouraging long, diverse generations perform best; AlpacaEval win-rate correlates with unique-token count at Pearson r=0.96.

6 Related Work

Prior work has expanded instruction tuning and language-model evaluation, but existing frameworks often emphasize base-model benchmarks or omit open-ended instruction following. This work combines benchmark, model-based, and human-based evaluation.

  • Instruction tuning: Instruction tuning fine-tunes pretrained language models to follow diverse natural-language requests and supports zero-shot generalization.Prior studies emphasize diverse prompts, task coverage, and data diversity.
  • Language-model evaluation: Broad evaluation frameworks such as HELM and LM Evaluation Harness often focus on base models rather than instruction-tuned models.Related work also evaluates factuality and reasoning while largely neglecting open-ended instruction following.
  • Language-model evaluation: AlpacaEval and Chatbot Arena extend evaluation to open-ended instruction following using model or human annotators, respectively.The paper combines traditional benchmarks with both model-based and human-based evaluation.

7 Conclusion

The paper evaluates publicly available instruction-tuning resources against proprietary models and finds that strong base models, dataset combinations, and broader evaluations remain important.

  • Strong base models are vital, while combining datasets works best on average but can slightly reduce performance on specific tasks.
  • The strongest open models do not yet match ChatGPT or GPT-4.
  • The evaluation highlights the need for continued development of stronger base models and more diverse instruction-tuning datasets.

A Limitations

The evaluation is broad but not exhaustive: it omits some capabilities, datasets, and recently released models, while open-ended comparisons remain subjective and reference-dependent.

  • The study does not explicitly evaluate multi-turn dialogue or summarization abilities.
  • Computational cost prevented exhaustive coverage of all recently released instruction datasets and open models.
  • Open-ended instruction evaluation is subjective because annotators can hold different biases and preferences.
  • Model-based comparisons against Davinci-003 may reward models that avoid its shortcomings or fail to reward models sharing its strengths.
  • The released large instruction-tuned models initially used gated access because releasing them without well-tested guides carries some risk.

D Model Training Details and Compute

All models were fully fine-tuned for two epochs using scheduled learning rates, sequence-length limits, and large-scale optimization infrastructure.

  • All models were fully fine-tuned for two epochs with linear learning-rate decay and 3% linear warmup.The learning rate was 2e-5, reduced to 1e-5 for 30B and 65B models, with no weight decay.
  • Training used a maximum sequence length of 2048 tokens, reduced to 1024 for 30B and 65B models, truncating longer samples.
  • DeepSpeed and the ZeRO optimizer enabled large-scale model fine-tuning.

E Evaluation Setups

The evaluation combines benchmark-specific protocols for knowledge, reasoning, multilingual question answering, coding, and open-ended instruction following, with standardized inference settings where specified.

  • GSM and BBH are evaluated with and without chain-of-thought using their specified few-shot prompts and answer-extraction procedures.
  • TydiQA tests multilingual question answering with either a provided gold passage or no context.
  • Codex-Eval uses HumanEval for coding assessment, while open-ended win rate measures GPT-4 preferences against Davinci-003 generations.
  • Models generally use 8-bit loading, greedy decoding, and a maximum generation length of 512 tokens unless otherwise specified.

F Overview of All Automatic Evaluation Results

The paper combines automatic benchmark results with safety, human, and model-based evaluations across instruction-tuned models. These evaluations expose differences in capability, annotator subjectivity, and possible biases in GPT-4-based preference judgments.

  • Core benchmark evaluation: Table 8 compiles performance across core capability benchmarks, averaging scenarios within each benchmark and weighting benchmarks equally.The overview includes instruction-tuned and proprietary models across selected benchmarks.
  • Safety evaluation: Safety evaluation reports ToxiGen toxicity rates by targeted group and TruthfulQA percentages for informative and truthful answers.These results are presented separately across models in Tables 9 and 10.
  • Human evaluation: Human evaluation compares three model pairs on 332 instructions using anonymized, randomized outputs and acceptance and helpfulness judgments.The study uses 18 expert annotators, with agreement measured on a 119-example subset.
  • Human evaluation: 0.84 annotator agreement was achieved for output acceptance, while pairwise comparisons used tie-discounted accuracy after merging two better-than categories.The authors characterize human evaluation as substantially subjective despite reasonable agreement.
  • Model-based preference evaluation: R^2=.91 indicates that unique-token count explains a large proportion of GPT-4 evaluator win-rate variance, although the dummy evaluator generally overestimates win rates.The dummy evaluator awards wins to outputs containing more unique tokens.
Loading 2306.04751v2…