How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources

1 Introduction

• The authors argue evaluation should test core reasoning and fact recall alongside more subjective model- or human-annotated generation quality, since many comparability claims rest on few tasks and model-based metrics.Source: §1 · pp. 1–2
• The evaluation spans a dozen public instruction-tuning corpora and models from 6.7B to 65B, covering factual knowledge, reasoning, multilinguality, coding, safety, and open-ended instruction following.Results use automatic, model-based, and human-based metrics.Source: §1 · p. 2
• Instruction tuning on different datasets promotes specific skills, no dataset performs best across all evaluations, and stronger or longer-pretrained base models perform better across the board.The authors also find that domain- or capability-targeted datasets can be especially effective in their targeted areas.Source: §1 · p. 2
• Model-based evaluation’s top performers differ from benchmark-based automatic evaluation’s, and GPT-4 may favor long, diverse generations.Model-based preference scores also correlate strongly with the number of unique tokens generated, suggesting this bias can obscure capability differences.Source: §1 · p. 2
• TÜLU, a 7B–65B LLAMA suite trained on seven available datasets, achieves the best average performance across most model sizes while remaining within 29% of the best-performing model on each task.TÜLU 65B was, to the authors’ knowledge, the largest publicly released fully instruction-tuned LLAMA variant at the time of writing; it is not the overall best across evaluation settings considered independently.Source: §1 · p. 2
• Even TÜLU 65B fails to outperform ChatGPT, although it performs significantly better than similar smaller models.Source: §1 · p. 2

2.1 Instruction Tuning

• Instruction tuning adapts pretrained language models to understand and respond to natural-language requests that specify a task, supporting generalization to new scenarios and interaction by non-experts.Training approaches include supervised demonstrations and reinforcement learning from feedback; this work focuses on supervised learning because open resources for the RL-based approach remain rare.Source: §2.1 · p. 2
• Successful instruction tuning requires both a powerful pretrained model with broad web-scale knowledge and a diverse, representative dataset suited to potential downstream uses.The study examines both factors and introduces the open resources used in its experiments.Source: §2.1 · p. 2

2.2 Instruction Datasets

• Table 1 covers the investigated instruction datasets; CoT and FLAN V2 are sampled to 100K, and the table reports average turns and prompt and completion lengths.The caption labels the average-turn measure as average number of conservation turns.Source: Table 1 · p. 3
• The dataset sample spans repurposed NLP tasks (SuperNI, Flan V2 [31]), human-written sets (Dolly, Open Assistant 1 [26]), and proprietary-model generations including Alpaca.It also includes model-generated completions for user-shared prompts (ShareGPT3) and skill-focused resources for chain-of-thought [50] and code generation.Source: §2.2 · p. 3

2.3 Pretrained Models

• The study primarily uses LLAMA models [44] [45] from 6.7B to 65B parameters, adding LLAMA-2 in the camera-ready version; these models use similar parameter counts but more training tokens.The authors describe LLAMA as the largest, highest-quality pretrained models available to the community at the time, while noting restrictive licensing.Source: §2.3 · p. 3
• OPT [54] and Pythia [4] models comparable in size to LLAMA 6.7B are included to examine the effect of different base models.The authors expect their findings to hold for similar- or better-quality open models and future stronger base models.Source: §2.3 · p. 3

3.1 Unifying the Format

• All datasets are formatted as chatbot-style sequences with <|user|> and <|assistant|> markers and </s> endings, so multi-round interactions fit one causal-LM input.For ShareGPT, the authors use a reproduced dataset and split long conversations into blocks of at most 2048 tokens.Source: §3.1 · p. 3

3.2 Model Training Details

• Decoder-only models use teacher forcing and loss masking: input tokens are masked, and loss is computed only on assistant tokens between <|assistant|> and the next <|user|>.This lets conversation training predict assistant turns given preceding user prompts and conversation context.Source: §3.2 · p. 4, Figure 1 · p. 4

3.3 TÜLU: a Better Instruction-Tuned Model by Combining Resources

• The Human data mixture combines FLAN V2, CoT, Dolly, and Open Assistant 1, excluding SuperNI because FLAN V2 includes most of its tasks.The authors motivate dataset diversity as a way to improve instruction tuning.Source: §3.3 · p. 4
• The Human+GPT mixture adds GPT4-Alpaca, Code-Alpaca, and ShareGPT to the human mixture; concatenated-mixture LLaMA models are named TÜLU, with LLaMA-2-based versions designated TÜLU-1.1.The authors leave more complex sampling mixtures for future work.Source: §3.3 · p. 4

4 Evaluation Setup

• The authors compile prior evaluation approaches [9] [2] [47] [16] into a systematic, multi-faceted assessment of core abilities and open-ended instruction following.Source: §4 · p. 4

4.1 Facets of Evaluation

• MMLU [22] probes factual knowledge across 57 subjects, while GSM [11] tests mathematical reasoning and BBH [42] evaluates general reasoning across 23 challenging tasks.Source: §4.1 · pp. 4–5
• TyDiQA [10] tests non-English question answering across 11 typologically diverse languages using a gold passage; HumanEval [7], called Codex-Eval, tests functionally correct programs from docstrings.Source: §4.1 · p. 5
• Open-ended instructability is assessed with model-based and human evaluations, while ToxiGen and TruthfulQA [30] assess toxic generation and avoidance of known falsehoods.ToxiGen is used following LLAMA-2 [45].Source: §4.1 · p. 5

4.2 Model-Based Evaluation using GPT-4

• GPT-4 judged win rates against Davinci-003 on 805 instructions using AlpacaEval code and prompts, with randomized output order; vanilla LLAMA was excluded.The instructions combine Self-Instruct [47], Open Assistant [26], Anthropic helpful [3], Vicuna, and Koala evaluation sets.Source: §4.2 · p. 5

4.3 Human Evaluation

• Human evaluation used 332 instructions from Self-Instruct [47] and Vicuna [8], and assessed fine-tuned LLAMA-1 models because LLAMA-2 was unavailable at the time.Source: §4.3 · p. 5
• Raters judged individual responses acceptable only if they answered the request without significant errors or repetition, and compared pairs using a five-way helpfulness preference.Source: §4.3 · pp. 5–6
• The human judgments came from 18 expert annotators—AI2 researchers or UW students—all fluent English speakers with at least bachelor’s degrees.Source: §4.3 · p. 6

5.1 Analysis of Instruction Tuning Datasets and Base Models

• Task-specific data helped distinct abilities: chain-of-thought training benefited GSM mathematical reasoning, while Code-Alpaca helped Codex-Eval.The authors suggest task-specific datasets or distilled task-specific data may help ensure those tasks are well represented in training.Source: §5.1 · p. 6
• Dataset combinations achieved the best average benchmark performance, although they were best in only two of six individual evaluation settings.The authors identify improved dataset mixing or modular models such as mixture-of-experts as promising directions.Source: §5.1 · pp. 6–7

5.2 Pushing the Limits of Open Models

• Instruction tuning improved LLaMA models at all sizes, but relative gains shrank with scale; TÜLU 65B still lagged behind ChatGPT and GPT-4 across evaluation settings.The 65B base model was comparable or better than TÜLU 65B on MMLU, BBH, and TydiQA.Source: Table 5 · p. 7, §5.2 · pp. 7–8

5.1 Analysis of Instruction Tuning Datasets and Base Models

• LLaMA base models significantly outperformed comparable-sized OPT and Pythia models on the Human+GPT mix, and a later LLaMA-2 upgrade also improved performance.The authors associate LLaMA’s advantage with more pretraining tokens and suggest larger or higher-quality pretraining corpora are preferable.Source: §5.1 · p. 7
• Most evaluated datasets degraded performance on GSM and TydiQA versus vanilla models, while Self-Instruct degraded performance across most tasks.The authors hypothesize that missing chain-of-thought and multilingual data may cause forgetting, and that Self-Instruct’s weaker-model-generated data may be lower quality.Source: §5.1 · p. 7

5.3 Evaluation of Potential Risks and Harms

• GPT-distilled datasets yielded the best overall risk-benchmark performance, but larger models nearly eliminated toxic generations and TruthfulQA performance did not improve with size.On TruthfulQA, larger models produced more correct facts but also hedged or refused informative answers more often.Source: §5.3 · p. 8

5.4 Model-Based Evaluation Results for Open-Ended Generation

• ShareGPT performed best across model sizes, including outperforming TÜLU models more than twice its size; AlpacaEval win-rates also correlated with unique response-token counts (Pearson r=0.96).CoT, FLAN, and SuperNI performed poorly in open-ended instruction following despite improving capabilities assessed in Table 3.Source: §5.4 · pp. 8–9, Table 7 · p. 9, Figure 2 · p. 9

5.5 Human Evaluation Results for Open-Ended Generation

• Human evaluations generally favored larger TÜLU (65B over 7B) and distilled datasets, yet 7B beat human-mix TÜLU 65B in model preference while 65B was more likely to yield acceptable responses.The authors caution that model pairwise evaluation may not always reveal model deficiencies.Source: §5.5 · p. 9

Instruction Tuning of LMs

• Instruction tuning improves zero-shot performance on unseen tasks [39] [51] [49] [32] [9] and supports supervised finetuning [31]; prompt, task, and data diversity also matter [39] [9] [56].Prior studies vary the number of diverse prompts, the number of tasks, and data diversity as performance factors.Source: §6 · p. 10
• Models increasingly use instruction-augmented data generated or collected from larger proprietary models, including ChatGPT and GPT-4 [47, 23, 25, 53, 8, 15, 43, 52, 36].Source: §6 · p. 10
• Evaluation setups vary across models; prior dataset-mixture studies [31] [24] often test fewer instruction sources and focus on benchmarks, while QLoRA [14] covers fewer models, datasets, and evaluations.Source: §6 · p. 10

Evaluation of LMs

• HELM [28] and LM Evaluation Harness cover broad NLP tasks but often target base models, while Chung et al. [9] assess factuality and reasoning but largely omit open-ended instruction following.Source: §6 · p. 10
• GPT-4 [34] and PaLM v2 [2] releases include broad benchmark evaluations but omit open-ended instruction following, and closed data releases prevent testing for evaluation-data contamination.Source: §6 · p. 10
• AlpacaEval uses model annotators and Chatbot Arena uses humans; this work evaluates traditional benchmarks, model-based judgments, and human-based judgments.Gudibande et al. [20] argue that models trained on GPT outputs mimic style rather than content; this work finds performance varies across datasets and imitation data may improve performance when diverse in skills and domains.Source: §6 · p. 10

• Strong base models are vital, and combining datasets works best on average, though it slightly reduces task-specific peak performance; the strongest open models still trail ChatGPT and GPT-4.Source: §7 · p. 10
• The work extensively evaluates publicly available instruction-tuning resources against the strongest proprietary models then available.Source: §7 · p. 10
• The authors call for continued development of strong base models and broader, diverse datasets, and hope their released code and models spur research to close the gaps.Source: §7 · p. 10

A Limitations

• The evaluations omit multi-turn dialogue and summarization, while open-ended judgments are subjective and model-based comparisons to Davinci-003 may favor its shortcomings or underreward shared strengths.The paper notes that different annotators may have different biases and preferences, and that broad open-ended tasks are covered through model- and human-preference evaluations.Source: A Limitations · p. 15
• The study does not cover all recent datasets or models because of computational cost; it focuses on representative open datasets and the strongest widely available base model, leaving newer models such as Falcon for future work.The authors argue that their broad model coverage still helps identify which open resources work and where future efforts should focus: better base models and more diverse instruction-tuning datasets.Source: A Limitations · p. 15

B Broader Impact

• Widely available resources may pose fairly small harms, but releasing large instruction-tuned models without tested guides carries risk; the largest models were initially gated for access approval.Training and releasing the models through a gated setup required users to apply and be manually approved.Source: B Broader Impact · p. 15

C Instruction Datasets Details

• The datasets span instructed NLP and chain-of-thought collections, augmented task mixtures, employee-created instructions, crowdsourced multilingual conversations, and GPT-3-generated examples.These include SuperNI, CoT [50] from FLAN v2 [9], Flan V2 [9], Dolly, Open Assistant 1 [26], and Self-Instruct [47].Source: C Instruction Datasets Details · p. 15
• Unnatural Instructions uses Davinci-002 prompting [23], Alpaca uses a self-instruct-style method with Davinci-003, and Code-Alpaca applies the Alpaca method to code generation.GPT-4 Alpaca replaces Alpaca’s example generations with GPT-4 outputs [36], while Baize has ChatGPT converse with itself [52].Source: C Instruction Datasets Details · p. 16
• ShareGPT uses publicly shared chat interactions, split into chunks of at most 2048 tokens following the Vicuna setup, without further sample filtering.The authors use the html-cleaned variant and release it under Apache-2.0.Source: C Instruction Datasets Details · p. 16
• SuperNI and CoT account for only a small portion of the subsampled FLAN V2 data, and the datasets were not screened for offensive content, so models may produce toxic or harmful generations.The authors broadly use publicly available instruction-tuning datasets; human-authored datasets were largely created explicitly for model training with participant knowledge.Source: C Instruction Datasets Details · p. 16

D Model Training Details and Compute

• Training used two epochs, learning rates of 2𝑒−5 (1𝑒−5 for 30B and 65B models), no weight decay, and linear decay with 3% warmup.Models used a maximum sequence length of 2048, reduced to 1024 for 30B and 65B models; samples were truncated when necessary.Source: D Model Training Details and Compute · p. 16
• DeepSpeed and the ZeRO optimizer enabled large-scale finetuning, and all models were fully finetuned.Training was conducted primarily on the CSC LUMI GPU cluster, whose nodes each contain four AMD MI250x GPUs.Source: D Model Training Details and Compute · p. 16

Evaluation implementation

• MMLU uses official prompts with zero- and five-shot settings; across evaluations, models load in 8-bit mode and generation defaults to greedy decoding with a 512-token limit.The MMLU setup modifies the official evaluation script for batch processing; decoding defaults apply unless otherwise specified.Source: E Evaluation Setups · pp. 16–17

Reasoning benchmarks

• On GSM, models are evaluated with and without CoT using 8 few-shot examples and 200 randomly sampled cases from 1,319 testing examples; BBH uses official three-example prompts for both setups [50] [42].GSM extracts the last number in the model response as the final answer. For BBH CoT, evaluation extracts the first word after the designated answer phrase, or the entire response if that phrase is absent.Source: E Evaluation Setups · p. 16

Multilingual question answering

• TydiQA evaluates multilingual question answering with a gold passage or in closed-book mode, using one in-context example to establish the answer format [2].The gold-passage condition provides the passage containing the answer; the closed-book condition provides no context.Source: E Evaluation Setups · p. 17

Coding

• Codex-Eval uses 164 HumanEval Python problems and unbiased pass@k estimates, reporting pass@1 at temperature 0.1 and pass@10 at 0.8 [7].Models complete functions from their docstrings.Source: E Evaluation Setups · p. 17

Toxicity and truthfulness

• ToxiGen scores the percentage of generations classified as toxic, while TruthfulQA reports truthfulness, informativeness, and their intersection, emphasizing the combined metric [45] [30].ToxiGen uses 500 hateful prompts per group; TruthfulQA uses 818 questions and six in-context examples.Source: E Evaluation Setups · p. 17

Open-ended instruction following

• AlpacaEval compares responses to 805 prompts against Davinci-003 using GPT-4, with win rate measuring the share preferred over Davinci-003.The setup uses the alpaca_eval_gpt4_0314 annotator configuration and allows up to 8192 generated tokens.Source: E Evaluation Setups · p. 17

F Overview of All Automatic Evaluation Results

• ToxiGen results are broken down by targeted group, while TruthfulQA reports informative and truthful answer percentages; some groups remain specially targeted after instruction tuning.TruthfulQA also reports the percentage of answers that are both informative and truthful.Source: F Overview of All Automatic Evaluation… · p. 17, Table 9 · p. 19, Table 10 · p. 20
• Table 8 compiles this work’s models and proprietary models across core capability benchmarks, with an overall average weighting each benchmark equally.The average is calculated by averaging scenarios within each benchmark, then averaging across benchmarks.Source: F Overview of All Automatic Evaluation… · p. 17, Table 8 · p. 18

G.1 Setup

• Human evaluation covered 332 instructions and compared TÜLU 65B with ChatGPT, TÜLU 7B, and a 65B LLAMA model trained on the Human data mixture.The set included 252 Self-Instruct instructions [47] and 80 Vicuna instructions [8], using the same instructions across comparisons.Source: §G.1 · p. 21
• Eighteen expert annotators—AI2 researchers or UW students, all fluent English speakers with at least bachelor’s degrees—conducted the evaluation.Source: §G.1 · p. 21
• The evaluation website anonymized model identities and randomized output order; annotators judged each output’s acceptability and compared helpfulness, with external tools permitted.Figure 5 shows the interface and its acceptability and helpfulness questions.Source: §G.1 · p. 21, Figure 5 · p. 21

G.2 Inter-Annotator Agreement

• Annotators agreed at 0.84 on output acceptance and 0.72 on pairwise comparison using tie-discounted accuracy.For pairwise comparison, ties earned half credit when only one annotator selected a tie; “clearly better” and “slightly better” were merged. The agreement analysis used 119 examples.Source: §G.2 · pp. 21–22
• Despite reasonable agreement, human evaluation remains substantially subjective, raising reliability concerns for prior studies using few examples and motivating more scalable evaluation.Source: §G.2 · p. 22

H Further Investigation of Figure 2

• GPT-4’s win rates closely tracked a dummy evaluator favoring outputs with more unique tokens (R^2 = .91), although the dummy generally overestimated wins and token count was not the only preference.Source: Figure 6 · p. 22, H Further Investigation of Figure 2 · p. 22