PaperlayerView exact revision on arXiv

Source-linked AI summary

WizardLM: Empowering large pre-trained language models to follow complex instructions

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, Daxin Jiang

arXiv:2304.12244v3cs.CLcs.AI

TL;DR

LLMs need better instruction-following data, but manually creating diverse, high-complexity open-domain instructions is costly and difficult. The paper introduces Evol-Instruct to iteratively evolve instructions with LLMs, then fine-tunes LLaMA into WizardLM. WizardLM outperforms typical open-source baselines across broad evaluations, while the authors report limitations in evaluation scalability, reliability, and test-set coverage.

  • Problem

    LLMs often struggle with user instructions, while manually producing diverse, high-complexity open-domain instruction data is time-consuming and labor-intensive.

    Source: §1 · pp. 1–2

  • Method

    Evol-Instruct iteratively evolves seed instructions with LLMs, generates responses, filters failed evolutions, and uses the resulting data to fine-tune LLaMA into WizardLM.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    WizardLM significantly surpasses typical open-source baselines such as Alpaca and Vicuna across a wide range of benchmarks, including code, math, GPT-4, and human evaluations.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The findings support fine-tuning with AI-evolved instructions as a promising direction for enhancing LLMs.

    Source: §1 · p. 3, §5 · p. 9

    The authors acknowledge limitations in the scalability and reliability of automatic GPT-4 and human evaluations, and in the representativeness of the test set.

    Source: §5 · p. 9

Abstract

from arXiv · show

Training large language models (LLMs) with open-domain instruction following data brings colossal success. However, manually creating such instruction data is very time-consuming and labor-intensive. Moreover, humans may struggle to produce high-complexity instructions. In this paper, we show an avenue for creating large amounts of instruction data with varying levels of complexity using LLM instead of humans. Starting with an initial set of instructions, we use our proposed Evol-Instruct to rewrite them step by step into more complex instructions. Then, we mix all generated instruction data to fine-tune LLaMA. We call the resulting model WizardLM. Human evaluations on a complexity-balanced test bed and Vicuna's testset show that instructions from Evol-Instruct are superior to human-created ones. By analyzing the human evaluation results of the high complexity part, we demonstrate that outputs from our WizardLM are preferred to outputs from OpenAI ChatGPT. In GPT-4 automatic evaluation, WizardLM achieves more than 90\% capacity of ChatGPT on 17 out of 29 skills. Even though WizardLM still lags behind ChatGPT in some aspects, our findings suggest that fine-tuning with AI-evolved instructions is a promising direction for enhancing LLMs. Our code and data are public at https://github.com/nlpxucan/WizardLM

1 INTRODUCTION

LLMs often struggle to follow user instructions, while manually producing diverse, high-complexity open-domain instructions is costly and difficult. Evol-Instruct addresses this by using LLMs to generate varied instructions for fine-tuning, yielding WizardLM with stronger reported performance than open-source baselines.

  • LLMs generate fluent text but often struggle to follow user-specified instructions or goals.Source: §1 · p. 1
  • Human instruction creation is time-consuming, labor-intensive, and difficult to sustain for high-difficulty examples.Source: §1 · p. 2
  • Evol-Instruct uses LLMs to mass-produce open-domain instructions across varying topics and difficulty levels.It evolves seed instructions through in-depth and in-breadth transformations, filtering failed evolutions.Source: §1 · p. 2
  • WizardLM significantly surpasses typical open-source models such as Alpaca and Vicuna across multiple benchmarks.The reported advantages include code, math, GPT-4, and human evaluations.Source: §1 · p. 3
  • Instruction complexity is important for achieving strong supervised fine-tuning performance in large pre-trained language models.Source: §1 · p. 3

2 RELATED WORK

Instruction tuning developed from closed-domain cross-task generalization toward open-domain systems trained on diverse human-created instructions. This shift supported models capable of processing varied real-user requests.

  • Closed-domain instruction tuning trains language models across broad public NLP tasks and evaluates them on different tasks.Source: §2 · p. 3
  • T5 unified question answering, document summarization, and sentiment classification within a text-to-text training format.Source: §2 · p. 3
  • Open-domain instruction tuning uses diverse human-written instructions and responses spanning many forms and task types.Source: §2 · p. 3
  • Instruction-tuned GPT-3 produced InstructGPT, which can process a variety of real user instructions and contributed to ChatGPT's success.Source: §2 · p. 3

3 APPROACH

Evol-Instruct iteratively increases instruction complexity and diversity with an LLM, generates responses, filters failed evolutions, and merges the resulting datasets for fine-tuning. Its design combines depth-oriented rewriting with breadth-oriented creation while controlling difficulty growth.

Pipeline

  • The pipeline contains an Instruction Evolver and an Instruction Eliminator for producing and filtering evolved instructions.Source: §3 · p. 3
  • Each evolution rewrites every instruction with an Evol-Instruct prompt, generates a corresponding response, and produces a new evolution dataset.Source: §3.1 · p. 4

Dataset construction

  • Iterating for M evolutions yields M sequential datasets that are combined with the initial dataset for training.Source: §3.1 · p. 4, §3.3 · p. 6

In-depth evolving

  • In-depth evolving increases complexity through added constraints, deepening, concretization, more reasoning steps, and more complicated inputs.Source: §3.2 · p. 4
  • Difficulty growth is gradual because each evolution makes an instruction only a bit harder and adds at most 10 to 20 words.Source: §3.2 · p. 5

In-breadth evolving

  • In-breadth evolving creates new, rarer instructions in the same domain to expand topic coverage, skill coverage, and dataset diversity.Source: §3.2 · p. 5

Elimination evolving

  • Elimination removes evolutions lacking information gain, difficult to answer, containing only punctuation or stop words, or copying prompt-template language.Source: §3.2 · p. 6

Dataset construction

  • After evolution, initial and evolved samples are merged and randomly shuffled to distribute instructions across difficulty levels.Source: §3.3 · p. 6

4 EXPERIMENT

The experiments evaluate WizardLM against open-source baselines using automatic benchmarks, GPT-4 judgments, and human comparisons on WizardEval. Results favor WizardLM across most benchmarks, while ablations examine data seeds, evolution settings, model scale, instruction difficulty, and topic diversity.

Automatic Evaluation

  • WizardLM is evaluated against ChatGPT-3.5 and several open-source baselines across nine LLM benchmarks.The evaluation includes OpenLLM Leaderboard tasks, HumanEval, GSM8k, AlpacaEval, MT-Bench, and WizardEval judgments.Source: §4 · p. 6, §4.3 · p. 7
  • WizardLM shows a substantial performance advantage over same-sized open-source models, especially in math, code, and GPT-4 evaluations.The comparison includes Alpaca, Vicuna, Baize, CAMEL, and Tulu.Source: §4.3 · p. 7

Human Evaluation

  • WizardEval contains 218 real-world instructions spanning 29 skills and domains, with difficulty and complexity distributed more uniformly than in Vicuna and Alpaca test sets.The testbed covers areas including coding, math, reasoning, complex formats, writing, and extensive disciplines.Source: §4.4 · p. 8
  • In blind pairwise human evaluation, WizardLM performs significantly better than Alpaca and Vicuna, with annotator agreement above 0.6 on all Kappa scores.Ten annotators compared shuffled responses using relevance, knowledge, reasoning, calculation, and accuracy criteria.Source: §4.4 · p. 8

Ablation Study

  • Ablations indicate that ShareGPT is a better evolution seed than Alpaca, larger evolved datasets improve capacity, and Evol-Instruct works with Llama-2 and other base models.Evolved data also outperforms Supernatural Instructions in fine-tuning, while results extend across Llama-1, Llama-2, and Mistral bases.Source: §4.5 · pp. 8–9
  • Across four evolution rounds, increasing instruction complexity corresponds to improving average performance on nine automatic benchmarks.Difficulty scores from ChatGPT agree well with GPT-4 and human annotators, and in-breadth evolution produces greater topic dispersion than ShareGPT and Alpaca.Source: §4.5 · p. 9

5 CONCLUSIONS

The paper presents Evol-Instruct and reports that WizardLM surpasses typical open-source LLMs across benchmarks, while acknowledging evaluation limitations.

  • Evol-Instruct generates diverse and complex instruction data for LLMs.Source: §5 · p. 9
  • WizardLM significantly surpasses typical open-source LLMs such as Alpaca and Vicuna across well-recognized benchmarks.Source: §5 · p. 9
  • WizardLM outperforms baselines by a substantial margin in code, math, GPT-4, and human evaluations.Source: §5 · p. 9
  • Automatic GPT-4 and human evaluation methods pose challenges for scalability and reliability.Source: §5 · p. 9
  • The test set may not represent all scenarios or domains where LLMs can be applied or compared with other methods.Source: §5 · p. 9

A DEEPENING PROMPT

The deepening prompt instructs an LLM to rewrite an instruction into a more complex but still reasonable version while preserving its content and limiting verbosity.

A DEEPENING PROMPT

B CONCRETIZING PROMPT

C INCREASED REASONING STEPS PROMPT

D COMPLICATE INPUT PROMPT

The complicate-input prompt increases instruction complexity by embedding structured data or code while retaining a reasonable, human-answerable task.

D COMPLICATE INPUT PROMPT

H WIZARDEVAL ANALYSIS

WizardEval contains 218 real-world instructions spanning 29 skills, with a more uniform distribution of difficulty and complexity than Vicuna and Alpaca.

  • The test set includes skills such as Coding Generation and Debugging, Math, Reasoning, Complex Formats, Writing, and Extensive Disciplines.Source: H WIZARDEVAL ANALYSIS · p. 19
  • 218 instances cover 29 distinct skills in the Evol-Instruct test set.Each instance corresponds to an instruction for a specific skill.Source: H WIZARDEVAL ANALYSIS · pp. 19–20
  • WizardEval distributes instructions more uniformly across difficulty and complexity levels than Vicuna and Alpaca.Vicuna and Alpaca are described as skewed toward low-difficulty, low-complexity instructions.Source: H WIZARDEVAL ANALYSIS · p. 20
  • Figure 6 shows the distribution of instances across skills in the Evol-Instruct test set.Source: Figure 6 · p. 20

I DIFFERENT DIFFICULTY ANNOTATORS

The paper compares ChatGPT, GPT-4, and human annotators for assessing instruction difficulty, including agreement judgments over randomly paired instructions.

  • ChatGPT, GPT-4, and five human annotators assess the difficulty of sampled instructions.The experiment samples 600 instructions for difficulty assessment.Source: I DIFFERENT DIFFICULTY ANNOTATORS · p. 20
  • The difficulty-assessment results are reported in Table 3.Source: Table 3 · p. 20
  • 300 instruction pairs are formed from Alpaca, ShareGPT, and evolution rounds C1 to C4 for pairwise difficulty judgments.ChatGPT and five human annotators judge which instruction in each pair is more difficult.Source: I DIFFERENT DIFFICULTY ANNOTATORS · p. 20

J CLUSTER SCATTER PLOT

The cluster scatter analysis compares instruction embeddings from ShareGPT, Alpaca, and four evolution rounds, showing greater dispersion for the evolved instructions.

K HUMAN EVALUATION ASPECTS

Human evaluators compare model responses across five aspects: relevance, knowledge, reasoning, calculation, and accuracy.

L PERFORMANCE DETAILS OF DIFFERENT CHECKPOINTS

The paper reports final-checkpoint performance after three training epochs, while Table 4 compares checkpoints from different epochs. The best checkpoint generally varies by model scale and benchmark, which the authors attribute mainly to training fluctuations.

Related papers

Loading 2304.12244v3…