Source-linked AI summary
WizardLM: Empowering large pre-trained language models to follow complex instructions
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, Daxin Jiang
TL;DR
LLMs need better instruction-following data, but manually creating diverse, high-complexity open-domain instructions is costly and difficult. The paper introduces Evol-Instruct to iteratively evolve instructions with LLMs, then fine-tunes LLaMA into WizardLM. WizardLM outperforms typical open-source baselines across broad evaluations, while the authors report limitations in evaluation scalability, reliability, and test-set coverage.
Problem
LLMs often struggle with user instructions, while manually producing diverse, high-complexity open-domain instruction data is time-consuming and labor-intensive.
Method
Evol-Instruct iteratively evolves seed instructions with LLMs, generates responses, filters failed evolutions, and uses the resulting data to fine-tune LLaMA into WizardLM.
Results
WizardLM significantly surpasses typical open-source baselines such as Alpaca and Vicuna across a wide range of benchmarks, including code, math, GPT-4, and human evaluations.
Takeaways & Limitations
The findings support fine-tuning with AI-evolved instructions as a promising direction for enhancing LLMs.
Takeaways & Limitations
The authors acknowledge limitations in the scalability and reliability of automatic GPT-4 and human evaluations, and in the representativeness of the test set.
Abstract
from arXiv · showhide
Training large language models (LLMs) with open-domain instruction following data brings colossal success. However, manually creating such instruction data is very time-consuming and labor-intensive. Moreover, humans may struggle to produce high-complexity instructions. In this paper, we show an avenue for creating large amounts of instruction data with varying levels of complexity using LLM instead of humans. Starting with an initial set of instructions, we use our proposed Evol-Instruct to rewrite them step by step into more complex instructions. Then, we mix all generated instruction data to fine-tune LLaMA. We call the resulting model WizardLM. Human evaluations on a complexity-balanced test bed and Vicuna's testset show that instructions from Evol-Instruct are superior to human-created ones. By analyzing the human evaluation results of the high complexity part, we demonstrate that outputs from our WizardLM are preferred to outputs from OpenAI ChatGPT. In GPT-4 automatic evaluation, WizardLM achieves more than 90\% capacity of ChatGPT on 17 out of 29 skills. Even though WizardLM still lags behind ChatGPT in some aspects, our findings suggest that fine-tuning with AI-evolved instructions is a promising direction for enhancing LLMs. Our code and data are public at https://github.com/nlpxucan/WizardLM
1 INTRODUCTION
LLMs often struggle to follow user instructions, while manually producing diverse, high-complexity open-domain instructions is costly and difficult. Evol-Instruct addresses this by using LLMs to generate varied instructions for fine-tuning, yielding WizardLM with stronger reported performance than open-source baselines.
- LLMs generate fluent text but often struggle to follow user-specified instructions or goals.
- Human instruction creation is time-consuming, labor-intensive, and difficult to sustain for high-difficulty examples.
- Evol-Instruct uses LLMs to mass-produce open-domain instructions across varying topics and difficulty levels.It evolves seed instructions through in-depth and in-breadth transformations, filtering failed evolutions.
- WizardLM significantly surpasses typical open-source models such as Alpaca and Vicuna across multiple benchmarks.The reported advantages include code, math, GPT-4, and human evaluations.
- Instruction complexity is important for achieving strong supervised fine-tuning performance in large pre-trained language models.
2 RELATED WORK
Instruction tuning developed from closed-domain cross-task generalization toward open-domain systems trained on diverse human-created instructions. This shift supported models capable of processing varied real-user requests.
- Closed-domain instruction tuning trains language models across broad public NLP tasks and evaluates them on different tasks.
- T5 unified question answering, document summarization, and sentiment classification within a text-to-text training format.
- Open-domain instruction tuning uses diverse human-written instructions and responses spanning many forms and task types.
- Instruction-tuned GPT-3 produced InstructGPT, which can process a variety of real user instructions and contributed to ChatGPT's success.
3 APPROACH
Evol-Instruct iteratively increases instruction complexity and diversity with an LLM, generates responses, filters failed evolutions, and merges the resulting datasets for fine-tuning. Its design combines depth-oriented rewriting with breadth-oriented creation while controlling difficulty growth.
- Pipeline: The pipeline contains an Instruction Evolver and an Instruction Eliminator for producing and filtering evolved instructions.
- Pipeline: Each evolution rewrites every instruction with an Evol-Instruct prompt, generates a corresponding response, and produces a new evolution dataset.
- Dataset construction: Iterating for M evolutions yields M sequential datasets that are combined with the initial dataset for training.
- In-depth evolving: In-depth evolving increases complexity through added constraints, deepening, concretization, more reasoning steps, and more complicated inputs.
- In-depth evolving: Difficulty growth is gradual because each evolution makes an instruction only a bit harder and adds at most 10 to 20 words.
- In-breadth evolving: In-breadth evolving creates new, rarer instructions in the same domain to expand topic coverage, skill coverage, and dataset diversity.
- Elimination evolving: Elimination removes evolutions lacking information gain, difficult to answer, containing only punctuation or stop words, or copying prompt-template language.
- Dataset construction: After evolution, initial and evolved samples are merged and randomly shuffled to distribute instructions across difficulty levels.
4 EXPERIMENT
The experiments evaluate WizardLM against open-source baselines using automatic benchmarks, GPT-4 judgments, and human comparisons on WizardEval. Results favor WizardLM across most benchmarks, while ablations examine data seeds, evolution settings, model scale, instruction difficulty, and topic diversity.
- Automatic Evaluation: WizardLM is evaluated against ChatGPT-3.5 and several open-source baselines across nine LLM benchmarks.The evaluation includes OpenLLM Leaderboard tasks, HumanEval, GSM8k, AlpacaEval, MT-Bench, and WizardEval judgments.
- Automatic Evaluation: WizardLM shows a substantial performance advantage over same-sized open-source models, especially in math, code, and GPT-4 evaluations.The comparison includes Alpaca, Vicuna, Baize, CAMEL, and Tulu.
- Human Evaluation: WizardEval contains 218 real-world instructions spanning 29 skills and domains, with difficulty and complexity distributed more uniformly than in Vicuna and Alpaca test sets.The testbed covers areas including coding, math, reasoning, complex formats, writing, and extensive disciplines.
- Human Evaluation: In blind pairwise human evaluation, WizardLM performs significantly better than Alpaca and Vicuna, with annotator agreement above 0.6 on all Kappa scores.Ten annotators compared shuffled responses using relevance, knowledge, reasoning, calculation, and accuracy criteria.
- Ablation Study: Ablations indicate that ShareGPT is a better evolution seed than Alpaca, larger evolved datasets improve capacity, and Evol-Instruct works with Llama-2 and other base models.Evolved data also outperforms Supernatural Instructions in fine-tuning, while results extend across Llama-1, Llama-2, and Mistral bases.
- Ablation Study: Across four evolution rounds, increasing instruction complexity corresponds to improving average performance on nine automatic benchmarks.Difficulty scores from ChatGPT agree well with GPT-4 and human annotators, and in-breadth evolution produces greater topic dispersion than ShareGPT and Alpaca.
5 CONCLUSIONS
The paper presents Evol-Instruct and reports that WizardLM surpasses typical open-source LLMs across benchmarks, while acknowledging evaluation limitations.
- Evol-Instruct generates diverse and complex instruction data for LLMs.
- WizardLM significantly surpasses typical open-source LLMs such as Alpaca and Vicuna across well-recognized benchmarks.
- WizardLM outperforms baselines by a substantial margin in code, math, GPT-4, and human evaluations.
- Automatic GPT-4 and human evaluation methods pose challenges for scalability and reliability.
- The test set may not represent all scenarios or domains where LLMs can be applied or compared with other methods.
A DEEPENING PROMPT
The deepening prompt instructs an LLM to rewrite an instruction into a more complex but still reasonable version while preserving its content and limiting verbosity.
- A DEEPENING PROMPT: The deepening prompt asks a Prompt Rewriter to make instructions more complex and harder for systems such as ChatGPT and GPT4 to handle.
- A DEEPENING PROMPT: Rewritten instructions must remain reasonable and understandable to humans.
- A DEEPENING PROMPT: The rewriting must preserve non-text elements such as tables and code, as well as the original input.
- A DEEPENING PROMPT: The prompt directs the rewriter to complicate instructions using a specified method.
- A DEEPENING PROMPT: For inquiries about particular issues, the rewriter can increase the inquiry’s depth and breadth.
- B CONCRETIZING PROMPT: The concretizing variant replaces general concepts with more specific concepts.
- C INCREASED REASONING STEPS PROMPT: The increased-reasoning-steps variant rewrites simple tasks to explicitly request multiple-step reasoning.
D COMPLICATE INPUT PROMPT
The complicate-input prompt increases instruction complexity by embedding structured data or code while retaining a reasonable, human-answerable task.
- D COMPLICATE INPUT PROMPT: The examples require SQL database, XML data, Python code, HTML page, Shell command, or JSON data in rewritten prompts.
- D COMPLICATE INPUT PROMPT: One XML example supplies dated impressions and money fields, totals, and filters for a PHP data-fetching task.
- D COMPLICATE INPUT PROMPT: The embedded examples preserve concrete task inputs, such as code, tables, database rows, and XML records.
- D COMPLICATE INPUT PROMPT: Complicate Input uses data-format content as input to make rewritten instructions more difficult while keeping them reasonable and human-answerable.
- D COMPLICATE INPUT PROMPT: One SQL example asks for a query that returns the last record in each name group without using an inefficient approach.
- D COMPLICATE INPUT PROMPT: Other examples embed Python, HTML, Shell, and JSON content to turn short tasks into detailed technical questions.
- D COMPLICATE INPUT PROMPT: The prompt examples also include auxiliary judges for instruction equality and math-relatedness, returning categorical judgments.
H WIZARDEVAL ANALYSIS
WizardEval contains 218 real-world instructions spanning 29 skills, with a more uniform distribution of difficulty and complexity than Vicuna and Alpaca.
- The test set includes skills such as Coding Generation and Debugging, Math, Reasoning, Complex Formats, Writing, and Extensive Disciplines.
- 218 instances cover 29 distinct skills in the Evol-Instruct test set.Each instance corresponds to an instruction for a specific skill.
- WizardEval distributes instructions more uniformly across difficulty and complexity levels than Vicuna and Alpaca.Vicuna and Alpaca are described as skewed toward low-difficulty, low-complexity instructions.
- Figure 6 shows the distribution of instances across skills in the Evol-Instruct test set.
I DIFFERENT DIFFICULTY ANNOTATORS
The paper compares ChatGPT, GPT-4, and human annotators for assessing instruction difficulty, including agreement judgments over randomly paired instructions.
- ChatGPT, GPT-4, and five human annotators assess the difficulty of sampled instructions.The experiment samples 600 instructions for difficulty assessment.
- The difficulty-assessment results are reported in Table 3.
- 300 instruction pairs are formed from Alpaca, ShareGPT, and evolution rounds C1 to C4 for pairwise difficulty judgments.ChatGPT and five human annotators judge which instruction in each pair is more difficult.
J CLUSTER SCATTER PLOT
The cluster scatter analysis compares instruction embeddings from ShareGPT, Alpaca, and four evolution rounds, showing greater dispersion for the evolved instructions.
- BERT encodes each instruction into 768-dimensional embeddings before visualization.
- t-SNE reduces the embeddings to two dimensions, and k-means partitions each dataset into 20 clusters.
- The evolved instructions are more dispersed than ShareGPT and Alpaca in the scatter plot.The paper interprets this greater dispersion as better topic diversity.
- Figure 7 displays the cluster scatter plot across ShareGPT, Alpaca, and evolution rounds C1 through C4.
K HUMAN EVALUATION ASPECTS
Human evaluators compare model responses across five aspects: relevance, knowledge, reasoning, calculation, and accuracy.
- Relevance measures whether a model correctly interprets the context and questions.
- Knowledgeable assesses accurate use of varied and detailed knowledge for problem-solving.
- Reasoning evaluates correct reasoning processes or valid reasoning concepts for solving problems.
- Calculation evaluates mathematical computations in math, biology, chemistry, and physics.
- Accuracy evaluates whether the model performs correctly for a given instruction.
L PERFORMANCE DETAILS OF DIFFERENT CHECKPOINTS
The paper reports final-checkpoint performance after three training epochs, while Table 4 compares checkpoints from different epochs. The best checkpoint generally varies by model scale and benchmark, which the authors attribute mainly to training fluctuations.
- 3 epochs: The main experiments report only the final checkpoint to align with previous works.
- Table 4 compares model checkpoints from epochs 2.5, 2.75, and 3.
- For 13B models, WizardLM-13b (ShareGPT Seed) performs best on every benchmark except GSM8k.
- For 65B/70B models, WizardLM-70b performs best across all reported benchmarks.
- The authors attribute checkpoint differences mainly to fluctuations on some benchmarks during training.