PaperlayerView exact revision on arXiv

Source-linked AI summary

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, Daxin Jiang

arXiv:2306.08568v2cs.CLcs.AI

TL;DR

Code LLMs have seen limited fine-grained instruction tuning despite strong code-task performance. WizardCoder adapts Evol-Instruct to coding by evolving instruction data before fine-tuning Code LLMs, achieving state-of-the-art results across five code-generation benchmarks and outperforming several closed models. The paper also identifies instruction complexity as important while noting a remaining gap behind GPT4.

  • Problem

    Most Code LLMs primarily focus on pre-training, while fine-grained instruction tuning in the code domain has received limited exploration.

    Source: §1 · p. 1

  • Method

    WizardCoder adapts Evol-Instruct to coding by evolving Code Alpaca instructions with coding-specific methods and fine-tuning StarCoder and CodeLlama on the resulting data.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    WizardCoder achieves state-of-the-art performance across five code-generation benchmarks, surpasses open-source Code LLMs, and WizardCoder 34B exceeds GPT3.5 on HumanEval+ with 64.6 vs. 63.4.

    Source: §4.3 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The analysis highlights instruction complexity as pivotal for coding performance, supporting complex instruction fine-tuning as the paper’s central direction.

    Source: §1 · p. 2, §6 · p. 9

    On one benchmark, the 34B model is not included because the evaluation framework is not designed for instruction-fine-tuned models and CodeLlama-34B does not support code insertion.

    Source: §5 · p. 8

Abstract

from arXiv · show

Code Large Language Models (Code LLMs), such as StarCoder, have demonstrated exceptional performance in code-related tasks. However, most existing models are solely pre-trained on extensive raw code data without instruction fine-tuning. In this paper, we introduce WizardCoder, which empowers Code LLMs with complex instruction fine-tuning, by adapting the Evol-Instruct method to the domain of code. Through comprehensive experiments on four prominent code generation benchmarks, namely HumanEval, HumanEval+, MBPP, and DS-1000, we unveil the exceptional capabilities of our model. It surpasses all other open-source Code LLMs by a substantial margin. Moreover, our model even outperforms the largest closed LLMs, Anthropic's Claude and Google's Bard, on HumanEval and HumanEval+. Our code, model weights, and data are public at https://github.com/nlpxucan/WizardLM

1 INTRODUCTION

Code Evol-Instruct adapts instruction evolution to coding, addressing limited fine-grained instruction tuning for Code LLMs. WizardCoder models are produced by evolving code instructions and fine-tuning StarCoder and CodeLlama, with strong benchmark results against open- and closed-source systems.

  • Most previous Code LLMs primarily emphasize pre-training, while fine-grained instruction tuning in the code domain remains limited.Source: §1 · p. 1
  • Code Evol-Instruct automatically increases code-instruction complexity using coding-specific heuristics, adversarial samples, time/space requirements, and evolving stop controls.Source: §1 · p. 2
  • The method first evolves Code Alpaca instruction data, then fine-tunes StarCoder and CodeLlama on the generated instruction-following set.Source: §1 · p. 2
  • WizardCoder 15B outperforms Claude and Bard, while WizardCoder 34B matches GPT3.5 on HumanEval and surpasses it on HumanEval+.Source: §1 · pp. 2–3
  • WizardCoder models surpass other open-source Code LLMs by a substantial margin in coding tasks.Source: §1 · p. 3

2 RELATED WORK

Related work spans general and code-focused LLMs, instruction fine-tuning, and methods for creating more complex instruction data. WizardCoder builds on these lines while targeting open-source code models.

  • Closed-source LLMs such as GPT3, GPT4, PaLM, Bard, Chinchilla, Gopher, and Claude are accessible through APIs or may be inaccessible altogether.Source: §2 · p. 3
  • Open-source LLM releases include GPT-NeoX, GPT-J, UL2, GLM, OPT, and LLaMA, but generally do not match closed-source model performance.Source: §2 · p. 3
  • Code LLMs including Codex, Code-Davinci, PaLM-Coder, CodeGen, CodeT5, CodeGeeX, StarCoder, and CodeLlama address code understanding and generation.Source: §2 · p. 3
  • Instruction fine-tuning developed from multitask training for cross-task generalization toward human-instruction data intended to improve alignment with user inputs.Source: §2 · pp. 3–4
  • WizardLM’s Evol-Instruct evolves existing general instruction data into more complex and diverse datasets, inspiring the coding-domain adaptation.Source: §2 · p. 4

3 WIZARDCODER: SOTA OPEN-SOURCE CODE LLM

WizardCoder is trained by iteratively evolving Code Alpaca instructions with coding-specific transformations and then fine-tuning pre-trained Code LLMs. The process increases task complexity while using controlled stopping and cumulative training data.

3 WIZARDCODER: SOTA OPEN-SOURCE CODE LLM

  • Code Evol-Instruct iteratively evolves the Code Alpaca dataset before fine-tuning pre-trained Code LLMs with the evolved data.Source: §3 · p. 4
  • The coding-domain evolution process is designed to automatically increase instruction complexity and improve fine-tuning effectiveness.Source: §3.1 · p. 4

3.1 CODE EVOL-INSTRUCT

  • The evolution prompt asks GPT-based generation to increase a programming question’s difficulty using a selected evolution method.Source: §3.1 · p. 4
  • Its heuristics add constraints, replace common requirements with specific ones, increase reasoning steps, and introduce erroneous reference code.Source: §3.1 · p. 5
  • Additional evolution strategies include occasional higher time or space complexity requirements and adversarial erroneous-code samples.Source: §3.1 · pp. 4–5

3.2 TRAINING WizardCoder

  • Training starts from StarCoder 15B and CodeLlama-34B-Python and fine-tunes them on the evolved code instruction-following dataset.Source: §3.2 · p. 5
  • The dataset begins with around 20k Code Alpaca samples, merges evolved data from prior rounds with the original set, and halts evolution when development performance drops.Source: §3.2 · p. 5

4 EXPERIMENT

WizardCoder is evaluated against closed- and open-source Code LLM baselines across five code-generation benchmarks and multiple programming languages. Its results show substantial advantages over open-source models, with strong performance against closed-source systems on HumanEval+ and across MultiPL-E and DS-1000.

Baselines and Benchmarks

  • The evaluation covers HumanEval, HumanEval+, MBPP, DS-1000, and MultiPL-E, alongside numerous closed- and open-source baselines.MultiPL-E assessment spans eight programming languages.Source: §4 · p. 5, §4.1 · pp. 5–6, §4.4 · p. 7

HumanEval and HumanEval+

  • 64.6 vs. 63.4: WizardCoder 34B surpasses GPT3.5 (ChatGPT) on HumanEval+ under single-attempt greedy decoding.WizardCoder 34B ranks second on the benchmark.Source: §4.3 · pp. 6–7
  • 59.8 vs. 53.0 and 59.8 vs. 44.5: WizardCoder 15B outperforms Claude-Plus and Bard on HumanEval+.The comparisons use the EvalPlus leaderboard setting with one generated solution per problem.Source: §4.3 · pp. 6–7

Open-Source Comparisons

  • WizardCoder demonstrates a substantial performance advantage over all evaluated open-source models on HumanEval and MBPP.These comparisons estimate pass@1 by generating n=200 samples per problem with temperature 0.2 and top p=0.95.Source: §4.3 · p. 7, Table 1 · p. 7

Multi-Language Coding

  • WizardCoder models achieve superior performance across all eight evaluated programming languages on the MultiPL-E benchmark.MultiPL-E results use pass@1 with temperature 0.2, top p=0.95, maximum length 512, and n=50.Source: §4.4 · p. 7, Table 2 · p. 8

DS-1000

  • WizardCoder demonstrates significant superiority over other models on DS-1000 data-science problems using overall and per-library pass@1 results.The benchmark covers workflows spanning seven libraries and reports insertion scores in these experiments.Source: §4.5 · pp. 7–8, Table 3 · p. 8

5 ANALYSIS

The analysis tests whether WizardCoder’s gains arise from evolution-model choices, evolution rounds, increased data quantity, or greater test-set similarity. Results point primarily to instruction complexity and identify three evolution rounds as optimal in the reported setting.

Evolution Models and Rounds

  • 73.8 (34B) and 62.2 (15B) are the HumanEval Pass@1 scores achieved when GPT-4 replaces GPT-3.5 for evolved rounds.Using CodeLlama-Instruct-34B yields 70.1 (34B) and 55.5 (15B).Source: §5 · p. 8
  • Three evolution rounds produce the highest pass@1 scores on both the MBPP-400 dev set and HumanEval.All models are fine-tuned with 200 steps, and MBPP training and development data are merged into MBPP-400.Source: §5 · p. 8, Figure 3 · p. 9

Complexity and Quantity

  • With similar sample or token counts per round, seed-data models still lag behind evolved-round models, while combining rounds performs best.This comparison separates the effect of evolution from simply adding samples or tokens.Source: §5 · p. 9, Table 5 · p. 9

Complexity and Similarity

  • Evolution does not produce higher similarity scores between HumanEval test samples and top-1 retrieved training samples, whose scores remain relatively low across rounds.The analysis uses gte-large retrieval and GPT-4-generated average similarity scores.Source: §5 · p. 9
  • The reported findings attribute the primary performance gain to more complex evolved data rather than merely increased sample or token counts or greater test-set similarity.The quantity comparison and similarity analysis provide separate evidence for this conclusion.Source: §5 · p. 9

6 CONCLUSION AND FUTURE WORK

The conclusion reports state-of-the-art WizardCoder performance across five code-generation benchmarks and highlights instruction complexity as important. It also notes that WizardCoder remains behind GPT-4, motivating future performance improvements.

Conclusion

  • WizardCoder achieves state-of-the-art performance over existing open-source Code LLMs on HumanEval, HumanEval+, MBPP, DS-1000, and MultiPL-E.The conclusion identifies instruction complexity as pivotal to performance enhancement.Source: §6 · p. 9
  • WizardCoder 15B surpasses Claude and Bard, while WizardCoder 34B is comparable to GPT3.5 on HumanEval and surpasses it on HumanEval+.The conclusion presents these comparisons as evidence of WizardCoder’s benchmark performance.Source: §6 · p. 9

Future Work

  • WizardCoder still falls significantly behind GPT4, which the paper identifies as a direction for future work.The stated future direction is to further augment the model’s performance.Source: §6 · p. 9

A PROMPT FORMATS

The prompt-format appendix lists evaluation instructions for several benchmarks. It distinguishes zero-shot and three-shot formats and aligns DS-1000 insertion and MultiPL-E prompts with their evaluation requirements.

HumanEval and HumanEval+

  • HumanEval and HumanEval+ use a zero-shot prompt asking the model to create a Python script for the problem.The format includes an instruction describing the task and an input providing further context.Source: A PROMPT FORMATS · p. 16

MBPP

  • MBPP uses a three-shot prompt that appends three test examples to the Python-script instruction.The examples are labeled Test Example 1, Test Example 2, and Test Example 3.Source: A PROMPT FORMATS · p. 16

DS-1000

  • DS-1000 completion uses the same general instruction format for completing a request, while DS-1000 insertion uses StarCoder’s specialized insertion symbol and prompt format.The insertion format is required to adhere to the benchmark specifications.Source: A PROMPT FORMATS · p. 16

MultiPL-E

  • MultiPL-E adopts StarCoder’s prompt format to align with the bigcode-evaluation-harness evaluation code.The format choice is driven by compatibility with the benchmark’s provided evaluation framework.Source: A PROMPT FORMATS · p. 16

B BASELINES DETAILS

The baseline results combine reported results, API evaluations, and evaluations of open-source checkpoints. StarCoder’s MBPP result was re-evaluated because its original score used a smaller cleaned benchmark.

Baseline Sources

  • GPT3.5, GPT4, and many Code LLM baseline results are taken from prior reports or papers, including StarCoder and CodeT5+.The listed sources include GPT4’s report, EvalPlus, StarCoder’s paper, and CodeT5+’s paper.Source: B BASELINES DETAILS · pp. 16–17
  • Bard and Claude baseline results are evaluated through Google’s and Anthropic’s APIs, respectively.Other listed models are evaluated using open-source checkpoints or results from their papers.Source: B BASELINES DETAILS · pp. 16–17

MBPP Baseline

  • StarCoder’s MBPP score was re-evaluated on the original 500-problem benchmark because its reported result used a cleaned version containing 397 problems.The discrepancy was identified through personal contact and motivated evaluation on the original benchmark.Source: B BASELINES DETAILS · p. 17

C SIMILARITY CHECKING AND DATA FILTERING

The similarity-checking procedure asks GPT-4 to compare two coding tasks and assign a 1–10 similarity score based on shared characteristics. An additional filtering step uses embeddings retrieval and GPT-4 judgments to reduce test-to-training data leakage.

D EVOL EXAMPLES

The evolved examples show instructions becoming progressively more constrained and feature-rich across rounds. Evolution adds filters, operations, statistics, error handling, data integrity requirements, transfer learning, and larger task scope.

Evolution examples

MongoDB query evolution

  • MongoDB instructions evolve from filtering by category to adding brand and price constraints, color alternatives, sorting, and a ten-document limit.The final version selects clothes excluding Nike, restricts prices to 100–500, accepts red or blue, sorts by date added descending, and limits results.Source: D EVOL EXAMPLES · p. 17

TensorFlow task evolution

  • TensorFlow instructions progress from handwritten-digit classification to training-size constraints and transfer learning with a pre-trained model.The examples specify either 1000 or 500 MNIST samples, with the later 500-sample version requiring a CNN and feature extraction.Source: D EVOL EXAMPLES · p. 18

Java task evolution

  • Java instructions expand from storing students and courses to database operations, statistics, grades, average-grade calculation, and error handling.Source: D EVOL EXAMPLES · p. 18
  • The Java sequence eventually shifts from student-course management to a library system with book operations, statistics, average reading time, error handling, and data integrity constraints.Source: D EVOL EXAMPLES · p. 18

E USING DIFFERENT EVOLUTION EXECUTION MODELS

The authors explored CodeLlama-Instruct-34B for generating evolved instructions but found its response-generation coding performance relatively low. They therefore fine-tuned it on the code-alpaca dataset before using it for response generation.

Related papers

Loading 2306.08568v2…