PaperlayerView exact revision on arXiv

Source-linked AI summary

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, Daxin Jiang

arXiv:2306.08568v2cs.CLcs.AI

TL;DR

Code LLMs have seen limited fine-grained instruction tuning despite strong code-task performance. WizardCoder adapts Evol-Instruct to coding by evolving instruction data before fine-tuning Code LLMs, achieving state-of-the-art results across five code-generation benchmarks and outperforming several closed models. The paper also identifies instruction complexity as important while noting a remaining gap behind GPT4.

  • Problem

    Most Code LLMs primarily focus on pre-training, while fine-grained instruction tuning in the code domain has received limited exploration.

  • Method

    WizardCoder adapts Evol-Instruct to coding by evolving Code Alpaca instructions with coding-specific methods and fine-tuning StarCoder and CodeLlama on the resulting data.

  • Results

    WizardCoder achieves state-of-the-art performance across five code-generation benchmarks, surpasses open-source Code LLMs, and WizardCoder 34B exceeds GPT3.5 on HumanEval+ with 64.6 vs. 63.4.

  • Takeaways & Limitations

    The analysis highlights instruction complexity as pivotal for coding performance, supporting complex instruction fine-tuning as the paper’s central direction.

  • Takeaways & Limitations

    On one benchmark, the 34B model is not included because the evaluation framework is not designed for instruction-fine-tuned models and CodeLlama-34B does not support code insertion.

Abstract

from arXiv · show

Code Large Language Models (Code LLMs), such as StarCoder, have demonstrated exceptional performance in code-related tasks. However, most existing models are solely pre-trained on extensive raw code data without instruction fine-tuning. In this paper, we introduce WizardCoder, which empowers Code LLMs with complex instruction fine-tuning, by adapting the Evol-Instruct method to the domain of code. Through comprehensive experiments on four prominent code generation benchmarks, namely HumanEval, HumanEval+, MBPP, and DS-1000, we unveil the exceptional capabilities of our model. It surpasses all other open-source Code LLMs by a substantial margin. Moreover, our model even outperforms the largest closed LLMs, Anthropic's Claude and Google's Bard, on HumanEval and HumanEval+. Our code, model weights, and data are public at https://github.com/nlpxucan/WizardLM

1 INTRODUCTION

Code Evol-Instruct adapts instruction evolution to coding, addressing limited fine-grained instruction tuning for Code LLMs. WizardCoder models are produced by evolving code instructions and fine-tuning StarCoder and CodeLlama, with strong benchmark results against open- and closed-source systems.

  • Most previous Code LLMs primarily emphasize pre-training, while fine-grained instruction tuning in the code domain remains limited.
  • Code Evol-Instruct automatically increases code-instruction complexity using coding-specific heuristics, adversarial samples, time/space requirements, and evolving stop controls.
  • The method first evolves Code Alpaca instruction data, then fine-tunes StarCoder and CodeLlama on the generated instruction-following set.
  • WizardCoder 15B outperforms Claude and Bard, while WizardCoder 34B matches GPT3.5 on HumanEval and surpasses it on HumanEval+.
  • WizardCoder models surpass other open-source Code LLMs by a substantial margin in coding tasks.

2 RELATED WORK

Related work spans general and code-focused LLMs, instruction fine-tuning, and methods for creating more complex instruction data. WizardCoder builds on these lines while targeting open-source code models.

  • Closed-source LLMs such as GPT3, GPT4, PaLM, Bard, Chinchilla, Gopher, and Claude are accessible through APIs or may be inaccessible altogether.
  • Open-source LLM releases include GPT-NeoX, GPT-J, UL2, GLM, OPT, and LLaMA, but generally do not match closed-source model performance.
  • Code LLMs including Codex, Code-Davinci, PaLM-Coder, CodeGen, CodeT5, CodeGeeX, StarCoder, and CodeLlama address code understanding and generation.
  • Instruction fine-tuning developed from multitask training for cross-task generalization toward human-instruction data intended to improve alignment with user inputs.
  • WizardLM’s Evol-Instruct evolves existing general instruction data into more complex and diverse datasets, inspiring the coding-domain adaptation.

3 WIZARDCODER: SOTA OPEN-SOURCE CODE LLM

WizardCoder is trained by iteratively evolving Code Alpaca instructions with coding-specific transformations and then fine-tuning pre-trained Code LLMs. The process increases task complexity while using controlled stopping and cumulative training data.

  • 3 WIZARDCODER: SOTA OPEN-SOURCE CODE LLM: Code Evol-Instruct iteratively evolves the Code Alpaca dataset before fine-tuning pre-trained Code LLMs with the evolved data.
  • 3 WIZARDCODER: SOTA OPEN-SOURCE CODE LLM: The coding-domain evolution process is designed to automatically increase instruction complexity and improve fine-tuning effectiveness.
  • 3.1 CODE EVOL-INSTRUCT: The evolution prompt asks GPT-based generation to increase a programming question’s difficulty using a selected evolution method.
  • 3.1 CODE EVOL-INSTRUCT: Its heuristics add constraints, replace common requirements with specific ones, increase reasoning steps, and introduce erroneous reference code.
  • 3.1 CODE EVOL-INSTRUCT: Additional evolution strategies include occasional higher time or space complexity requirements and adversarial erroneous-code samples.
  • 3.2 TRAINING WizardCoder: Training starts from StarCoder 15B and CodeLlama-34B-Python and fine-tunes them on the evolved code instruction-following dataset.
  • 3.2 TRAINING WizardCoder: The dataset begins with around 20k Code Alpaca samples, merges evolved data from prior rounds with the original set, and halts evolution when development performance drops.

4 EXPERIMENT

WizardCoder is evaluated against closed- and open-source Code LLM baselines across five code-generation benchmarks and multiple programming languages. Its results show substantial advantages over open-source models, with strong performance against closed-source systems on HumanEval+ and across MultiPL-E and DS-1000.

  • Baselines and Benchmarks: The evaluation covers HumanEval, HumanEval+, MBPP, DS-1000, and MultiPL-E, alongside numerous closed- and open-source baselines.MultiPL-E assessment spans eight programming languages.
  • HumanEval and HumanEval+: 64.6 vs. 63.4: WizardCoder 34B surpasses GPT3.5 (ChatGPT) on HumanEval+ under single-attempt greedy decoding.WizardCoder 34B ranks second on the benchmark.
  • HumanEval and HumanEval+: 59.8 vs. 53.0 and 59.8 vs. 44.5: WizardCoder 15B outperforms Claude-Plus and Bard on HumanEval+.The comparisons use the EvalPlus leaderboard setting with one generated solution per problem.
  • Open-Source Comparisons: WizardCoder demonstrates a substantial performance advantage over all evaluated open-source models on HumanEval and MBPP.These comparisons estimate pass@1 by generating n=200 samples per problem with temperature 0.2 and top p=0.95.
  • Multi-Language Coding: WizardCoder models achieve superior performance across all eight evaluated programming languages on the MultiPL-E benchmark.MultiPL-E results use pass@1 with temperature 0.2, top p=0.95, maximum length 512, and n=50.
  • DS-1000: WizardCoder demonstrates significant superiority over other models on DS-1000 data-science problems using overall and per-library pass@1 results.The benchmark covers workflows spanning seven libraries and reports insertion scores in these experiments.

5 ANALYSIS

The analysis tests whether WizardCoder’s gains arise from evolution-model choices, evolution rounds, increased data quantity, or greater test-set similarity. Results point primarily to instruction complexity and identify three evolution rounds as optimal in the reported setting.

  • Evolution Models and Rounds: 73.8 (34B) and 62.2 (15B) are the HumanEval Pass@1 scores achieved when GPT-4 replaces GPT-3.5 for evolved rounds.Using CodeLlama-Instruct-34B yields 70.1 (34B) and 55.5 (15B).
  • Evolution Models and Rounds: Three evolution rounds produce the highest pass@1 scores on both the MBPP-400 dev set and HumanEval.All models are fine-tuned with 200 steps, and MBPP training and development data are merged into MBPP-400.
  • Complexity and Quantity: With similar sample or token counts per round, seed-data models still lag behind evolved-round models, while combining rounds performs best.This comparison separates the effect of evolution from simply adding samples or tokens.
  • Complexity and Similarity: Evolution does not produce higher similarity scores between HumanEval test samples and top-1 retrieved training samples, whose scores remain relatively low across rounds.The analysis uses gte-large retrieval and GPT-4-generated average similarity scores.
  • Complexity and Similarity: The reported findings attribute the primary performance gain to more complex evolved data rather than merely increased sample or token counts or greater test-set similarity.The quantity comparison and similarity analysis provide separate evidence for this conclusion.

6 CONCLUSION AND FUTURE WORK

The conclusion reports state-of-the-art WizardCoder performance across five code-generation benchmarks and highlights instruction complexity as important. It also notes that WizardCoder remains behind GPT-4, motivating future performance improvements.

  • Conclusion: WizardCoder achieves state-of-the-art performance over existing open-source Code LLMs on HumanEval, HumanEval+, MBPP, DS-1000, and MultiPL-E.The conclusion identifies instruction complexity as pivotal to performance enhancement.
  • Conclusion: WizardCoder 15B surpasses Claude and Bard, while WizardCoder 34B is comparable to GPT3.5 on HumanEval and surpasses it on HumanEval+.The conclusion presents these comparisons as evidence of WizardCoder’s benchmark performance.
  • Future Work: WizardCoder still falls significantly behind GPT4, which the paper identifies as a direction for future work.The stated future direction is to further augment the model’s performance.

A PROMPT FORMATS

The prompt-format appendix lists evaluation instructions for several benchmarks. It distinguishes zero-shot and three-shot formats and aligns DS-1000 insertion and MultiPL-E prompts with their evaluation requirements.

  • HumanEval and HumanEval+: HumanEval and HumanEval+ use a zero-shot prompt asking the model to create a Python script for the problem.The format includes an instruction describing the task and an input providing further context.
  • MBPP: MBPP uses a three-shot prompt that appends three test examples to the Python-script instruction.The examples are labeled Test Example 1, Test Example 2, and Test Example 3.
  • DS-1000: DS-1000 completion uses the same general instruction format for completing a request, while DS-1000 insertion uses StarCoder’s specialized insertion symbol and prompt format.The insertion format is required to adhere to the benchmark specifications.
  • MultiPL-E: MultiPL-E adopts StarCoder’s prompt format to align with the bigcode-evaluation-harness evaluation code.The format choice is driven by compatibility with the benchmark’s provided evaluation framework.

B BASELINES DETAILS

The baseline results combine reported results, API evaluations, and evaluations of open-source checkpoints. StarCoder’s MBPP result was re-evaluated because its original score used a smaller cleaned benchmark.

  • Baseline Sources: GPT3.5, GPT4, and many Code LLM baseline results are taken from prior reports or papers, including StarCoder and CodeT5+.The listed sources include GPT4’s report, EvalPlus, StarCoder’s paper, and CodeT5+’s paper.
  • Baseline Sources: Bard and Claude baseline results are evaluated through Google’s and Anthropic’s APIs, respectively.Other listed models are evaluated using open-source checkpoints or results from their papers.
  • MBPP Baseline: StarCoder’s MBPP score was re-evaluated on the original 500-problem benchmark because its reported result used a cleaned version containing 397 problems.The discrepancy was identified through personal contact and motivated evaluation on the original benchmark.

C SIMILARITY CHECKING AND DATA FILTERING

The similarity-checking procedure asks GPT-4 to compare two coding tasks and assign a 1–10 similarity score based on shared characteristics. An additional filtering step uses embeddings retrieval and GPT-4 judgments to reduce test-to-training data leakage.

  • GPT-4 compares two coding tasks for overlap in function names, code structures, topics, and contents.The required output is only a similarity score.
  • The similarity score ranges from 1 for completely different tasks to 10 for identical tasks.
  • All test samples are used as queries to retrieve the top 5 training samples with the gte-large embeddings model.GPT-4 then evaluates each retrieved-training and test-sample pair.
  • GPT-4 makes a simplified binary decision of “yes” or “no” when judging whether a retrieved sample matches the test sample.

D EVOL EXAMPLES

The evolved examples show instructions becoming progressively more constrained and feature-rich across rounds. Evolution adds filters, operations, statistics, error handling, data integrity requirements, transfer learning, and larger task scope.

  • Evolution examples: The examples are presented to illustrate the influence of Code Evol-Instruct.
  • MongoDB query evolution: MongoDB instructions evolve from filtering by category to adding brand and price constraints, color alternatives, sorting, and a ten-document limit.The final version selects clothes excluding Nike, restricts prices to 100–500, accepts red or blue, sorts by date added descending, and limits results.
  • TensorFlow task evolution: TensorFlow instructions progress from handwritten-digit classification to training-size constraints and transfer learning with a pre-trained model.The examples specify either 1000 or 500 MNIST samples, with the later 500-sample version requiring a CNN and feature extraction.
  • Java task evolution: Java instructions expand from storing students and courses to database operations, statistics, grades, average-grade calculation, and error handling.
  • Java task evolution: The Java sequence eventually shifts from student-course management to a library system with book operations, statistics, average reading time, error handling, and data integrity constraints.

E USING DIFFERENT EVOLUTION EXECUTION MODELS

The authors explored CodeLlama-Instruct-34B for generating evolved instructions but found its response-generation coding performance relatively low. They therefore fine-tuned it on the code-alpaca dataset before using it for response generation.

  • CodeLlama-Instruct-34B showed relatively low coding performance when generating responses to evolved instructions.
  • The authors fine-tuned CodeLlama-Instruct-34B on the code-alpaca dataset and used the resulting model for response generation.
Loading 2306.08568v2…