PaperlayerView exact revision on arXiv

Source-linked AI summary

Magicoder: Empowering Code Generation with OSS-Instruct

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, Lingming Zhang

arXiv:2312.02120v2cs.CLcs.AIcs.SE

TL;DR

Existing synthetic code-instruction methods rely on narrow predefined tasks or heuristics and may inherit bias from their language models and task designs. Magicoder addresses this with OSS-INSTRUCT, which uses open-source code snippets to generate diverse coding challenges, while combining it with Evol-Instruct for enhanced models. Magicoder models substantially outperform similarly sized code models, including MagicoderS-CL surpassing ChatGPT on HumanEval+ pass@1.

  • Problem

    Existing instruction-data methods depend on narrow predefined tasks or heuristics and may inherit bias from the underlying LLMs and task designs.

  • Method

    OSS-INSTRUCT uses powerful LLMs and random open-source code snippets to generate diverse, realistic, and controllable coding problems for training code models.

  • Results

    Magicoder models outperform studied code models with similar or larger parameter counts, while MagicoderS-CL surpasses ChatGPT on HumanEval+ at 66.5 versus 65.9 pass@1.

  • Takeaways & Limitations

    The fully open-sourced models, training data, and source code support future research on open code-generation models and synthetic instruction data.

  • Takeaways & Limitations

    The study focuses on models with no more than 7B parameters, limiting evidence about OSS-INSTRUCT’s impact on larger models; weaker models may also struggle with its sophisticated prompt.

Abstract

from arXiv · show

We introduce Magicoder, a series of fully open-source (code, weights, and data) Large Language Models (LLMs) for code that significantly closes the gap with top code models while having no more than 7B parameters. Magicoder models are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets to generate diverse instruction data for code. Our main motivation is to mitigate the inherent bias of the synthetic data generated by LLMs through the wealth of open-source references for the production of more realistic and controllable data. The orthogonality of OSS-Instruct and other data generation methods like Evol-Instruct further enables us to build an enhanced MagicoderS. Both Magicoder and MagicoderS substantially outperform state-of-the-art code models with similar or even larger sizes on a wide range of coding benchmarks. Notably, MagicoderS-CL-7B based on CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in pass@1 ). Overall, OSS-Instruct opens a new direction for crafting diverse synthetic instruction data for code using abundant open-source references.

1. Introduction

Code generation remains challenging, while synthetic instruction methods can inherit bias from narrow task seeds and heuristics. OSS-INSTRUCT uses open-source code references to generate diverse coding instructions, enabling Magicoder models to achieve strong benchmark performance with open-source models of at most 7B parameters.

  • Motivation: Synthetic coding-instruction methods may inherit LLM and predefined-task bias because they rely on narrow seed tasks, templates, or evolution heuristics.Code Alpaca uses 21 seed tasks, while Code Evol-Instruct relies on five heuristics.
  • OSS-INSTRUCT: OSS-INSTRUCT prompts a powerful LLM to generate coding problems and solutions from randomly selected open-source code snippets.The method uses 75K synthetic examples to fine-tune CodeLlama-Python-7B into Magicoder-CL.
  • OSS-INSTRUCT: 75K synthetic examples are used to fine-tune CodeLlama-Python-7B, producing Magicoder-CL.
  • Results: 66.5 vs. 65.9 pass@1 on HumanEval+ is achieved by MagicoderS-CL against ChatGPT.MagicoderS-CL also reaches 70.7 vs. 72.6 pass@1 on HumanEval and is described as producing more robust code.
  • Results: 76.8 pass@1 on HumanEval is achieved by MagicoderS-DS, which also outperforms DeepSeek-Coder-Instruct-6.7B on HumanEval+ and MBPP+ with 8× less fine-tuning tokens.
  • Contributions: The paper introduces OSS-INSTRUCT as a route to diverse, realistic, and controllable coding instruction data and fully releases model weights, training data, and source code.

2. OSS-INSTRUCT: Instruction Tuning from Open Source

OSS-INSTRUCT generates coding problems and solutions by conditioning a teacher LLM on open-source seed snippets. The approach uses diverse code sources and produces examples spanning realistic programming tasks, while similarity analysis examines how its synthetic data relates to HumanEval.

  • 2. OSS-INSTRUCT: Instruction Tuning from Open Source: OSS-INSTRUCT prompts an LLM to generate a coding problem and solution from a seed code snippet collected from open source.The seed snippet provides generation control and encourages problems reflecting real-world programming scenarios.
  • 2.1. Generating Coding Problems: 80K seed snippets are collected from 80K code documents, including 40K Python snippets and 5K from each of eight other languages.Each snippet contains 1–15 consecutive lines and is passed to a teacher model through a prompt template.
  • 2.2. Data Cleaning and Decontamination: Data cleaning removes identical samples and duplicates sharing seed snippets, while some other noise, such as incomplete solutions, is retained as potentially useful.
  • 2.3. Qualitative Examples of OSS-INSTRUCT: Qualitative examples show that OSS-INSTRUCT can derive realistic machine-learning problems from imports, shell scripts, and incomplete class definitions.The examples illustrate inspiration from distinct code structures and semantics.
  • 2.3. Qualitative Examples of OSS-INSTRUCT: Cosine similarity analysis pairs each of 75K synthetic samples with its closest HumanEval example using TF-IDF embeddings.The study compares OSS-INSTRUCT with Code Alpaca and evol-codealpaca-v1.
  • 2.3. Qualitative Examples of OSS-INSTRUCT: OSS-INSTRUCT has lower average similarity to HumanEval than the other studied data-generation techniques, while SELF-INSTRUCT has the highest average similarity.The authors state that OSS-INSTRUCT’s improvements are therefore not merely due to same-distribution data.

3. Evaluation

The evaluation spans Python, multilingual code generation, and data-science coding tasks, using pass@1 and greedy decoding for core benchmark comparisons. OSS-INSTRUCT-based models consistently improve their base models and often match or exceed larger open-source and proprietary baselines.

  • Evaluation Setup: 75K synthetic samples are used to finetune CODELLAMA-PYTHON-7B and DeepSeek-Coder-Base 6.7B, with MagicoderS adding about 110K Evol-Instruct samples.Magicoder is trained first with OSS-INSTRUCT; MagicoderS continues finetuning on evol-codealpaca-v1.
  • Comparison with DeepSeek-Coder: MagicoderS-DS surpasses DeepSeek-Coder-Instruct-6.7B on all evaluated benchmarks with 8× fewer training tokens and closely matches its 33B counterpart.The comparison uses models derived from DeepSeek-Coder-Base-6.7B and reports superior performance after OSS-INSTRUCT-based finetuning.

4. Ablations of Data Source

The ablations examine how programming-language composition, direct finetuning, and teacher strength affect OSS-INSTRUCT’s gains. Results indicate cross-language transfer, benefits from semantically consistent synthetic instructions, and performance beyond simple teacher distillation.

  • 4.1. Impact of the Language Distribution: 75K training examples comprise approximately 43K Python-only and 32K non-Python samples, classified by whether generated data contains “python” as a substring.
  • 4.1. Impact of the Language Distribution: 10.4 percentage points: Magicoder-CL improves over the base model on Python-only evaluation when trained exclusively on non-Python data.Training on Python or non-Python data substantially boosts corresponding evaluations, while non-Python data also improves out-of-distribution Python performance.
  • 4.1. Impact of the Language Distribution: Combining Python and non-Python data produces a larger Python boost but slightly reduces multilingual performance compared with multilingual-only training.The paper attributes the decrease to Python comprising around 57% of the combined instruction-tuning data.
  • 4.2. OSS-INSTRUCT vs. Direct Finetuning: Finetuning on 75K comment-function pairs worsens the base model, whereas OSS-INSTRUCT provides a substantial improvement.The authors conjecture that noise and inconsistency in the paired data explain the degradation; OSS-INSTRUCT instead translates loosely related fragments into semantically consistent instruction-tuning data.
  • 4.3. OSS-INSTRUCT with A Less Powerful Teacher: Magicoder-CL-Mixtral-7B outperforms both CODELLAMA-PYTHON and its Mixtral-8x7B-Instruct-v0.1 teacher across HumanEval+ and MBPP+.These results suggest OSS-INSTRUCT activates the base model’s capabilities and leverages information from seed snippets beyond teacher distillation.

5. Related Work

Prior work establishes code foundation models and synthetic instruction tuning as routes to stronger code generation, while benchmarks increasingly test functional correctness more rigorously. The related literature also motivates methods that generate instruction data with LLMs.

  • Foundation models for code: Code foundation models are pretrained on billions of lines of code and support software engineering tasks including generation, repair, and testing.
  • Instruction tuning with synthetic data: Instruction tuning finetunes pretrained LLMs on instruction-response pairs, but obtaining high-quality instructional data is laborious.
  • Instruction tuning with synthetic data: SELF-INSTRUCT uses a foundation LLM to generate synthetic instruction-response pairs and then distills the resulting knowledge through instruction tuning.Alpaca and Code Alpaca apply this paradigm to finetune LLaMA with ChatGPT-generated instructions.
  • Evaluating LLMs for code: HumanEval, MBPP, APPS, and CodeContests commonly evaluate single-function generation, while EvalPlus strengthens HumanEval and MBPP with 80× and 35× more tests.The expanded tests address the risk that insufficient tests can produce misleading functional-correctness evaluations.

6. Conclusion and Future Work

The paper concludes that OSS-INSTRUCT enables strong, fully open code models from open-source code references, and that combining it with Evol-Instruct further improves the Magicoder series. Future work targets larger bases and higher-quality data generation.

  • OSS-INSTRUCT generates diverse coding challenges from open-source code snippets and substantially improves Magicoder over its base LLM.Magicoder models use fewer than 7B parameters yet outperform evaluated models up to 16B parameters, including 15B WizardCoder.
  • Combining OSS-INSTRUCT with Evol-Instruct produces enhanced MagicoderS models that rival leading models such as ChatGPT on HumanEval.
  • The authors fully open-source model weights, training data, and source code to support future research in code LLMs.They plan to apply OSS-INSTRUCT to larger base models and improve data quality through seed-code distribution and more advanced teachers such as GPT-4.

Impact Statement

OSS-INSTRUCT uses open-source code to generate diverse and controllable instruction data for code generation and understanding. The paper also recognizes misuse risks, including intentionally generating vulnerable code for exploitation.

  • OSS-INSTRUCT leverages abundant open-source material to generate diverse and controllable synthetic instruction data for code generation and understanding.The authors connect this approach to domain-specific settings where real data may be private and scarce.
  • Deliberate generation of vulnerable code could enable malicious exploitation, so the authors emphasize adherence to ethical guidelines.
  • The method’s prompt template combines a high-level task description, a code snippet, and response guidelines.
  • Qualitative examples include generations inspired by method definitions, shell scripts, library imports, class signatures, code statements, and comments.

A.3. Breakdown of OSS-INSTRUCT Dataset

OSS-INSTRUCT-generated data is organized into ten coding-specific categories and characterized by distributions of problem and solution lengths.

  • Categories: Ten coding-specific categories are manually designed using GitHub topic tags and OctoPack-inspired guidance.INSTRUCTOR embeddings and cosine similarity assign each sample to the categories.
  • Length distribution: Figure 7 plots the token-count distribution of generated problems and solutions.The x-axis measures tokens per problem or solution, while the y-axis counts samples.

B.1. Data Generation

OSS-INSTRUCT generates coding problems and solutions from open-source code snippets, followed by model training and benchmark evaluation across several settings.

  • Data generation: gpt-3.5-turbo-1106 generates self-contained coding problems and correct solutions from randomly extracted 1–15-line code snippets.Greedy decoding is used to maximize consistency between generated problems and solutions.
  • Data generation: Figure 5 illustrates OSS-INSTRUCT-generated problems and solutions from seed snippets, while Figures 6 and 7 summarize categories and token counts.The examples omit detailed requirements, implementations, and explanations.
  • Model training: The models are first finetuned on about 75K synthetic OSS-INSTRUCT examples using CODELLAMA-PYTHON-7B and DeepSeek-Coder-Base 6.7B.Training uses two epochs and two NVIDIA A100-80GB GPUs.
  • Evaluation: MagicoderS-CL-7B outperforms WizardCoder-SC-15B on the selected APPS test subset despite having less than half as many parameters.Magicoder-CL also significantly outperforms its base model and WizardCoder-CL.
  • Evaluation: MagicoderS-DS achieves the best result among evaluated baselines and substantially outperforms DeepSeek-Coder-6.7B-Instruct.This result is reported for the DeepSeek-Coder-based model.
  • Evaluation: Magicoder shows superior DS-1000 insertion-format performance compared with all other evaluated baselines.The experiment uses DeepSeek-Coder and excludes CODELLAMA-PYTHON because it does not support fill-in-the-middle.

C.3. Impact of Removing Noisy Data

Removing partially implemented code from OSS-INSTRUCT is evaluated against retaining noisy samples, alongside examples of improved requirement handling after instruction tuning.

  • Noise removal: Retaining noisy samples improves CODELLAMA-PYTHON-7B pass@1 performance on HumanEval+ relative to removing partially implemented examples.The comparison finetunes both versions for two epochs.
  • Requirement understanding: Magicoder-DS-6.7B checks whether an input is of the correct type, whereas DeepSeek-Coder-6.7B-Base considers only the integer case.The example concerns validating an input for a perfect-cube task.
  • Requirement understanding: Magicoder-DS-6.7B handles the illustrated validity requirement more precisely than the base model.The comparison is specifically described for the perfect-cube example.
  • Requirement understanding: The prompt explicitly adds a requirement to check whether the input is valid.The base prompt asks whether an integer is a cube, while the note adds input validation.
  • Requirement understanding: The tuned implementation rejects invalid inputs before checking whether the number is a perfect cube.The shown conditions reject non-integers and negative values.

D.3. Generalizing to Unseen Tasks

Magicoder-DS-6.7B responds plausibly to unseen coding requests, including corner cases, novel task formats, and multi-component game or application programs.

  • Unseen tasks: Magicoder-DS-6.7B generates a plausible Python coding exercise for an unseen request that the base model cannot interpret.The task asks for an easy Python coding exercise and is not incorporated in OSS-INSTRUCT.
  • Complex programs: MagicoderS-DS-6.7B generates real-world, complex programs such as a Python turtle snake game.The game includes movement, food, scoring, growth, and collision handling.
  • Complex programs: The generated snake game increases the score by 10 points per food item and increases speed as the snake grows.The game ends when the snake hits the border or its own body.
  • Complex programs: The generated Java Othello program uses an 8x8 board, validates moves, flips pieces, and ends when neither player has a valid move.Players enter row and column numbers, and invalid moves prompt another input.
  • Complex programs: The generated Gradio application rotates an input image 45 degrees clockwise and returns a text description of the rotation.The implementation uses PIL for rotation and Gradio for the interface.

E.4. Developing Machine Learning Pipeline with PyTorch

The example develops a PyTorch CNN pipeline for classifying MNIST handwritten digits, training it for five epochs, evaluating test accuracy, and plotting training loss. The described pipeline uses two convolutional layers with standard optimization and evaluation steps.

  • Training: The model uses CUDA when available and trains for 5 epochs with batch size 100 and learning rate 0.001.These hyperparameters are explicitly specified in the example.
  • Training: Cross-entropy loss and Adam optimize the CNN through forward, backward, and parameter-update steps.Each training iteration computes model outputs, evaluates loss, backpropagates gradients, and updates the optimizer.
  • Monitoring: The example records training loss across epochs and plots it as a Training Loss chart.The chart uses epoch on the horizontal axis and loss on the vertical axis.
  • Evaluation: After training, the pipeline evaluates predictions on the test set and prints accuracy for 10,000 test images.The evaluation compares predicted labels with ground-truth labels while disabling gradient computation.
  • Data and model: The pipeline loads MNIST data and defines a CNN with two convolutional layers.The implementation includes data loading, convolutional blocks, and a fully connected output layer.
  • Scope: The demonstrated application is constrained to models with no more than 7B parameters because of resource limitations.The authors state that this prevents observing the method’s impact on larger models.
Loading 2312.02120v2…