Source-linked AI summary
OctoPack: Instruction Tuning Code Large Language Models
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, Shayne Longpre
TL;DR
Code LLMs are difficult to steer across coding and natural-language tasks, and permissive instruction data is needed without relying on closed-model outputs. The paper builds Git-commit instruction data and a broader six-language benchmark, finding that CommitPackFT mixed with OASST yields the strongest permissive models across the evaluated settings.
Problem
Code LLMs mainly continue code, limiting tasks such as explanation, while existing instruction-tuned models rely on OpenAI-generated data classified here as non-permissive.
Method
The paper constructs CommitPack and filtered CommitPackFT from Git commits, then evaluates instruction-tuned Code LLMs with HumanEvalPack across three tasks and six languages.
Results
CommitPackFT combined with OASST produces the best average score, a 23% absolute improvement over StarCoder, and OctoCoder and OctoGeeX are the best permissively licensed Code LLMs.
Takeaways & Limitations
Git commits provide useful code instructions for generalizing Code LLMs across diverse languages and natural coding tasks.
Takeaways & Limitations
HumanEvalPack may not remain decontaminated for all models, including GPT-4, and its benchmark currently covers only limited models and tasks.
Abstract
from arXiv · showhide
Finetuning large language models (LLMs) on instructions leads to vast performance improvements on natural language tasks. We apply instruction tuning using code, leveraging the natural structure of Git commits, which pair code changes with human instructions. We compile CommitPack: 4 terabytes of Git commits across 350 programming languages. We benchmark CommitPack against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2% pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis) across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models, OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among all permissive models, demonstrating CommitPack's benefits in generalizing to a wider set of languages and natural coding tasks. Code, models and data are freely available at https://github.com/bigcode-project/octopack.
1 INTRODUCTION
The paper applies instruction tuning to code models using permissively licensed data, motivated by the brittleness of comment-based prompting and concerns about OpenAI-generated training data. It introduces CommitPack and HumanEvalPack, with OctoCoder outperforming other openly licensed models while remaining below GPT-4.
- Instruction tuning may improve Code LLM steerability and support tasks such as natural-language code explanation, which comment-based prompting handles brittly.
- The paper avoids closed-source model outputs and benchmarks xP3x, Self-Instruct, OASST, and CommitPack as permissive instruction-data sources.
- HumanEvalPack covers Code Repair, Code Explanation, and Code Synthesis across Python, JavaScript, Java, Go, C++, and Rust.
- OctoCoder surpasses other openly licensed models but falls short of GPT-4, which performs especially strongly on synthesis and worse on repair and explanation.
- CommitPack provides 4TB of permissively licensed code commits across 350 programming languages, while CommitPackFT is a filtered 2GB instruction-tuning variant.
2 COMMITPACK: CODE INSTRUCTION DATA
The paper constructs CommitPack from filtered Git commits, exploiting commit messages as naturally occurring coding instructions. It creates a smaller high-quality subset for instruction tuning and compares it with alternative instruction datasets.
- Figure 2 displays language distributions for CommitPack and CommitPackFT and task distributions for 59K Python-subset samples classified by GPT-4.
- The authors hypothesize that code can generalize across languages during instruction tuning and therefore build instruction data from Git commits’ natural structure.
- CommitPack contains almost 4TB of code from 350 programming languages after quality, licensing, and single-file-change filters.
- CommitPackFT reduces the collection to 2GB across 277 languages using strict filters for specific, multiword imperative commit messages without external references.
- For instruction tuning, the authors select 5,000 random CommitPackFT samples across the six languages used for evaluation.
- The comparison includes xP3x, Self-Instruct, and OASST, using code subsets, permissive synthetic instructions, or filtered conversational data respectively.
3 HUMANEVALPACK: EVALUATING INSTRUCTION TUNED CODE MODELS
HumanEvalPack broadens code-model evaluation beyond synthesis to repair and explanation across six languages. Its tasks use human-written problems and executable pass@k scoring, with contamination remaining a caveat for some models.
- HumanEvalPack: HumanEvalPack’s benchmark problems are human-created and use pass@k as the evaluation metric.
- HumanEvalPack: Figure 3 presents one Python HumanEval problem in the three scenarios, including a HumanEvalFix bug caused by a missing abs statement.
- HumanEvalPack: HumanEvalPack evaluates HumanEvalFix, HumanEvalExplain, and HumanEvalSynthesize across Python, JavaScript, Java, Go, C++, and Rust.
- HumanEvalFix: HumanEvalFix asks models to repair buggy functions using accompanying unit tests; the authors manually add 984 comparable bugs across six languages.
- HumanEvalExplain: HumanEvalExplain asks models to explain correct functions and regenerate code from their explanations, enabling executable scoring instead of heuristic explanation metrics.
- HumanEvalSynthesize: HumanEvalSynthesize asks models to generate correct code from natural-language docstrings or comments and corresponds to the original HumanEval task.
- Limitations: Existing decontamination supports fair evaluation for manually extended Python HumanEval, but this may not hold for models such as GPT-4.
4 OCTOCODER: BEST COMMERCIALLY LICENSED CODE LLM
COMMITPACKFT substantially improves code repair and, when mixed with OASST, produces OCTOCODER’s strongest overall performance across HumanEvalPack. OCTOCODER leads permissive models across the benchmark’s tasks and languages, while remaining challenged by output-format control and lower-resource languages.
- Instruction-data ablations: COMMITPACKFT is critical for code repair, likely because about 20% of its data consists of bug fixes.
- Instruction-data ablations: Natural-language targets are necessary for code explanation; code-only instruction data makes models produce code instead of explanations.
- Instruction-data ablations: 23% absolute improvement over StarCoder is achieved by COMMITPACKFT+OASST, selected for OCTOCODER’s final training.
- Comparisons with other models: OCTOCODER achieves the highest average score across all three evaluation scenarios among permissive models.
- Comparisons with other models: Instruction tuning on many COMMITPACKFT languages improves generalization to HumanEvalPack’s newly added programming languages.
- Task-specific behavior: 46.2% pass@1 on original Python HumanEval gives OCTOCODER a relative 38% improvement over StarCoder.
5 RELATED WORK
Prior code models often target specific tasks or continue code without reliably following instructions, especially when natural-language output is required. HumanEvalPack addresses evaluation gaps across repair, explanation, and synthesis.
- Code models have been developed for specialized tasks including code summarization and code editing.
- Large-scale pretrained code models primarily continue code, limiting their usefulness for natural-language tasks such as code explanation.
- HumanEval is a standard code-synthesis benchmark, but automatic multilingual extensions can be error-prone because they translate tests rather than solutions.
- Existing code-repair benchmarks have limitations: Python bugs lack execution support, while QuixBugs has only 40 Python and Java samples centered on generic functions.
- Chaining code explanation with synthesis enables execution-based pass@k evaluation instead of heuristic metrics such as BLEU or METEOR.
- HUMANEVALPACK provides 18 scores across 6 languages and 3 tasks, while remaining limited by the small number of models capable of generating code.
6 CONCLUSION
The paper introduces datasets, a multilingual multi-task benchmark, and permissively licensed instruction-tuned Code LLMs. COMMITPACKFT combined with natural-language data performs best among tested instruction datasets, although the models remain below GPT-4.
- The work studies training and evaluation of Code LLMs that follow instructions.
- COMMITPACK contains 4TB of Git commits across 350 programming languages, while COMMITPACKFT is a filtered 2GB instruction-focused variant.
- HUMANEVALPACK is a human-written benchmark spanning 3 tasks across 6 programming languages.
- COMMITPACKFT combined with natural-language data leads to the best performance in the instruction-data ablations.
- OCTOCODER and OCTOGEEX are the best permissively licensed Code LLMs available, but closed-source models such as GPT-4 outperform them.
D DATASET CREATION
COMMITPACK is built from filtered GitHub commit metadata and affected files, then reduced to COMMITPACKFT by selecting commits whose messages resemble specific, high-quality instructions. The study also compares alternative instruction-data sources.
- COMMITPACK is scraped from GitHub commit metadata and pre- and post-commit code files after quality, licensing, and single-file-change filters.
- COMMITPACKFT is further filtered to select commits resembling high-quality instructions, with contamination checks against HumanEval finding no solution or docstring presence.
E COMPARING DATA BEFORE AND AFTER FILTERING
Filtering makes commit messages longer and more informative while shortening code samples to increase training signal per token. COMMITPACK also captures repeated file changes across repository histories.
- Filtering increases the mean lengths of commit subjects and messages, suggesting that COMMITPACKFT messages are more informative.
- Code lengths decrease significantly after filtering because the process limits the number of allowed tokens.
- The filtering objective is to maximize the amount of training signal per token.
- Code before and after a commit is usually largely the same, reflecting small changes between revisions.
- COMMITPACK contains 1,934,255 repositories, and about 49.3% also appear in The Stack.
- Unlike The Stack’s fixed-point raw files, COMMITPACK contains code-file changes as commits, so one file may appear multiple times across its history.
G PRETRAINING ON COMMITPACK
The paper pretrains models on COMMITPACK’s commit format and finds strong HUMANEVALFIX performance, while noting limitations from model size and task universality.
- Pretraining setup: COMMITPACK provides a commit-format pretraining dataset, and SANTACODERPACK uses the same format with a SantaCoder architecture.The format is <commit_before>code_before<commit_msg>message<commit_after>code_after.
- Results: StarCoder’s commit-format pretraining yields very strong HUMANEVALFIX performance, often surpassing instruction-tuned OCTOCODER.The format is evaluated zero-shot on HUMANEVALFIX.
- Limitations: The commit format is not suitable for HUMANEVALEXPLAIN, limiting its universality.
- Results: SANTACODERPACK performs comparably to SantaCoder overall but slightly worse on Python, possibly because multilingual coverage distributes capacity across more languages.The authors hypothesize a multilingual tax from accommodating Go, C++, and Rust.
- Limitations: SANTACODERPACK may be bottlenecked by its 1.1B-parameter model size, motivating further research on pretraining and instruction-tuning integration.
I FULL INSTRUCTION DATA ABLATIONS
The ablations compare instruction-data mixtures and code-difference formats, finding that COMMITPACKFT plus OASST is strongest overall and line diffs improve code fixing across languages.
- Full instruction data ablations: COMMITPACKFT plus OASST is the best-performing instruction-data mixture in the reported StarCoder ablations.Other tested mixtures do not outperform this combination.
- Full instruction data ablations: Formatting aligned with pretraining improves HUMANEVALFIX relative to default formatting but lowers performance on other tasks, producing a worse average score of 35.3.
- Full instruction data ablations: Target-loss masking gives the best HUMANEVALSYNTHESIZE performance, but its average is worse than computing loss over the full sequence.
- Line diff format for fixing code: The unified diff format includes unchanged lines around modifications, limiting efficiency through unnecessary output duplication.
- Line diff format for fixing code: The line diff format requires unambiguous application while minimizing output length by avoiding unchanged code.Line numbers are added during finetuning so models can copy them, though this adds input tokens.
- Line diff format for fixing code: Line-diff finetuning significantly improves HUMANEVALFIX performance across all evaluated languages and outperforms commit-format finetuning.It may converge more slowly because its format differs substantially from raw pretraining code.
- Line diff format for fixing code: The current line-diff implementation handles code insertion poorly because inserted lines can shift the line numbers of subsequent lines.The format is also not useful for explanation or synthesis tasks.
K RESULTS ON HUMANEVALFIXDOCS
HUMANEVALFIXDOCS evaluates fixing buggy functions from docstrings rather than tests, while the benchmark construction uses manually injected bugs across six languages and varied bug types.
- HUMANEVALFIXDOCS: HUMANEVALFIXDOCS provides docstrings as ground truth for fixing buggy functions, an easier setting than solving from unit tests.Models can solve from docstrings through pure code synthesis without inspecting the buggy function.
- Bug types: The benchmark includes missing-logic bugs, such as omitting an abs operation.
- Bug types: It also includes excess-logic bugs, such as incorrectly adding 1 to a result.
- Bug types: Value-misuse bugs include incorrectly initializing a product accumulator to 0 instead of the required value.
- Bug types: Operator-misuse bugs include checking equality with 0 incorrectly.
- Bug types: Variable-misuse bugs include dividing by the mean incorrectly, while function-misuse bugs include using lower() incorrectly.
M PERFORMANCE BREAKDOWN BY HUMANEVALFIX BUG TYPE
The bug-type analysis finds excess-logic errors especially difficult, while the prompting and evaluation setup compares models across explanation and synthesis tasks using standardized formats.
- Bug-type results: Models struggle most with HUMANEVALFIX bugs requiring removal of excess logic.WizardCoder solves 11% of excess-logic bugs while solving about four times more value-misuse bugs.
- Bug-type results: OCTOCODER and OCTOGEEX show more stable performance across bug types than WizardCoder.
- Explanation evaluation: StarCoder can use FIM prompting for explanation generation because it encountered that technique during pretraining.The generated docstring is treated as an explanation-like output.
- Explanation evaluation: Pass@k evaluates generated solutions from explanations, whereas BLEU and METEOR compare explanations against ground-truth docstrings.BLEU and METEOR require a reference explanation.
- Prompting: Prompt construction combines an instruction, optional context, and optional function start, with formats varying by model.The function start helps models complete the function directly.
- Prompting: GPT-4 prompting omits the function start in the default format because the model did not continue reliably from a provided function start.The synthesize explanation prompt includes the function start when the generated description may omit the function name.
- Qualitative examples: Examples include correct and incorrect model outputs across HUMANEVALFIX, HUMANEVALEXPLAIN, and HUMANEVALSYNTHESIZE.The examples show both successful fixes and failures caused by incorrect logic or compilation errors.
- Prompting: The evaluation uses task-specific prompts for Java, Go, explanation, synthesis, and bug fixing.
S LIMITATIONS AND FUTURE WORK
The paper identifies broader model capabilities and evaluation as future-work priorities, including multi-file editing, output-length control, and more robust benchmarking. It also highlights reward-model training as an additional use for commit data.
- Model Execution: Model execution is left for future work as a way to inspect failing tests and iteratively improve code repairs.The proposed strategy would execute provided or generated code, identify failing tests, and revise the function until tests pass.
- Multi-file changes: CommitPack excludes multi-file changes, limiting its scope to commits that modify a single file.The authors suggest supporting repository-level context and multi-file input and modification as future directions.
- Length-awareness: Current Code LLMs, including OctoCoder, frequently exceed requested character limits in HumanEvalExplain.The authors connect this difficulty to the need for length-aware generation and note that human raters may prefer longer text regardless of content.
- Better evaluation: Evaluation remains sensitive to prompting, output processing, and programming-language version compatibility.The authors use model-specific or intuitive prompts, strict stopping and postprocessing, and execution environments matched to the installed language version.
- Reward Models: CommitPack can also support reward-model training because post-commit code generally represents a human-preferred version.A reward model could compare code before and after a commit and learn that the changed version is better.