PaperlayerView exact revision on arXiv

Source-linked AI summary

OctoPack: Instruction Tuning Code Large Language Models

Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, Shayne Longpre

arXiv:2308.07124v2cs.CLcs.AI

TL;DR

Code LLMs are difficult to use for natural-language tasks such as code explanation, while existing code-instruction benchmarks commonly focus only on synthesis. The paper introduces Git-commit-based instruction tuning and a benchmark spanning three tasks and six languages, producing the best permissively licensed Code LLMs across HumanEvalPack.

  • Problem

    Existing Code LLMs mainly continue code, limiting natural-language tasks, while prior benchmarks commonly cover only code synthesis.

    Source: §1 · p. 2, §3 · p. 4, §5.1 · p. 8

  • Method

    The paper constructs filtered Git-commit instruction data and extends HumanEval to three coding tasks across six programming languages.

    Source: §2 · p. 3, §3 · p. 4

  • Results

    OctoCoder and OctoGeeX achieve the best performance among permissively licensed Code LLMs across HumanEvalPack, while GPT-4 outperforms them.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Git-commit instruction data combined with natural-language data supports stronger permissive Code LLMs across diverse coding tasks.

    Source: §1 · p. 2, §4.1 · p. 6, §6 · p. 9

    HumanEvalPack results should be interpreted carefully because existing decontamination may not hold for all models, including GPT-4.

    Source: §3 · p. 5

Abstract

from arXiv · show

Finetuning large language models (LLMs) on instructions leads to vast performance improvements on natural language tasks. We apply instruction tuning using code, leveraging the natural structure of Git commits, which pair code changes with human instructions. We compile CommitPack: 4 terabytes of Git commits across 350 programming languages. We benchmark CommitPack against other natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B parameter StarCoder model, and achieve state-of-the-art performance among models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2% pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis) across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models, OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among all permissive models, demonstrating CommitPack's benefits in generalizing to a wider set of languages and natural coding tasks. Code, models and data are freely available at https://github.com/bigcode-project/octopack.

1 INTRODUCTION

The paper instruction-tunes Code LLMs using permissively licensed data, introducing CommitPack and HumanEvalPack to improve steerability and evaluate generalization across coding tasks and languages. OctoCoder surpasses openly licensed models, while HumanEvalPack exposes weaknesses that the original HumanEval benchmark may miss.

Motivation

Data

  • The work introduces CommitPack, a 4TB dataset of permissively licensed Git commits spanning 350 programming languages, plus a filtered 2GB variant for high-quality instruction tuning.The paper benchmarks CommitPack alongside xP3x, Self-Instruct data generated with a permissive Code LLM, and OASST.Source: §1 · p. 2

Evaluation

  • HumanEvalPack expands code evaluation to synthesis, repair, and explanation across Python, JavaScript, Java, Go, C++, and Rust.The benchmark is designed to assess Code LLM generalization across a wider range of coding tasks and settings.Source: §1 · p. 2

Results

  • OctoCoder surpasses all other openly licensed models, although GPT-4 remains stronger on Python code synthesis and performs worse on the introduced code-fixing and explanation tasks.GPT-4 scores 86.6% pass@1 on Python HumanEval, while its weaker performance on HumanEvalPack motivates more challenging evaluation variants.Source: §1 · p. 2

2 COMMITPACK: CODE INSTRUCTION DATA

The authors construct CommitPack by exploiting Git commits as naturally paired code changes and human instructions, then filter it into a smaller, higher-quality instruction-tuning set. They compare this resource with three permissive alternatives spanning multilingual, synthetic, and conversational instruction data.

Motivation

  • The dataset design is motivated by evidence that instruction tuning on languages absent from tuning data can improve performance, leading the authors to hypothesize analogous benefits for code.They therefore leverage the natural structure of Git commits, which pair human instructions with code changes.Source: §2 · p. 3

COMMITPACK

  • CommitPack contains almost 4 TB of Git commit data spanning 350 programming languages, while CommitPackFT reduces this to 2 GB across 277 languages through stricter instruction-quality filtering.The filtering retains commercially friendly, single-file commits and selects messages with specific uppercase imperative openings, multiple words, and no external references.Source: §2 · pp. 3–4

COMMITPACKFT

  • For model instruction tuning, the authors sample 5,000 CommitPackFT examples across the six evaluated programming languages, while also exploring pretraining on the full dataset.Figure 2 summarizes the language distributions of CommitPack and CommitPackFT and the task distribution of the 59K-sample Python subset.Source: Figure 2 · p. 3, §2 · p. 4

Alternatives

  • The alternatives are 5,000 permissively licensed code samples from xP3x, 5,003 StarCoder-generated Self-Instruct examples, and 8,587 filtered OASST dialogues.xP3x is drawn from a multilingual collection, Self-Instruct uses the method of Wang et al. (2022a) and StarCoder, and OASST is adapted from a filtered variant of Dettmers et al. (2023).Source: §2 · p. 4

3 HUMANEVALPACK: EVALUATING INSTRUCTION TUNED CODE MODELS

HumanEvalPack expands HumanEval from code synthesis to code fixing, explanation, and synthesis across Python, JavaScript, Java, Go, C++, and Rust. It evaluates all tasks through executable code performance while preserving decontamination validity, with caution for models such as GPT-4.

HumanEvalPack

  • Figure 3 illustrates the three HumanEvalPack scenarios using the first Python HumanEval problem, including a missing “abs” statement bug for HumanEvalFix.The benchmark’s subtasks are HumanEvalFix, HumanEvalExplain, and HumanEvalSynthesize.Source: Figure 3 · p. 4
  • HumanEvalPack covers three code-input/output scenarios—NL+C→C, NL+C→NL, and NL→C—across six programming languages.The benchmark extends HumanEval beyond synthesis to support bug fixing and code explanation.Source: §3 · p. 4
  • HumanEvalFix asks models to repair subtly buggy functions using accompanying unit tests across 984 manually constructed bugs.The dataset contains 164 HumanEval solutions expanded across six languages, with bugs designed to preserve execution while causing test failures.Source: §3 · p. 5
  • HumanEvalExplain evaluates code explanations by regenerating executable code from each model’s explanation rather than scoring text with heuristic metrics.Solution overlap of at least 20 characters is removed, and explanations are constrained to the function docstring’s character length; the docstring itself is withheld.Source: §3 · p. 5
  • HumanEvalSynthesize asks models to generate code from natural-language descriptions and corresponds to the original HumanEval benchmark (Chen et al., 2021).Instruction-tuned models receive an explicit task instruction, while pretrained-only models use the function header and docstring.Source: §3 · p. 5
  • Performance is measured with executable pass@1: the likelihood that one sampled generation solves every test case.For most models, estimates use 20 samples with temperature 0.2 and top-p 0.95; GPT-4 uses one sample, changing Python synthesis from 75.0% to 75.2% while reducing cost 20-fold.Source: §3 · p. 5
  • Manual extension of HumanEval preserves existing decontamination for fair evaluation, but results may remain contaminated for models such as GPT-4.HumanEval’s widespread use has led many training datasets to be decontaminated, though this protection does not necessarily apply to every model.Source: §3 · p. 5

4 OCTOCODER: BEST COMMERCIALLY LICENSED CODE LLM

OctoCoder achieves the strongest overall performance among permissively licensed models, while CommitPackFT and natural-language instruction data improve repair and explanation. Its gains extend across languages and tasks, although targeted repair and output-length control remain difficult.

4.1 ABLATING INSTRUCTION DATA CHOICES

  • Natural-language targets are necessary for code explanation, whereas code-only data causes models to output code and short xP3x targets bias explanations toward brevity.OASST is mixed into ablations because it contains many natural-language targets; mixing xP3x with OASST performs comparatively poorly on explanation.Source: §4.1 · pp. 5–6
  • COMMITPACKFT is critical for code repair, and mixing it with filtered OASST yields the best average score and a 23% absolute improvement over StarCoder.The combination is selected for OCTOCODER, while the same data produces OCTOGEEX from CodeGeeX2.Source: §4.1 · pp. 5–6

4.2 COMPARING WITH OTHER MODELS

  • Instruction tuning generalizes to programming languages unseen in instruction data, and OCTOCODER’s language coverage improves performance on HumanEvalPack’s new languages.Performance after tuning also correlates with pretraining weight: OCTOCODER is strongest on Python, Java, and JavaScript but weakest on low-resource Rust.Source: §4.2 · p. 6, HUMANEVALEXPLAIN · p. 8
  • OCTOCODER has the highest average score across all three evaluation scenarios among permissive models, while 6B-parameter OCTOGEEX outperforms all prior permissive Code LLMs; GPT-4 performs best overall.Table 2 reports zero-shot pass@1 (%) performance across HUMANEVALPACK; GPT-4 is closed-source and likely much larger than the other evaluated models.Source: §4.2 · p. 6, Table 2 · p. 7

HUMANEVALEXPLAIN

  • HUMANEVALFIX is most challenging because models often regenerate buggy code or introduce bugs, whereas OCTOCODER performs comparatively well through small, targeted changes.Removing excess code is the most difficult bug type.Source: HUMANEVALEXPLAIN · p. 8
  • On Python synthesis, OCTOCODER reaches 46.2% pass@1, a relative 38% improvement over StarCoder, while GPT-4 reaches 86.6% and approaches saturation.Models also struggle to switch between code and natural-language explanations and to obey specified character limits.Source: HUMANEVALEXPLAIN · p. 8

5 RELATED WORK

Prior code models target specific tasks or continue code, limiting broader applicability; instruction tuning and human-created multilingual benchmarks address this gap across diverse coding tasks.

5.1 CODE MODELS

  • Instruction tuning broadens code models beyond task-specific heuristics and code continuation to support diverse coding and natural-language tasks.Task-specific models include summarization and editing, while pretrained models are difficult to use for code explanation without instruction following.Source: §5.1 · p. 8

5.2 INSTRUCTION MODELS

  • OctoPack differs from recent instruction-tuned coding models by using commit-based instructions rather than CodeAlpaca’s synthetic OpenAI-generated data, while covering more tasks, languages, and commit data.CoEditor uses 1,650 Python commit-history repositories for auto-editing; this work generalizes commit-based instruction tuning to broader coding tasks.Source: §5.2 · p. 9

5.3 CODE BENCHMARKS

  • HumanEvalPack extends prior translated benchmarks by using human-created tests, correct solutions, buggy solutions, and metadata across six languages.Automatic translation is error-prone and translates tests rather than the solutions needed for tasks such as code explanation.Source: §5.3 · p. 9
  • HUMANEVALFIX provides 164 executable buggy functions across six languages, addressing Quixbugs’ limited size, generic problems, and decontamination difficulty.Unlike Python bug datasets without execution, HUMANEVALFIX includes solutions and unit tests for specific HumanEval-derived problems.Source: §5.3 · p. 9
  • Chaining code explanation with code synthesis enables execution-based pass@k evaluation instead of relying on BLEU, METEOR, or other heuristic metrics.Prior code-explanation evaluations used METEOR or BLEU, whereas execution-based evaluation directly tests the synthesized code.Source: §5.3 · p. 9
  • HumanEvalPack produces 18 scores for nine models across six languages and three tasks, while acknowledging limited model coverage and future extensions to languages or tasks.The benchmark may be extended to code efficiency or code classification, but many code-generating models remain unavailable for evaluation.Source: §5.3 · p. 9

6 CONCLUSION … C COMMITPACK AND COMMITPACKFT LANGUAGES

The paper introduces instruction-tuning datasets and a multilingual benchmark for Code LLMs, finding that filtered commit data combined with natural-language data performs best. OctoCoder and OctoGeeX lead permissively licensed models but remain behind closed-source systems such as GPT-4.

6 CONCLUSION

  • COMMITPACK contains 4TB of Git commits across 350 programming languages, while COMMITPACKFT filters this data into 2GB of high-quality instruction-like code.Source: §6 · p. 9
  • HUMANEVALPACK evaluates 3 coding tasks across 6 programming languages, enabling broader assessment of instruction-tuned code models.Source: §6 · p. 9
  • COMMITPACKFT combined with natural-language data achieves the best performance among the ablated instruction-data configurations, while OctoCoder and OctoGeeX are the strongest permissively licensed models but trail GPT-4.Source: §6 · p. 9

A CONTRIBUTIONS

  • The project’s contributors covered dataset construction, quality filtering, ablations, model training and evaluation, data analysis, and paper development.Source: A CONTRIBUTIONS · p. 22

B ARTIFACTS

  • The appendix catalogs the study’s used and produced artifacts.Source: Table 3 · p. 22

APPENDIX

C COMMITPACK AND COMMITPACKFT LANGUAGES

  • The appendix reports the programming-language distribution of COMMITPACK and COMMITPACKFT, including specialized language codes and abbreviations.Source: Table 4 · p. 29

D DATASET CREATION

The authors construct COMMITPACK by filtering GitHub commit metadata and scraping pre- and post-change code, then derive the smaller COMMITPACKFT instruction-focused subset. They also prepare filtered comparison datasets from xP3x and OASST for benchmarking.

COMMITPACK

  • COMMITPACK combines filtered GitHub activity metadata with scraped pre- and post-commit files, yielding approximately 4 terabytes after inaccessible repositories are discarded.The source archive covers more than 2.8 million repositories and over 145 million unique commits.Source: D DATASET CREATION · p. 29

COMMITPACKFT

  • COMMITPACKFT applies additional quality-oriented filters to COMMITPACK, reducing it by roughly 1000× to nearly 2 gigabytes across 277 languages without detected HumanEval contamination.The authors analyze 56K Python samples and use GPT-4 to classify commits into task categories.Source: D DATASET CREATION · p. 29

xP3x

  • The xP3x comparison subset covers code datasets including APPS, CodeContests, Jupyter Code Pairs, MBPP, XLCoST, Code Complex, Docstring Corpus, Great Code, and State Changes.These datasets are drawn from xP3x (Muennighoff et al., 2022b).Source: D DATASET CREATION · p. 32

OASST

  • The authors reuse a filtered OASST variant from prior work and remove responses that refuse user requests, while noting that its code fraction is likely underestimated.Programming languages and code content are estimated by counting language-marked code fences, missing unmarked or unspecified-language code.Source: D DATASET CREATION · p. 32

E COMPARING DATA BEFORE AND AFTER FILTERING · F COMPARING COMMITPACK AND THE STACK

Filtering COMMITPACK into COMMITPACKFT produces more informative commit messages and substantially more code-change signal per token with fewer repeated code tokens. Compared with The Stack, COMMITPACK covers commit histories from largely overlapping repositories but is larger because it includes successive code changes.

E COMPARING DATA BEFORE AND AFTER FILTERING

F COMPARING COMMITPACK AND THE STACK

G PRETRAINING ON COMMITPACK · H HUMANEVALPACK STATISTICS

COMMITPACK supports both pretraining and instruction-tuning experiments, but its commit format is less universal across coding tasks. HUMANEVALPACK statistics are broadly similar across languages, with Rust as the main exception.

G PRETRAINING ON COMMITPACK

  • COMMITPACK pretraining uses six evaluation languages, 35B filtered tokens repeated nearly four times to 131B tokens, and takes 14 days.The data use the <commit_before>code_before<commit_msg>message<commit_after> code_after format; StarCoder also received parts of COMMITPACK during pretraining.Source: G PRETRAINING ON COMMITPACK · p. 33
  • StarCoder’s commit-format pretraining on HUMANEVALFIX often surpasses instruction-tuned OCTOCODER, while the format is unsuitable for HUMANEVALEXPLAIN.Table 10 reports zero-shot pass@1 (%) performance for these pretraining experiments.Source: G PRETRAINING ON COMMITPACK · p. 33, Table 10 · p. 33
  • SANTACODERPACK matches SantaCoder overall, but performs slightly worse on Python, possibly because multilingual training creates a capacity tax.SANTACODERPACK accommodates Go, C++, and Rust in addition to SantaCoder’s Python, JavaScript, and Java.Source: G PRETRAINING ON COMMITPACK · p. 33
  • SANTACODERPACK’s 1.1B-parameter size may bottleneck performance, motivating further study of pretraining dynamics and unifying pretraining with instruction tuning.Future work could also examine whether code-commit data affect natural-language tasks.Source: G PRETRAINING ON COMMITPACK · p. 33

H HUMANEVALPACK STATISTICS

  • HUMANEVALPACK docstring statistics are broadly similar across languages because docstrings are largely shared, except for Rust.Table 11 summarizes statistics across 164 samples for each language.Source: H HUMANEVALPACK STATISTICS · p. 33, Table 11 · p. 33
  • Rust omits end-of-docstring function-call examples because it is verbose and fully type-annotated, whereas such examples are more useful for untyped JavaScript.Rust’s maximum solution length is especially large, while JavaScript lacks type annotations.Source: H HUMANEVALPACK STATISTICS · p. 33

I FULL INSTRUCTION DATA ABLATIONS · J LINE DIFF FORMAT FOR FIXING CODE

The ablations show that COMMITPACKFT + OASST remains the strongest overall instruction-data mixture, while formatting and target-loss changes trade performance across tasks. For code repair, the proposed line diff format shortens diffs and improves SantaCoder performance across languages, but has insertion and task-scope limitations.

I FULL INSTRUCTION DATA ABLATIONS

  • COMMITPACKFT + OASST performs best overall among the tested instruction-data mixtures, while formatting and target-loss ablations improve some tasks but lower average performance to 35.3 or below the baseline.Formatting helps HUMANEVALFIX but hurts other tasks; target loss is best for HUMANEVALSYNTHESIZE yet has a worse average than full-sequence loss.Source: I FULL INSTRUCTION DATA ABLATIONS · p. 34, Table 12 · p. 34
  • Manual selection of 1,178 high-quality samples does not beat random selection for OCTOCODER but performs better for OCTOGEEX, motivating its use for OCTOGEEX.The selected set contains 725 OASST samples and 89, 61, 86, 72, 70, and 75 COMMITPACKFT samples across Python, JavaScript, Java, Go, C++, and Rust.Source: I FULL INSTRUCTION DATA ABLATIONS · p. 34

J LINE DIFF FORMAT FOR FIXING CODE

  • Unified diff generation still requires unchanged lines around modifications, limiting efficiency, whereas the commit format is mismatched with SantaCoder’s pretraining and lacks special delimiter tokens.SantaCoder was finetuned on Python, Java, and JavaScript because it was pretrained on those three languages.Source: J LINE DIFF FORMAT FOR FIXING CODE · pp. 34–35
  • The line diff format applies changes sequentially and reduces diff length by 70% versus unified diff while retaining only changed lines.It is designed to be unambiguous when applied to pre-commit code and to avoid unchanged-code duplication.Source: J LINE DIFF FORMAT FOR FIXING CODE · p. 35
  • Both diff formats require line-number prediction, so the line-diff finetuning data adds line numbers for copying, slightly reducing efficiency through extra input tokens.This simplifies generation because models need not count line numbers in raw code, but the diff format may converge more slowly than the commit format.Source: J LINE DIFF FORMAT FOR FIXING CODE · p. 35
  • SantaCoder finetuning with the line diff format significantly improves HUMANEVALFIX performance across all languages and outperforms both prior finetuning and the commit format.The comparison is summarized in Table 13; the commit format improves over no finetuning only on JavaScript and Java.Source: J LINE DIFF FORMAT FOR FIXING CODE · p. 35, Table 13 · p. 36
  • The current line diff implementation handles code insertion poorly because inserted lines can shift the line numbers of subsequent lines, and it is unsuitable for explanation and synthesis.Future work could support both line-diff repair and ordinary explanation or synthesis outputs.Source: J LINE DIFF FORMAT FOR FIXING CODE · p. 36

K RESULTS ON HUMANEVALFIXDOCS · L HUMANEVALFIX BUG TYPES

HUMANEVALFIXDOCS evaluates bug repair from docstrings, where OctoCoder performs strongly but StarCoder with its pretrained commit format performs slightly better. HUMANEVALFIX systematically covers multiple manually injected bug types, consistently assigning each problem’s type across programming languages despite implementation differences.

K RESULTS ON HUMANEVALFIXDOCS

  • StarCoder using its pretrained commit format performs slightly better than OctoCoder on HUMANEVALFIXDOCS, while Diff Codegen 2B performs poorly because its predicted diffs are often irrelevant to the bug.HUMANEVALFIXDOCS supplies docstrings as ground truth instead of unit tests, a generally easier setting because models can solve by pure code synthesis without inspecting the buggy function.Source: K RESULTS ON HUMANEVALFIXDOCS · p. 37, Table 14 · p. 37

L HUMANEVALFIX BUG TYPES

  • HUMANEVALFIX bugs were manually added to HumanEval solutions, and each problem retains the same bug type across programming languages despite non-parallel solution code sometimes shifting the affected location.Table 15 provides the overview, with Figures 11–16 giving Python examples.Source: L HUMANEVALFIX BUG TYPES · p. 37, Table 15 · p. 37
  • The benchmark includes missing-logic and excess-logic bugs, exemplified by omitted absolute-value logic and an incorrectly added 1.These examples are shown in Figures 11 and 12.Source: Figure 11 · p. 38, Figure 12 · p. 38
  • It also includes value-misuse and operator-misuse bugs, such as initializing a product to 0 or incorrectly checking equality with 0.These examples are shown in Figures 13 and 14.Source: Figure 13 · p. 38, Figure 14 · p. 39
  • Additional categories are variable misuse and function misuse, exemplified by dividing by the mean and incorrectly using the lower() function.These examples are shown in Figures 15 and 16.Source: Figure 15 · p. 39, Figure 16 · p. 39

M PERFORMANCE BREAKDOWN BY HUMANEVALFIX BUG TYPE … R.8 DIFF CODEGEN

The appendices analyze bug-type performance, alternative explanation evaluation, training and prompting choices, and qualitative successes and failures across models. They show that bug-removal difficulty, evaluation design, prompting, and model-specific behaviors materially shape HumanEvalPack outcomes.

M PERFORMANCE BREAKDOWN BY HUMANEVALFIX BUG TYPE

  • 11% of WizardCoder’s excess-logic fixes succeed, whereas OCTOCODER and OCTOGEEX are more stable across bug types and GPT-4 performs best across all types.The breakdown covers HUMANEVALFIX Python pass@1 performance by bug type.Source: M PERFORMANCE BREAKDOWN BY… · p. 40, Table 16 · p. 40

N HUMANEVALEXPLAIN WITH FILL-IN-THE-MIDDLE

O HUMANEVALEXPLAIN BLEU AND METEOR COMPARISON

  • Pass@k avoids ground-truth explanations, whereas BLEU and METEOR use function docstrings as references; under this setup, Rust receives the highest scores.BLEU and METEOR use the best of 20 generations and average scores across 164 samples per language.Source: O HUMANEVALEXPLAIN BLEU AND METEOR… · p. 41, Table 18 · p. 41

P HYPERPARAMETERS

  • OCTOCODER uses a 5e-4 cosine-scheduled learning rate, batch size 32, one epoch, 35 steps, sequence length 2048, 2.2 million tokens, and LoRA; OCTOGEEX uses 35 steps, batch size 48, and learning rate 5e-5.SantaCoder uses learning rate 5e-5, batch size 64, and up to 200,000 steps, while SANTACODERPACK uses 250k steps, global batch size 64, sequence length 8192, and learning rate 2 × 10^-4.Source: P HYPERPARAMETERS · p. 41

Q PROMPTS

  • Evaluation prompts define an instruction, optional context, and optional function start, combined in model-specific ways using authors’ formats or intuitive defaults without prompt optimization.Source: Q PROMPTS · p. 41

R.1 OCTOCODER; R EXAMPLES

  • OCTOCODER solves examples across code synthesis, explanation, and repair, including a Go explanation-to-solution case where its generated solution is correct, compact, and more efficient than the original.The appendix also presents a correct C++ repair and an incorrect Java synthesis example.Source: Figure 26 · p. 45, §R.1 · pp. 46–47, Figure 27 · p. 46

R.2 GPT-4

  • GPT-4 exhibits both strong repairs and failures, including an unused Go import causing compilation failure, missed or newly introduced bugs, and a successful concise explanation-to-solution example.Its Rust repair adds abs and improves efficiency, while another Python repair misses that n should be squared rather than cubed.Source: Figure 29 · p. 48, Figure 30 · p. 49, Figure 31 · p. 50

R.3 WIZARDCODER; R.4 BLOOMZ; R.5 STARCODER; R.6 INSTRUCTCODET5+; R.8 DIFF CODEGEN

  • WizardCoder leaves a misspelled function unchanged; BLOOMZ omits key task details; StarCoder copies code instead of explaining; InstructCodeT5+ repeats buggy code; and Diff Codegen 2B edits docstrings or imports instead of fixing bugs.StarChat-β is also described as producing excessively long explanations that are truncated by the character limit.Source: Figure 34 · p. 53, Figure 35 · p. 54, Figure 36 · p. 55

S LIMITATIONS AND FUTURE WORK · T VERSION CONTROL · U OCTOBADPACK

The paper identifies execution, multi-file modeling, output-length awareness, and evaluation as important limitations and future-work directions, while proposing commit-derived reward modeling. It also records version-control changes and introduces OCTOBADPACK as OCTOPACK’s “evil brother.”

S LIMITATIONS AND FUTURE WORK

  • Executing generated code to identify failing tests and iteratively repair functions is proposed as a future strategy for improving HUMANEVALPACK performance.The authors leave this exploration to future work.Source: S LIMITATIONS AND FUTURE WORK · p. 58
  • Supporting multi-file inputs and modifications is proposed as a future direction because COMMITPACK filters out commits affecting multiple files.This direction depends on repository-level context and sufficiently long context windows.Source: S LIMITATIONS AND FUTURE WORK · p. 58
  • Current Code LLMs, including OCTOCODER, frequently exceed prescribed character limits in HUMANEVALEXPLAIN.The paper attributes this difficulty partly to instruction-tuning data shaped by human preferences for longer text, even when longer outputs add literary bloat.Source: S LIMITATIONS AND FUTURE WORK · p. 58
  • Evaluation remains challenging because prompting, output processing, programming-language versions, and comprehensiveness can all affect measured code quality.The paper recommends training-format prompts for OCTOCODER and OCTOGEEX, version-aware execution, and future human-aligned evaluation that avoids LLM-evaluator bias.Source: S LIMITATIONS AND FUTURE WORK · pp. 58–59
  • COMMITPACK and COMMITPACKFT could support reward-model training because post-commit code generally represents a human-preferred version of the code.Such reward models could guide language models toward code preferred by humans, following prior preference-learning approaches (Ouyang et al., 2022).Source: S LIMITATIONS AND FUTURE WORK · p. 59

T VERSION CONTROL

  • The version-control section adds appendices on HUMANEVALEXPLAIN metrics and Fill-in-the-Middle, clarifies COMMITPACKFT’s token-limit motivation and languages, fixes StarCoder HUMANEVALFIXDOCS, and records writing improvements.It specifies that CommitPack has 350 languages while CommitPackFT has 277 after filtering.Source: T VERSION CONTROL · p. 59

U OCTOBADPACK

  • Figure 40 presents OCTOPACK alongside its “evil brother” OCTOBADPACK.The figure contrasts the two named entities without supplying further quantitative results.Source: Figure 40 · p. 60

Related papers

Loading 2308.07124v2…