PaperlayerView exact revision on arXiv

Source-linked AI summary

Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions

Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, Arjun Guha

arXiv:2312.12450v6cs.SEcs.AIcs.LGcs.PL

TL;DR

Instructional code editing has received less attention than other code-synthesis tasks, even though users expect Code LLMs to edit code. The paper introduces CANITEDIT and a tailored training dataset; its evaluations find a gap between closed and open models and improved editing performance after fine-tuning.

  • Problem

    Instructional code editing is comparatively understudied despite users expecting Code LLMs to modify code in response to natural-language instructions.

    Source: §1 · p. 1

  • Method

    The paper builds CANITEDIT, a benchmark of hand-crafted editing problems with dual instruction types and hidden tests, and introduces a tailored code-editing training dataset and methodology.

    Source: §1 · p. 2, §3 · p. 4, §6 · p. 10

  • Results

    Closed models significantly outperform open models overall; GPT-4 exceeds DeepSeekCoder-Instruct 33b by an average of 13% in pass@1 for both descriptive and lazy tasks, while fine-tuning improves editing performance across model sizes.

    Source: §5.2 · p. 8, §5.3 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    The benchmark and training approach provide tools and insights for evaluating future improvements in instructional code editing for AI-based software development research and practice.

    Source: §6 · p. 10

    The study targets Python only, so its findings do not establish whether similar results hold for other programming languages.

    Source: §6 · p. 10

Abstract

from arXiv · show

A significant amount of research is focused on developing and evaluating large language models for a variety of code synthesis tasks. These include synthesizing code from natural language, synthesizing tests from code, and synthesizing explanations of code. In contrast, the behavior of instructional code editing with LLMs is understudied. These are tasks in which the model is provided a block of code and an instruction to modify the code. The editing instruction may ask for a feature to be added or removed, describe a bug and ask for a fix, or ask for a different kind of solution. We introduce a carefully crafted benchmark of code editing tasks and use it to evaluate several cutting edge LLMs. Our evaluation exposes a significant gap between the capabilities of state-of-the-art open and closed models. For example, even GPT-3.5-Turbo is better than the best open model at code editing tasks. We also introduce a new, carefully curated, permissively licensed training dataset of code editing tasks coupled with natural language instructions. Using this training dataset, we show that we can fine-tune open Code LLMs to significantly improve their code editing capabilities, closing the gap between open and closed models. All code, data, and models are available at https://github.com/nuprl/CanItEdit.

1 Introduction

Code LLM research has focused more on code synthesis than code editing, despite users expecting models to edit code. The paper defines instructional editing as modifying code from an instruction and introduces a benchmark and training approach for the task.

1 Introduction

  • Instructional code editing maps original code and an instruction to modified code; descriptive instructions are detailed model-generated reflections, while lazy instructions are shorter human-written requests.The two instruction types can differ in detail and in how clearly they express user intent.Source: §1 · p. 2
  • CANITEDIT provides 105 hand-crafted editing problems, descriptive and lazy instructions, and hidden tests for benchmarking Code LLMs.The evaluation measures correct code modifications without unnecessary code.Source: §1 · p. 2
  • The evaluation reports significant performance disparities between leading closed and open models, while fine-tuning open Code LLMs on the proposed dataset improves editing performance.The paper also proposes a training methodology and dataset tailored to code editing.Source: §1 · pp. 2–3
  • The paper proposes ExcessCode to quantify the volume of unused code produced when a model generates correct edits.The authors also report evaluating contemporary Code LLMs and fine-tuning models of varying sizes.Source: §1 · p. 3

2 Related Work

Prior work covers code generation and several forms of code editing, but existing benchmarks leave gaps in evaluating varied instructional editing tasks. This paper positions CANITEDIT as a focused, handcrafted benchmark and reports code-editing training comparisons.

Instruction-following Language Models

  • Commit messages have been used as code-editing instructions, but the authors report that instruction-tuned models are less effective than models explicitly trained for editing.The paper uses commit messages as instructions for code editing.Source: §2 · p. 3

Code Generation Benchmarks

  • Code-generation benchmarks commonly use pass@k but mainly evaluate single-function generation, while HumanEvalFix uses a fixed instruction and SWE-Bench spans broader programming tasks.CANITEDIT focuses specifically on editing and is handcrafted, unlike SWE-Bench’s GitHub pull-request and issue sources.Source: §2 · p. 3

Code Editing Using Large Language Models

  • Prior LLM editing studies focused on bug fixing, fill-in-the-middle completion, or instruction-free editing; instruction-guided editing without fixed locations lacked effective benchmark evaluation.CodeEditorBench uses fixed prompts and lacks diverse topics and dual instruction sets, while InstructCoder’s evaluation raises reproducibility and comparability concerns.Source: §2 · pp. 3–4

3 The CANITEDIT Dataset

CANITEDIT contains 105 Python editing challenges spanning varied computer-science topics and edit types. Each challenge has descriptive and lazy instructions, and the dataset was manually authored and checked with extensive hidden tests.

Benchmark Overview

  • CANITEDIT contains 105 Python code-editing challenges spanning topics such as algorithms, mathematics, language processing, and game programming, with some requiring popular external libraries.Examples of libraries include NumPy, Pandas, and PyTorch.Source: §3 · p. 4
  • The dataset evenly divides its 105 problems among corrective, perfective, and adaptive edits, with 35 problems in each category.The categories respectively cover fixing errors, enhancing features, and meeting new requirements.Source: §3 · pp. 4–5
  • Each problem has descriptive and lazy instructions intended to lead to the same after segment, making 210 instruction-conditioned problem instances.Descriptive instructions provide comprehensive guidance; lazy instructions offer minimal direction and require more inference.Source: §3 · pp. 4–5

Dataset Creation

  • Eight experienced Python programmers created the problems using a shared template and guidance, with lead review and weekly team reviews.The team wrote the before code, hidden tests, after code, and both instruction types for each problem.Source: §3 · p. 5
  • Hidden test suites used varied testing techniques and were verified for 100% line coverage, passing the after code and failing at least one test on the before code.The team also manually reviewed tests and checked instructions and model completions for consistency and clarity.Source: §3 · p. 5

4 Fine-Tuning

The section builds two commit-derived training datasets and fine-tunes Code LLMs for instructional code editing. It selects the combined dataset for final models and describes the filtering, training setup, and dataset statistics.

Fine-tuning setup

  • EDITCODER models use full-parameter fine-tuning with loss calculated only on the after-code segment; the study trains 1.3b, 6.7b, and 33b variants.The 6.7b model is used for ablations as a balance between size and performance; the authors also focus on the DeepSeekCoder-Base family.Source: §4 · p. 5

EditPackFT

  • EditPackFT filters Python CommitPackFT commits for nonempty, changed code, excludes TODO/FIXME/BUG outputs, and requires parseable after-code.The selection process aims to make the dataset more suitable for code editing.Source: §4 · p. 6

Commits2023FT

  • Commits2023FT applies the EditPackFT filters and imperative-verb-plus-noun message criteria to recent data gathered from 416,792 Python file changes.After filtering and retaining one file from multifile commits, it contains 24,129 Python file changes.Source: §4 · pp. 6–7

Dataset statistics

  • Commits2023FT code segments average 119.3 lines versus EditPackFT’s 29.2, and their mean Levenshtein distances are 406.6 versus 197.1 characters.The section describes the larger Commits2023FT changes as more varied.Source: §4 · pp. 6–7, Table 2 · p. 6
  • Commits2023FT messages average 23.1 tokens (standard deviation 35.2) versus EditPackFT’s 10.1-token mean; the datasets contain 199 and 184 unique verbs, respectively.The CommitPackFT statistics are reported as similar before and after the EditPackFT filtering.Source: §4 · pp. 6–7, Table 2 · p. 6

Ablation Datasets

  • Combining Commits2023FT and EditPackFT yields the best results by a significant margin, so the final models are trained on the combined dataset.The authors attribute this result to increased data and expanded length distributions.Source: §4 · p. 7

5 Evaluation

The evaluation measures functional correctness and edit precision across open and closed models, then tests whether fine-tuning improves CANITEDIT performance. Closed models lead open models overall, while EDITCODER substantially improves the open-model results.

5.2 Results with Existing Models

  • 13% higher average pass@1 for GPT-4 than DeepSeekCoder-Instruct 33b on both prompt types; DeepSeekCoder-Instruct is the only open model to exceed any closed model.The authors caution that DeepSeekCoder-Instruct uses an undisclosed instruction-tuning dataset, complicating direct comparisons with other open models.Source: Table 3 · p. 8, §5.2 · p. 8

5.1 Evaluation Metrics

  • pass@1 measures functional correctness, while ExcessCode measures unnecessary changed lines using the fraction not covered by tests among passing completions.The evaluation includes varied open and closed models, each prompted with its recommended template.Source: §5.1 · pp. 7–8, §5 · p. 7

5.2 Results with Existing Models

  • 8.68 higher average pass@1 for descriptive than lazy prompts; larger models generally show lower ExcessCode on lazy instructions.The authors suggest that descriptive prompts' greater detail may contribute to their pass@1 advantage.Source: §5.2 · p. 9
  • Larger models correlate with higher pass@1 and lower ExcessCode, a pattern visible across DeepSeekCoder-Base, StarCoderBase, and StarCoder2.The passage describes this as a correlation, not a causal effect of model size.Source: §5.2 · p. 9
  • StarCoder beats similarly sized DeepSeek and CodeLlama models; OctoCoder trails, while StarCoder2's larger-model gains may reflect training duration or architecture.The authors caution that StarCoder2's results do not establish a benefit from shifting training data from commits to issues.Source: §5.2 · p. 9

5.3 Results after Fine-Tuning on Commits

  • Commits2023FT+EditPackFT is the top-performing tested training dataset for DeepSeekCoder-6.7b-Base on both descriptive and lazy instructions.The ablation compares datasets by token and item counts, pass@1, and ExcessCode for both prompt types.Source: §5.3 · p. 9, Table 4 · p. 9
  • 10.7% higher pass@1 and lower ExcessCode than DeepSeekCoder-Base-33b; EDITCODER-33b surpasses all open models on both instruction types.The largest reported gain over a corresponding base model is 45.1% pass@1 for EDITCODER-6.7b.Source: §5.3 · p. 9
  • Both EDITCODER-33b and EDITCODER-6.7b beat GPT-3.5-Turbo on descriptive pass@1; at higher temperatures, EDITCODER-33b beats it on pass@10 and pass@100 for both instruction types.EDITCODER-33b matches GPT-3.5-Turbo on lazy-instruction pass@1 and surpasses GPT-4 on descriptive pass@100 at higher temperatures.Source: §5.3 · pp. 9–10

6 Conclusion

The paper presents CANITEDIT, a benchmark of 105 hand-written editing problems with lazy and descriptive instructions and comprehensive tests. Its conclusion reports a gap between closed and open models and improved editing performance from fine-tuning; the appendix adds evaluation details and analyses.

6 Conclusion

  • CANITEDIT evaluates instructional code editing with 105 hand-written problems, paired lazy and descriptive instructions, and a comprehensive test suite.The conclusion also reports a significant gap between closed and open models and improved code editing after fine-tuning with a custom dataset and methodology.Source: §6 · p. 10

A Additional Evaluation Details

  • Appendix A adds analyses of higher-sampling results, change types, HumanEvalPack, and StarCoder versions, along with the evaluation prompts.It also provides an ExcessCode implementation.Source: A Additional Evaluation Details · p. 15

A.1 Computing ExcessCode

  • ExcessCode is computed from the number of lines missing code coverage divided by the number of lines changed between the original and modified code.The missing-line count is obtained with a code coverage tool; the paper uses Coverage.py.Source: §A.1 · p. 15

A.2 Results At Higher Sampling Parameters

  • The higher-sampling evaluation uses temperature 0.8, top-p 0.9, and 100 samples, and reports pass@1, pass@10, pass@100, and ExcessCode.It covers fewer models than the main evaluation because of higher computational costs.Source: §A.2 · p. 16

A.2.1 Results

Under higher sampling parameters, open models generally lose pass@1 while benefiting more from multiple trials than closed models. The descriptive–lazy performance disparity persists, and ExcessCode rises for both instruction types.

Results

  • Open models generally lose 2–3% in pass@1 versus the main evaluation, while closed models maintain consistent pass@1 performance.Open-model gains from multiple trials are larger than closed-model gains; EDITCODER-33b exceeds all models, including GPT-4, in pass@100 for descriptive instructions but lags GPT-4 across lazy-instruction metrics and tends to produce more excess code.Source: §A.2.1 · p. 17
  • Lazy instructions gain 57.76% in pass@10 and 22.57% in pass@100, versus 48.18% and 17.23% for descriptive instructions.The performance disparity between descriptive and lazy instructions persists under higher sampling and multiple trials.Source: §A.2.1 · p. 17
  • Average ExcessCode rises to 0.507 for descriptive instructions and 0.449 for lazy instructions, from main-evaluation averages of 0.392 and 0.235, respectively.The paper expects higher sampling to produce a broader range of completions and thus more superfluous code.Source: §A.2.1 · p. 17

A.3 Results Per Change Type

Results vary by change type: GPT-4 leads in functional correctness, while EDITCODER compares favorably with GPT-3.5-Turbo on corrective and adaptive changes but not perfective ones. Corrective changes tend to incur less excess code than adaptive changes.

Results Per Change Type

  • GPT-4 outperforms the other models in functional correctness across all change types.Table 6 aggregates lazy and descriptive prompts across problems and reports pass@1 and ExcessCode by change type.Source: §A.3 · p. 17, Table 6 · p. 17
  • Corrective changes typically incur minimal excess code, whereas adaptive changes tend to introduce the most.Some models achieve perfect scores for corrective changes.Source: §A.3 · p. 17
  • EDITCODER-33b surpasses GPT-3.5-Turbo on corrective and adaptive changes, while EDITCODER-6.7b is comparable; both EDITCODER models underperform on perfective changes.The authors suggest that perfective changes may be underrepresented in training data, noting fewer verbs such as “refactor” and “improve,” and propose balancing the data as future work.Source: §A.3 · p. 18
  • Perfective changes are the most challenging in this dataset, followed by adaptive changes, with corrective changes the simplest.The per-type metrics are aggregated across lazy and descriptive prompts, with each pass@1 and ExcessCode metric calculated across 70 problems per type.Source: §A.3 · pp. 17–18

A.4 Evaluation on HumanEvalPack

The appendix compares CANITEDIT with HumanEvalPack, compares StarCoder generations on CANITEDIT, describes evaluation prompts, and introduces training-process and loss-masking ablations. HumanEvalPack includes single-function generation and corrective tasks, while CANITEDIT targets broader instructional editing.

A.4 Evaluation on HumanEvalPack

  • On HumanEvalPack, EDITCODER-33b outperforms GPT-4 on bug fixing, while EDITCODER-6.7b exceeds larger models such as CodeLlama-Instruct-34b on synthesis.HumanEvalPack comprises 164 single-function problems; HumanEvalFix tests corrective changes, while HumanEvalSynthesize tests code generation, and unlike CANITEDIT’s instructional editing, HumanEvalFix gives no explicit instruction about intent.Source: §A.4 · pp. 18–19
  • CANITEDIT pass@1 and HumanEvalPack Synthesize pass@1 have a Pearson correlation of 0.661 across models with results on both benchmarks.The authors characterize this as a moderate positive correlation and say the benchmarks complement each other: CANITEDIT emphasizes broader software-engineering knowledge across domains.Source: §A.4 · p. 19

A.5 Comparison of StarCoder Models

  • On CANITEDIT, StarCoder2 outperforms StarCoder at 15b and 3b, but StarCoderBase-7b significantly outperforms StarCoder2-7b.At 7b and 3b, StarCoder2 tends to generate more excess code; the authors attribute its improvements to broader training data and architectural enhancements, not issue data’s superiority over commit data.Source: §A.5 · p. 20, Table 8 · p. 20

A.6 Prompt Templates Used in Evaluation

  • Evaluation uses prompts aligned with each model’s training, while keeping prompts as similar as possible; models not trained on commits or explicit editing tasks receive a basic one-shot example.StarCoder2 requires substantial prompt engineering because of the natural-language characteristics of GitHub issue data.Source: §A.6 · p. 20

B Training Details

  • The training-details appendix covers EDITCODER’s training process and ablations of the loss-masking technique.Source: B Training Details · p. 23

B.1 Training Tools and Configuration

EDITCODER models use a fine-tuning pipeline with shuffled, deduplicated examples and loss computed only on the target code. In a comparison on CANITEDIT, loss masking improved performance and training stability.

B.1 Training Tools and Configuration

  • Training examples place the before-code segment, instruction, and after-code segment in sequence, with loss computed only on the after-code segment.The data were randomly shuffled and deduplicated, and models were selected using held-out validation performance.Source: §B.1 · p. 23, §B.2 · p. 23
  • Larger EDITCODER models overfit more quickly, suggesting that a larger training dataset could improve results.Training used HuggingFace Transformers, DeepSpeed ZeRO 3, and one machine with eight NVIDIA H100 GPUs; hyperparameter search might also yield better results.Source: §B.1 · p. 23

B.2 Effect of Loss Masking

  • Loss masking outperformed training without masking on CANITEDIT, decreased ExcessCode and its standard error, and produced more stable, faster-converging training.The comparison used DeepSeekCoder-6.7b-Base trained on EditPackFT.Source: §B.2 · p. 23

C Example CANITEDIT Benchmark Problems

The examples illustrate CANITEDIT's varied editing problems, from reorganizing code to domain-specific and open-ended changes. Completion examples also show that models can add unnecessary code, which ExcessCode measures when solutions pass tests.

C Example CANITEDIT Benchmark Problems

  • CANITEDIT examples span perfective refactoring, domain-specific edits, adaptive strategy design, and corrective Sudoku changes.The Sudoku task adds constraints for nine 3x3 subgrids to both the solver and verifier.Source: C Example CANITEDIT Benchmark Problems · p. 25
  • The OOP refactor turns a single message-formatting function into a multi-class design without adding features, testing both behavior and class hierarchy.The required design includes an abstract Message class, TextMessage and ImageMessage subclasses, and a MessageFactory.Source: C Example CANITEDIT Benchmark Problems · p. 25
  • The group-theory task changes a C4 class to C8 and requires edits across code sections using group knowledge not explained in the prompt.The task's domain-specific changes include the size and elements methods.Source: C Example CANITEDIT Benchmark Problems · p. 25
  • The Tic-Tac-Toe strategy task leaves game rules unstated and requires a new strategy that beats CornerStrategy while leaving existing classes unchanged.Tests check that Game remains intact and that the new strategy consistently wins.Source: C Example CANITEDIT Benchmark Problems · pp. 25, 31
  • The Sudoku task adds missing 3x3-subgrid uniqueness checks to both solver logic and board validation; the existing row and column checks provide cues.The task uses Z3, but the required features can be inferred from the existing code.Source: C Example CANITEDIT Benchmark Problems · pp. 25, 34

D.1 Excess Code Generation

  • An EDITCODER-1.3b completion correctly changes flatten but adds an unwanted unflatten method, earning an ExcessCode score of 66.67.ExcessCode evaluation requires a completion to pass the test suite so line coverage can be assessed.Source: §D.1 · p. 36

D.2 Model-Specific Completion Capabilities

Examples show different model strengths: EDITCODER-33b solves one A* conversion problem, while GPT-4 solves a KL-divergence problem that fine-tuned models do not. Other completions illustrate excess code and algorithmic errors.

D.2 Model-Specific Completion Capabilities

  • EDITCODER-33b solved the A* maze-search problem, while GPT-4 solved a KL-divergence problem that all fine-tuned models failed to solve.The KL-divergence task requires sampling knowledge; GPT-4 also exhibited an error where heuristic values persisted across A* iterations.Source: §D.2 · p. 36, Figure 14 · p. 42, Figure 15 · p. 43
  • GPT-4 is noted for incorrect persistence of heuristic values across A* iterations, while the EDITCODER-33b example converts UCS to A* with Manhattan distance.The conversion adds a standalone Manhattan-distance function.Source: §D.2 · p. 36, Figure 14 · p. 42
  • EDITCODER-1.3b adds an unnecessary unflatten method after changing flatten, while EDITCODER-33b adds an unnecessary doc_len attribute.The latter attribute is not needed because Python can retrieve a list's length directly.Source: Figure 12 · p. 38, Figure 13 · p. 40

E Using LLMs in Code Editing Tasks

The section examines human-written editing requests and model-generated instructions. Human requests are usually informal and terse, while Reflexion-generated instructions can be detailed but may be misleading or more verbose than necessary.

E.1 Human-Instructed Code Editing

  • Of one million conversations, 4,188 contained code-related requests and 831 of those involved code editing; nearly all sampled editing requests were informal and terse.The count relies on searching for markdown-formatted code blocks, so the actual number of code-related requests is likely higher.Source: §E.1 · pp. 44–45

E.2 Model-Generated Instructions for Editing Code

  • Reflexion uses test failures to prompt a model-generated reflection that serves as an instruction for editing the failing program, iterating until tests pass or a stop condition is reached.The process generates and evaluates a candidate program against unit tests before producing a reflection after failure.Source: §E.2 · pp. 45–46
  • The illustrated Reflexion instruction pinpoints a logic issue and suggests an approach, but it is not a complete solution and may be misleading or incorrect.It is more verbose than the human examples, and the text says that this much detail may be unnecessary for the task.Source: §E.2 · p. 46
Loading 2312.12450v6…