PaperlayerView exact revision on arXiv

Source-linked AI summary

To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing

Amir M. Ebrahimi, Mohammed Mehedi Hasan, Aaditya Bhatia, Gopi Krishnan Rajbahadur, Ahmed E. Hassan

arXiv:2607.28887v1cs.SEcs.AIcs.LG

TL;DR

Issue-resolution benchmarks test whether patches resolve issues, not whether they make developer-required deletions. The paper measures deletion behavior across repository repairs, builds a deletion-only benchmark, and finds that models retain required code while deletion-focused training reduces avoidance.

  • Problem

    Issue-resolution benchmarks measure whether patches resolve issues, not whether they make developer-required removals, leaving models’ deletion behavior underexamined.

    Source: §2 · p. 2

  • Method

    The paper compares model patches with developer deletions, tests deletion-sensitive variants, builds the CanItDelete deletion-only benchmark, and pilots deletion-focused post-training.

    Source: §2.1 · p. 2, §1 · p. 2, §3 · p. 3

  • Results

    Across repository repairs and deletion-only tasks, models retain required code; adding 12.8k deletion examples to a 7B model’s post-training mixture cuts CanItDelete deletion avoidance by 13.9 percentage points.

    Source: §1 · p. 2, §7 · p. 7

  • Takeaways & Limitations

    A deletion-training pilot suggests deletion avoidance is underrepresented in code post-training rather than beyond models’ reach.

    Source: §1 · p. 2

    Evidence is bounded: the deletion-sensitive check covers 34 deletion-heavy tasks, and the post-training pilot uses one 7B model, leaving deployment-scale and cross-language transfer open.

    Source: §7 · pp. 7–8

Abstract

from arXiv · show

Large language models increasingly write and repair production code, yet evidence is mounting that their test-passing patches leave codebases harder to maintain. We identify one concrete source: deletion avoidance, the systematic tendency to retain code that an intended edit requires removing. Across the five leading models on the official SWE-bench Verified leaderboard, deletion recall against the developer patch reaches at most 71.7% even on tasks all five solve, and models reach the right file for over 92% of required deletions but cut the exact line in under 52% of cases. Instead, 29.0% of passing patches wrap the targeted code in a guard or fallback, a pattern we call Guard-and-Go. Such patches pass because the original tests rarely check removal: when we retrofit 34 Verified tasks with tests that fail if the targeted code remains, four frontier models spanning closed and open weights fall from 63.2% to 41.9%. Because real repairs mix removal with addition, we curate CanItDelete, a benchmark of 200 tasks mined from real commits whose entire required edit is deletion. Even with the addition work gone, the best model still fails one task in five, and smaller open models fall to 18.0%. We then ablate GPT-5.6 Sol under four cumulative prompts; success moves little until we supply the exact lines, which nearly eliminate incomplete deletion yet raise success only to 80.5% because the model then deletes beyond the spans or adds code instead. Finally, through a pilot study we show one potential fix: teaching deletion during post-training reduces deletion avoidance and improves broader code-editing performance, suggesting the behavior is undertrained rather than beyond reach.

1 Introduction

This introduction frames deletion avoidance as retaining code an intended edit requires removing, then previews evidence from real repairs, a deletion-only benchmark, and a post-training intervention.

Motivation

  • 46.4% of agent-authored fixes are rejected, while maintainers merged already-passing agent pull requests at a rate 24 percentage points below the benchmark score.The cited GitHub corpus reports 46.4% of agent-authored fixes rejected; maintainers reviewing 296 passing pull requests across three SWE-bench Verified repositories cited verbosity and departures from repository conventions.Source: §1 · p. 1

Deletion avoidance in the wild

  • Deletion avoidance is the observable patch behavior of retaining code an intended edit requires removing; Guard-and-Go keeps that code while routing execution around it.In the illustrated task, both the developer and model patches pass the same tests and are recorded as resolved, although the model retains the obsolete assignment under a condition.Source: §1 · p. 1, Figure 1 · p. 1
  • Exact-line deletion rates are 44.6% to 51.6%, Guard-and-Go covers 29.0% of passing patches, and deletion-sensitive tests cut resolution by 21.3 percentage points.Models edit the file containing a required deletion for over 92% of target lines, but remove the exact line in only 44.6% to 51.6% of cases; the retrofit covered four frontier models.Source: §1 · pp. 1–2

CanItDelete: deletion in isolation

  • Across CanItDelete’s twelve models, success runs from 79.0% down to 18.0%, and GPT-5.6 Sol still fails 19.5% of tasks after receiving exact spans.Incomplete deletion dominates failures for ten of twelve models; after exact spans, GPT-5.6 Sol’s remaining failures mostly involve deleting past the boundary or adding code. The benchmark contains 200 deletion-only tasks mined from real commits and uses a deterministic occurrence-aware evaluator.Source: §1 · p. 2

Deletion is learnable

  • Adding 12.8k deletion examples, 0.7% of post-training tokens, cuts CanItDelete deletion avoidance by 13.9 percentage points and raises SWE-bench Verified by 5.3 points and CanItEdit by 1.4.The authors report this as a proof of concept at a single scale.Source: §1 · p. 2

Contributions

  • The work defines and measures deletion avoidance in repository repairs, releases CanItDelete and its diagnostic ladder, and provides measurement code, replication code, and prompts in supplementary material.The materials are withheld from public release for anonymity and available on acceptance.Source: §1 · p. 2

2 Do LLMs Avoid Deleting Code in Practice?

SWE-bench Verified checks whether patches resolve issues, not whether they remove the code developers removed. Across five models, patches often preserve that code—even when tests pass—by routing execution around it with added control flow.

2.1 Study Design

  • The study fixes the OpenHands scaffold, analyzes 254 unanimous-outcome tasks among 377 deletion tasks, and counts a match only when the model deletes the same file and location.The authors treat developer patches as behavioral references rather than unique correct solutions and compare solved versus failed tasks descriptively because failed tasks may be harder for other reasons.Source: §2.1 · p. 2, §2 · p. 2

2.2 Passing Patches Still Leave Developer-Removed Code in Place

  • On SWE-bench Verified, correct-scored patches leave 28.3% to 34.8% of an average task’s reference deletions in place; recall ranges from 65.2% for Kimi-K2 to 71.7% for Opus-4.5.Passing tests therefore narrows divergence from the developer’s deletions without eliminating it.Source: §2.2 · pp. 2–3
  • On solved tasks, models modify the file for 92.5% to 94.4% of reference deletions but remove the exact line in only 44.6% to 51.6% of cases.Exact-line removal lags scope modification by 21.4 to 27.8 percentage points, so localization explains only part of the gap.Source: §2.2 · p. 3
  • On the 57 tasks all five models fail, deletion recall is 19.8% to 30.4%; solved and failed task sets differ at Holm-adjusted p < 10−8 for every model.Cliff’s δ ranges from 0.485 to 0.543, and every model deletes less than the developer on both task sets.Source: §2.2 · p. 3

2.3 Models Substitute Added Control Flow for Removal

  • Among passing pairs, 29.0% use Guard-and-Go to retain developer-removed logic and route execution around it; 190 of 655 failing pairs use the same strategy.Guard-and-Go passes in 72.2% of cases, versus 85.2% for Delete-and-Replace and 39.6% for non-reference alternatives.Source: §2.3 · p. 3
  • Retained Path as Live Fallback accounts for 221 of 550 typed Guard-and-Go pairs (40.2%), while passing Guard-and-Go patches exceed developer patch size in 61.1% of pairs.In this fallback form, the guard handles the reported case while the developer-removed logic remains the default for other inputs; the median patch-size ratio is 1.67×.Source: §2.3 · p. 3

3 Do Passing Tests Detect Missing Deletions?

The section tests whether original suites detect missing removals by retrofitting deletion-sensitive checks onto deletion-heavy SWE-bench Verified tasks.

3 Do Passing Tests Detect Missing Deletions?

  • The retrofit retains 34 tasks from 69 deletion-heavy Verified tasks, validating targets whose removal makes a source-level check fail on the base revision and pass after the developer patch.An AST-aware procedure prioritizes removed conditions, control-flow statements, and complete blocks; checks require the target to be absent from its enclosing scope.Source: §3 · pp. 3–4
  • The experiment uses newly generated patches from GPT-5.6 Sol, Opus 4.8, GLM-5.2, and DeepSeek-V4-Pro rather than the older submissions analyzed earlier.The submissions analyzed earlier predate December 2025, so the experiment tests whether the behavior persists in current systems.Source: §3 · p. 4
  • 63.2% of attempts pass the original suites, but only 41.9% also satisfy the deletion-sensitive check, a decline of 21.3 percentage points.Among attempts accepted by the original suites, 33.7% retain the validated target the developer repair removes; every model declines by 17.6 to 23.5 percentage points.Source: §3 · p. 4, Table 3 · p. 4
  • The retrofit identifies an evaluation gap, but does not establish that deletion is the only valid repair or explain why models fail to remove targets.The 34 tasks are deletion-heavy by construction and do not represent Verified as a whole; full repository tasks also entangle deletion with localization, boundary identification, and implementation work.Source: §3 · p. 4

4 CanItDelete: A Diagnostic Benchmark for Deletion Avoidance

CanItDelete isolates deletion behavior with real-commit tasks whose only required edit is removal. It finds deletion avoidance persists even when models receive exact spans, while scope errors remain.

4.1 Benchmark Design

  • CanItDelete isolates deletion behavior in 200 real-commit tasks requiring only removal, using an evaluator and four cumulative cues to diagnose failures.The tasks emphasize multi-site deletion: each spans at least three separated hunks, and the benchmark includes 151 Python and 49 JavaScript-family tasks. Its evaluator requires the full target to be absent while preserving unrelated executable structure and behavior; the diagnostic modes add cues from explicit deletion intent to exact lines.Source: §4 · p. 4, §4.1 · pp. 4–5, Figure 2 · p. 5

4.2 Deletion Avoidance Persists in Isolation

  • Claude Opus 4.8 leads at 79.0% deletion-compliant success, followed by GPT-5.6 Sol at 74.0%, yet incomplete deletion remains the dominant failure.Across twelve models, deletion-only success ranges from 18.0% to 79.0%, and incomplete deletion accounts for 69.8% of failures; some models instead over-delete or edit out of scope.Source: Figure 3 · p. 5, §4.2 · pp. 5–6

4.3 Models Fail Even When Given the Exact Lines to Delete

  • Exact spans raise success by 6.5–31.5 points and reduce incomplete deletion to 0.6–3.0% for four of five ladder models.Claude Opus 4.8 reaches 97.7% with occurrence-specific spans, but up to 26.0% of tasks still fail after complete target removal; explicit instructions and region pointers move success little.Source: §4.3 · p. 6

5 Can Deletion-Focused Post-Training Reduce Deletion Avoidance?

In a single-model pilot, adding deletion supervision improved targeted deletion completion and transferred most clearly to repository repair, but shifted errors toward over-deletion rather than teaching boundary control.

5 Can Deletion-Focused Post-Training Reduce Deletion Avoidance?

  • A 7B in-house model received 12,821 deletion examples, adding 112.1M tokens to a 15.9B-token code-only mixture, or approximately 0.7% of training.The baseline and intervention used the same recipe; CanItDelete evaluation problems were excluded from training, and results were averaged over three inference runs per benchmark.Source: §5 · p. 6
  • No studied benchmark declines by more than 0.2 points, but increased over-deletion shows that completing deletion and preserving its boundary are distinct training objectives.The pilot signals a potential fix, not a solution; whether the effects hold at deployment scale remains future work.Source: §5 · pp. 6–7
  • SWE-bench Verified rises 5.3 percentage points, CanItEdit rises 1.40, and EditBench changes −0.19; 377 of 500 Verified tasks require deletion from a non-test file.The effect is selective rather than uniform, consistent with a deletion-specific mechanism.Source: §5 · pp. 6–7
  • CanItDelete success rises from 6.5% to 13.7%, while incomplete deletion falls from 80.4% to 66.5%.The 13.9 percentage-point reduction splits into 7.2 points of compliant edits and 6.7 points of complete-but-invalid edits; over-deletion alone rises by 6.2 points.Source: §5 · pp. 6–7, Table 4 · p. 7

6 Related Work

Prior work studies additive bias, test-based evaluation limits, and repository repair, but does not isolate deletion behavior in real repairs. CanItDelete makes purely subtractive edits the target and diagnoses intent, localization, and scope-preservation failures.

Additive bias and deletion behavior

  • Prior studies find additive preferences in people and LLMs, unnecessary action by coding agents, and harmful over-deletion, but not retention of developer-removed code in real repairs.The cited studies use synthetic, no-op, or adjacent editing settings.Source: §6 · p. 7

Validity of test-based code-editing evaluation

  • SWE-bench audits report weak tests, mislabeled passes, and flawed specifications, while deletion-only Kali patches can pass tests without fixing defects; none isolates deletion behavior.EDIT-Bench tests exercise only the edited region in 56% of cases, leaving room for specification gaming.Source: §6 · p. 7

Benchmarks for code editing and repair

  • SWE-bench, instructed-editing benchmarks, and localization studies do not isolate purely subtractive edits; CanItDelete does so with a diagnostic ladder for intent, localization, and scope preservation.Existing benchmarks' reference patches combine addition, modification, and removal.Source: §6 · p. 7

7 Conclusion and Limitations · A Analyzing Deletion Avoidance Quantatitively

Deletion avoidance persists in code models and can make test-passing patches less merge-ready; modest deletion supervision reduces it, but the evidence is limited in scope and scale.

7 Conclusion and Limitations

  • Patches marked resolved by SWE-bench Verified retain a quarter to a third of developers’ deletions, often substituting added control flow for removal.Because tests rarely check whether targeted code was removed, resolution rates can overstate merge-ready behavior.Source: §7 · p. 7
  • Deletion avoidance persists when deletion is the entire task, while supplying exact spans trades retention for over-deletion.The authors characterize this as inadequate control over removal rather than lack of capability.Source: §7 · p. 7
  • Modest deletion supervision reduces deletion avoidance and improves repository-level repair, suggesting the deficit is undertrained rather than intrinsic.Source: §7 · p. 7
  • The in-the-wild analysis uses SWE-bench Verified submissions with uncontrolled decoding settings, and deletion-sensitive testing covers 34 deletion-heavy tasks.Source: §7 · p. 7
  • CanItDelete instructions were drafted with evaluated model GPT-5.6 Sol from most-starred repositories, whose post-edit files may occur in training data.Source: §7 · p. 7
  • The pilot trains one 7B model and reports three-run means without variance; deployment-scale and cross-language effects remain open.Source: §7 · pp. 7–8

A.1 Model Choices for Empirical Analysis … A.5 Patch Size Comparison between Developer Patch and Model Generated Patch with Guard-and-Go

The appendix describes how models were selected and how deletion behavior was measured across patch matching, passing-versus-failing tasks, localization, and patch size.

A.1 Model Choices for Empirical Analysis

  • Models were selected from recent official SWE-bench Verified submissions using the OpenHands scaffold, retaining the latest eligible submission per model family after a July 1, 2025 cutoff.The selected model submissions are summarized in Table 5.Source: §A.1 · p. 10, Table 5 · p. 10

A.2 Deletion Precision and Recall between Developer Patch and Model Patch

  • Table 6 reports mean per-task deletion precision and recall, counting a match only when a generated deletion targets the same base-commit source location as the human patch.Source: Table 6 · p. 10

A.3 Statistical Test for Precision and Recall

  • Table 7 compares deletion recall for all-passed tasks (n = 197) and all-failed tasks (n = 57), reporting medians, IQRs, Holm-adjusted p-values, and Cliff’s δ.Source: Table 7 · p. 10
  • Table 8 compares deletion precision for all-passed tasks (n = 197) and all-failed tasks (n = 57), reporting medians, IQRs, Holm-adjusted p-values, and Cliff’s δ.Source: Table 8 · p. 10

A.4 Exploring the Deletion Avoidance from Localization point of View

  • Figure 5 measures file-, scope-, and exact-line overlap across required deletions in the 197 tasks solved by all five models.Source: Figure 5 · p. 11

A.5 Patch Size Comparison between Developer Patch and Model Generated Patch with Guard-and-Go

  • Table 9 compares model-generated and developer patch size for passing Guard-and-Go pairs, measuring LOC as added plus deleted lines.Source: Table 9 · p. 11

B Construction of the Guard-and-Go Taxonomy · C Construction of Deletion-Focused F2P Tests · C.1 Selecting Behaviorally Important Deletions

The appendices define how Guard-and-Go behaviors were coded and how deletion-focused tests were constructed: they classify guarded retention patterns, select important deletions, and target deletions missed by visible tests.

B Construction of the Guard-and-Go Taxonomy

  • Among 684 pairs where a model retained code deleted by the human patch and added a guard to route around it, only pairs with both lines quoted were analyzed.The classifier had to recover both the retained lines and the added guard to justify each label.Source: B Construction of the Guard-and-Go… · p. 11
  • Two models independently open-coded pairs, and six rounds covering 30 pairs produced a saturated taxonomy of 23 categories after the final two rounds added none.The coders inspected the problem statement, human and model patches, and classifier rationale; an axial coder compared codes after each round.Source: B Construction of the Guard-and-Go… · p. 11, Table 10 · p. 12
  • The authors consolidated 23 categories into ten types, keeping dead-code rerouting distinct from retained code that remains live as a fallback.Claude Opus 4.7 proposed the groupings, which the authors reviewed and finalized before labeling pairs.Source: B Construction of the Guard-and-Go… · p. 11
  • An author checked five examples per type and confirmed structural fit to the definitions, but did not assess patch correctness, maintainability, or intent.The sample comprised 50 pairs across the ten types.Source: B Construction of the Guard-and-Go… · p. 11

C Construction of Deletion-Focused F2P Tests

  • Deletion-focused F2P tests were designed to fail when the targeted deleted code remains and pass when it is removed, targeting behavior not covered by visible tests.The construction first identifies a behaviorally important deletion, then creates a repository-native test.Source: C Construction of Deletion-Focused F2P… · p. 11

C.1 Selecting Behaviorally Important Deletions

  • From 69 tasks with deletions comprising at least 25% of changed lines, the analysis grouped meaningful deleted lines into units and found 166 units across 63 tasks; two tasks had none.Units could represent syntax elements such as conditions, assignments, signatures, or complete compound statements; imports counted only when removed with dependent code.Source: §C.1 · p. 12
  • The highest-scoring unit per task became the target; 24 of 63 selected units were covered by visible tests, leaving 39 candidates for deletion-focused F2P tests.The ranking favored control-flow changes, complete constructs, problem-statement entities, and deletions fully removed by the developer patch; standalone import removals were penalized. A model adopted a unit for patch-level analysis if it removed at least half its lines.Source: §C.1 · p. 12

C.2 Retrofitting and Validating F2P Tests … D.4 Deletion-Compliance Evaluator

The paper builds deletion-focused tests and a 200-task deletion-only benchmark, then specifies a deterministic evaluator for whether edits remove required code without changing unrelated behavior. The benchmark is structurally challenging, and only 34 of 39 candidate tests pass host validation.

C.2 Retrofitting and Validating F2P Tests

  • 34 of 39 candidate deletion tests pass host validation after being checked to fail on base source and pass on developer-patched source.The five excluded candidates lacked a stable source-level predicate that separated the base and gold versions.Source: §C.2 · pp. 12–13

D.1 Candidate Mining and Structural Ranking

  • 200 selected tasks span 35 repositories, with medians of 1,551 pre-edit LOC, 34 deleted lines, 4 deletion hunks, and 8 instruction words.All tasks contain at least three separated deletion hunks; the benchmark includes 151 Python and 49 JavaScript-family tasks, including 53 test files.Source: §D.1 · p. 14, Figure 6 · p. 14

D.2 Instruction Generation and Task Validation

  • GPT-5.6 Sol generates concise deletion-only requests, which pass mechanical checks and LLM and author reviews for grounding, completeness, and recoverable edit sites.The author checks each request against the full before-and-after record to ensure it covers every substantive deletion without broadening scope.Source: §D.2 · p. 14

D.3 Diagnostic Modes

  • The four diagnostic modes are cumulative, adding one controlled signal at each stage to distinguish missing information from execution and scope-control failures.Failure with exact lines isolates execution and scope control after intent, region, and boundaries have been supplied.Source: §D.3 · p. 14

D.4 Deletion-Compliance Evaluator

  • Deletion compliance requires the full target to be absent, remaining executable structure to match the reference, and no behavior-affecting directive or unrelated code change.Formatting, whitespace, and ordinary explanatory comments are accepted, but commenting out or disabling the target is not.Source: §D.4 · p. 14
  • The evaluator tracks target occurrences with diff-derived coordinates and anchors, combining structural parsers and fallbacks with raw-source checks for disabled or preserved targets.Failures are divided into incomplete deletion, when required code remains, and complete deletion with invalid edits, separating removal failure from boundary-preservation failure.Source: §D.4 · p. 14

D.5 Complete Diagnostic-Ladder Results

Across five models, exact-line prompts improve success and sharply reduce incomplete deletion, but they can replace omissions with edits that remove too much. The comparisons use fixed task sets within each model, with Claude Opus 4.8 evaluated on a smaller paired subset.

D.5 Complete Diagnostic-Ladder Results

  • Each model’s within-model comparison holds its task set fixed; Claude Opus 4.8 has usable paired responses for 173 tasks, versus 200 tasks for the other four models.The Opus trajectory therefore uses a smaller subset than the vanilla analysis.Source: §D.5 · p. 14
  • Exact lines improve success by 6.5–31.5 points across all five models; incomplete deletion falls to 0.6–3.0% for four, but remains 17.5% for Qwen3-235B.Explicit deletion changes success by only −2.5 to +2.5 points, while region pointers change it by 0.0–7.0 points.Source: §D.5 · p. 14
  • Complete-deletion invalid edits end at 1.7% for Claude Opus 4.8 and 16.5% for GPT-5.6 Sol, while Qwen3-235B rises from 20.5% to 26.0%.Exact localization can replace omitted deletions with edits that remove the target but change too much; the ladder separates finding every target occurrence from respecting its boundary.Source: §D.5 · pp. 14–15

E Deletion-Focused Training Data Selection · E.1 File-level examples. · E.2 Repository-level examples.

The deletion-focused training data combines 10,000 accepted single-file examples sampled across repositories, languages, and deletion difficulty with 2,821 repository-level examples built from Python-only deletion commits.

E.1 File-level examples.

  • File-level examples sample across repositories, languages, and deletion difficulty rather than retaining only CanItDelete’s most structurally difficult edits.Each example starts from an instruction and the complete pre-edit file.Source: §E.1 · p. 15
  • DeepSeek-V3.2 generates the complete edited file for each file-level example from its instruction and complete pre-edit file.Source: §E.1 · p. 15
  • 10,000 accepted file-level responses have required deletions confined to one file after deterministic rejection sampling under Section 4.1’s criteria.Source: §E.1 · p. 15

E.2 Repository-level examples.

  • Repository-level examples come from non-root, non-merge commits with Python-only deletions that touch at least two files and span 3–1,500 changed lines.Source: §E.2 · p. 15
  • For each repository-level commit, the authors generate an instruction and an F2P test that fails before the reference edit and passes afterward, then package the task in Harbor format.The passage attributes Harbor format to Merrill et al. 2026.Source: §E.2 · p. 15
  • 2,821 repository-level samples are obtained using mini-SWE-agent with MiniMax-M2.7 as teacher and rejection sampling based on generated F2P tests.Source: §E.2 · p. 15

Related papers

Loading 2607.28887v1…