Source-linked AI summary
To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing
Amir M. Ebrahimi, Mohammed Mehedi Hasan, Aaditya Bhatia, Gopi Krishnan Rajbahadur, Ahmed E. Hassan
TL;DR
LLM code patches can pass tests while retaining code that developer repairs remove, but this deletion behavior is poorly measured. The paper evaluates deletion avoidance in real repairs and deletion-only tasks, finding that models substitute added control flow for removal, while modest deletion supervision reduces the behavior.
Problem
SWE-bench measures issue resolution but not whether models remove obsolete logic, leaving their divergence from developer deletions insufficiently characterized.
Method
The paper analyzes five fixed-scaffold SWE-bench submissions, retrofits deletion-sensitive tests, releases the 200-task CanItDelete benchmark, and pilots deletion-focused post-training.
Results
Across repairs and deletion-only tasks, models commonly retain required deletions behind added control flow; adding deletion examples cuts CanItDelete avoidance by 13.9 percentage points.
Takeaways & Limitations
Deletion avoidance appears undertrained rather than intrinsic, and modest deletion supervision improves deletion behavior and broader repository-level code-editing performance.
Takeaways & Limitations
The evidence is bounded by submitted SWE-bench patches, 34 deletion-heavy tasks, GPT-5.6 Sol-drafted CanItDelete instructions, and a single 7B pilot, with broader-scale transfer unresolved.
Abstract
from arXiv · showhide
Large language models increasingly write and repair production code, yet evidence is mounting that their test-passing patches leave codebases harder to maintain. We identify one concrete source: deletion avoidance, the systematic tendency to retain code that an intended edit requires removing. Across the five leading models on the official SWE-bench Verified leaderboard, deletion recall against the developer patch reaches at most 71.7% even on tasks all five solve, and models reach the right file for over 92% of required deletions but cut the exact line in under 52% of cases. Instead, 29.0% of passing patches wrap the targeted code in a guard or fallback, a pattern we call Guard-and-Go. Such patches pass because the original tests rarely check removal: when we retrofit 34 Verified tasks with tests that fail if the targeted code remains, four frontier models spanning closed and open weights fall from 63.2% to 41.9%. Because real repairs mix removal with addition, we curate CanItDelete, a benchmark of 200 tasks mined from real commits whose entire required edit is deletion. Even with the addition work gone, the best model still fails one task in five, and smaller open models fall to 18.0%. We then ablate GPT-5.6 Sol under four cumulative prompts; success moves little until we supply the exact lines, which nearly eliminate incomplete deletion yet raise success only to 80.5% because the model then deletes beyond the spans or adds code instead. Finally, through a pilot study we show one potential fix: teaching deletion during post-training reduces deletion avoidance and improves broader code-editing performance, suggesting the behavior is undertrained rather than beyond reach.
1 Introduction
LLM coding agents often produce patches that run but are not merge-ready because they retain code an intended edit should remove. The introduction defines this deletion avoidance, demonstrates its impact on repository repairs, and presents deletion-only evaluation and post-training as measurement and mitigation paths.
- Motivation: 46.4% of agent-authored fixes in a large GitHub corpus are rejected, while some projects decline AI-generated contributions outright.Running code is distinct from being merge-ready for maintainers.
- Definition: Deletion avoidance is the systematic tendency to retain code an intended edit requires removing, often by routing execution around it conditionally.In the motivating example, both the developer and model patches pass the same tests, but the model retains an obsolete assignment under an ‘else’ guard.
- Empirical evidence: 21.3 percentage points: resolution falls when deletion-sensitive checks are added to 34 deletion-heavy SWE-bench Verified tasks across four frontier models.The retrofit causes 29 of 86 previously passing attempts to fail.
- CanItDelete: 200 tasks comprise CanItDelete, a benchmark mined from real commits whose entire required edit is deletion.It removes confounds from mixed additive-and-removal repairs and evaluates deletion in isolation.
- Mitigation: 13.9 percentage points: adding 12.8k deletion examples to a 7B model’s post-training mixture cuts deletion avoidance on CanItDelete.The 12.8k examples are 0.7% of tokens, while gains transfer to SWE-bench Verified and CanItEdit.
2 Do LLMs Avoid Deleting Code in Practice?
SWE-bench success does not ensure that models reproduce developers’ deletions: they often reach the relevant file but retain removed logic. A recurring substitute is Guard-and-Go, which preserves executable paths behind added control flow, including in passing patches.
- 28.3%–34.8% of reference deletions remain in patches that all five models’ SWE-bench Verified evaluations score as correct.Across 197 solved tasks, mean deletion recall ranges from 65.2% for Kimi-K2 to 71.7% for Opus-4.5.
- 92.5%–94.4% of reference deletions occur in files the models modify, but exact-line removal reaches only 44.6%–51.6%.Models localize the enclosing scope for 68.1%–74.4% of deletions, so coarse localization explains only part of the divergence.
- 19.8%–30.4% deletion recall on the 57 tasks all five models fail shows that deletion avoidance is more pronounced on unsolved tasks.Solved and failed task sets differ at Holm-adjusted p < 10^-8 for every model, with Cliff’s δ between 0.485 and 0.543.
- 29.0% of 1,703 passing model–task pairs use Guard-and-Go, retaining developer-removed logic while routing execution around it.Guard-and-Go passes in 72.2% of cases, below Delete-and-Replace at 85.2% but above non-reference alternatives at 39.6%; 190 of 655 failing pairs use it too.
- 61.1% of passing Guard-and-Go patches exceed their developer counterpart in size, with a median size ratio of 1.67×.The dominant typed form, Retained Path as Live Fallback, accounts for 40.2% of typed Guard-and-Go pairs and preserves the removed path as executable code.
3 Do Passing Tests Detect Missing Deletions?
Retrofitting 34 deletion-heavy SWE-bench Verified tasks with deletion-sensitive checks reveals that passing behavioral tests often fails to detect retained developer-removed code. Across 136 attempts, the stricter criterion reduces resolution by 21.3 percentage points, with 33.7% of originally accepted attempts retaining the validated target.
- Evaluation design: 34 tasks remain after source-level checks require each validated deletion target to be absent from its enclosing scope.Targets are identified from developer-removed conditions, control-flow statements, and complete blocks; checks must fail on the base revision and pass after the developer patch.
- Results: 63.2% of 136 generated attempts pass the original test suites, but only 41.9% also satisfy the deletion-sensitive check.Every model declines under the stricter criterion, with decreases ranging from 17.6 to 23.5 percentage points.
- Results: 33.7% of the 86 attempts accepted by the original suites retain the validated deletion target.The retained target appears despite the patch passing the original behavioral tests.
- Limitations: The experiment measures compliance with developer-specified removal and does not establish that deletion is the only behaviorally valid repair.The 34 tasks are deletion-heavy by construction, and full repository tasks also entangle deletion with localization, boundary identification, and accompanying implementation work.
- Interpretation: The retrofit reduces resolution by 21.3 percentage points, exposing a gap between passing behavioral tests and satisfying an explicit removal requirement.The paired comparison holds each task and model patch fixed under both evaluation criteria.
4 CanItDelete: A Diagnostic Benchmark for Deletion Avoidance
CanItDelete isolates deletion avoidance with 200 deletion-only tasks requiring multi-site removal and evaluates compliance deterministically. Models achieve 18.0–79.0% success, with incomplete deletion dominant, while exact spans reduce retention but expose over-editing.
- Benchmark construction: CanItDelete comprises 200 tasks mined from real commits whose complete required edit is deletion, isolating deletion from localization, replacement, and surrounding implementation work.Every task spans at least three separated deletion hunks, emphasizing multi-site deletion over one-line cleanup.
- Benchmark construction: The benchmark contains 151 Python tasks, 49 JavaScript-family tasks, and 53 tasks modifying test files across 35 repositories.Tasks were selected from the 100 most-starred active repositories in each language and ranked by pre-edit length, deleted lines, and deletion hunks.
- Evaluation: Deletion-compliant outputs must remove every required occurrence, preserve executable structure outside the target, and avoid behavior-affecting or unrelated changes.Commenting out or disabling the target does not count, and the evaluator is deterministic and occurrence-aware.
- Results: Across twelve models, deletion-only success ranges from 18.0% to 79.0%, while incomplete deletion accounts for 69.8% of failures.Claude Opus 4.8 leads at 79.0%, whereas weaker Qwen instruct models and earlier MiniMax releases reach 18.0–47.5%.
- Diagnostic ladder: Exact deletion spans raise success by 6.5–31.5 points and reduce incomplete deletion to 0.6–3.0% for four of five models.Yet invalid edits remain, reaching 26.0% for Qwen3-235B, showing that complete removal and scope preservation are distinct capabilities.
5 Can Deletion-Focused Post-Training Reduce Deletion Avoidance?
In a single-model pilot, adding deletion supervision to general code post-training reduced incomplete deletion and improved deletion-oriented performance. However, boundary control remained a distinct problem, with more over-deletion and no substantial benchmark regressions.
- Intervention: 12,821 deletion examples contributed 112.1M tokens, approximately 0.7% of a 15.9B-token code-only post-training mixture.The intervention used 10,000 file-level edits and 2,821 repository-level repairs, while keeping the training recipe otherwise identical.
- Deletion behavior: CanItDelete success increased from 6.5% to 13.7%, while incomplete deletion fell from 80.4% to 66.5%.The 13.9 percentage-point reduction in incomplete deletion split into 7.2 points becoming compliant edits and 6.7 becoming complete-but-invalid edits; over-deletion rose by 6.2 points.
- Benchmark effects: SWE-bench Verified rose 5.3 percentage points, CanItEdit rose 1.40, and EditBench was unchanged at −0.19.The gain concentrated where removal is part of the task: 377 of 500 Verified tasks require at least one deletion from a non-test file, unlike the instructed-editing benchmarks.
- Limitations: None of the studied benchmarks declined by more than 0.2 points, but over-deletion showed that deletion completion and boundary control are distinct training objectives.The intervention reduced incomplete deletion without eliminating the disposition to act on the code, which surfaced as over-deletion.
6 Related Work
Prior work links deletion avoidance in coding agents to broader human, linguistic, and LLM preferences for additive changes, while also documenting harmful over-deletion. Related studies further question whether passing tests verifies correct edits and establish repository-level, instructed-editing, and localization-informed repair benchmarks.
- Additive bias and deletion behavior: People, English corpora, and LLMs systematically favor additive over subtractive changes, while coding agents may act when inaction is correct.Prior work identifies additive bias across human behavior, corpus statistics, controlled LLM tasks, and coding-agent behavior.
- Additive bias and deletion behavior: LLM patches can also remove unrelated code and break working behavior.This documents an opposite failure mode to deletion avoidance: harmful over-deletion.
- Validity of test-based code-editing evaluation: Audits of SWE-bench report weak tests, mislabeled passes, and flawed task specifications, while 56% of EDIT-Bench tests exercise only the edited region.Narrow coverage can leave room for specification gaming, and deletion-only Kali patches can pass tests without repairing the defect.
- Benchmarks for code editing and repair: SWE-bench and its Verified subset evaluate repository-level repair, instructed-editing benchmarks evaluate natural-language modifications, and prior work finds localization information improves repair.These studies situate the paper among repository-level, instructed-editing, and localization-informed code-repair benchmarks.
7 Conclusion and Limitations · A Analyzing Deletion Avoidance Quantatitively
The paper concludes that deletion avoidance is a recurring limitation in code models: they retain substantial portions of required deletions, often replacing removal with added control flow that tests fail to detect. The evidence is limited by benchmark construction, uncontrolled settings, small-scale piloting, and unresolved generalization questions.
- 7 Conclusion and Limitations: Patches marked resolved by SWE-bench Verified retain a quarter to a third of developer deletions, substituting added control flow for removal.Because tests rarely check deletion, resolution rates can overstate merge-ready behavior.
- 7 Conclusion and Limitations: Deletion avoidance persists when deletion is the entire task, showing that models lack control over removal rather than deletion capability.Providing exact spans trades retention for over-deletion.
- 7 Conclusion and Limitations: The in-the-wild analysis relies on submitted SWE-bench Verified patches produced under uncontrolled decoding settings.This bounds how broadly the observed behavior can be interpreted.
- 7 Conclusion and Limitations: The deletion-sensitive check covers 34 deletion-heavy tasks, limiting conclusions about deletion avoidance beyond that evaluation.The check is a bounded retrofit rather than a broad test suite.
- A Analyzing Deletion Avoidance Quantatitatively: CanItDelete instructions were drafted with GPT-5.6 Sol from most-starred repositories whose post-edit files may appear in training data.The benchmark therefore carries possible model- and data-selection limitations.
- 7 Conclusion and Limitations: The pilot trains one 7B model and reports three-run means without variance, limiting confidence in the training intervention’s measured effects.The study is a small pilot rather than a deployment-scale evaluation.
- 7 Conclusion and Limitations: Whether deletion supervision holds at deployment scale or across other languages remains open.The supplied limitations explicitly leave both settings unresolved.
A.1 Model Choices for Empirical Analysis … A.5 Patch Size Comparison between Developer Patch and Model Generated Patch with Guard-and-Go
The empirical analysis standardizes model selection around recent OpenHands-based SWE-bench Verified submissions, then evaluates deletion fidelity, task-level associations, localization overlap, and patch-size differences in Guard-and-Go cases. The reported comparisons define how deletion precision and recall, localization, and patch size are measured across model-generated and developer patches.
- A.1 Model Choices for Empirical Analysis: Models were selected from official SWE-bench Verified submissions by retaining each eligible model family’s latest recent OpenHands-based submission.Submissions older than one year were excluded using July 1, 2025, as the cutoff date.
- A.1 Model Choices for Empirical Analysis: The selected submissions include GLM-4.6, GPT-5, Kimi K2, Claude, and Salesforce SAGE, with repository dates identifying the retained versions.The listed repository dates range from July 16, 2025, to November 27, 2025.
- A.2 Deletion Precision and Recall between Developer Patch and Model Patch: Deletion precision and recall are computed per task, with a deletion counted as a match only when it targets the same base-commit source location as the human patch.This definition underlies the comparison between developer patches and model-generated patches.
- A.3 Statistical Test for Precision and Recall: Deletion recall and precision are statistically compared between 197 all-passed and 57 all-failed eligible tasks using medians, IQRs, Holm-adjusted Mann–Whitney tests, and Cliff’s δ.The tables report these summaries separately for recall and precision.
- A.4 Exploring the Deletion Avoidance from Localization point of View: Localization analysis measures file-, scope-, and exact-line overlap across all required deletions in the 197 tasks solved by all five models.The figure isolates increasingly precise levels of overlap for the required deletions.
- A.5 Patch Size Comparison between Developer Patch and Model Generated Patch with Guard-and-Go: Patch-size analysis compares model-generated patches with their corresponding developer patches for passing Guard-and-Go pairs.LOC is measured as the sum of added and deleted lines.
B Construction of the Guard-and-Go Taxonomy · C Construction of Deletion-Focused F2P Tests · C.1 Selecting Behaviorally Important Deletions
The paper constructs a validated Guard-and-Go taxonomy from model patches that retain human-deleted code while routing execution around it, then uses behaviorally important deletions to design deletion-focused tests. The selection pipeline identifies 39 uncovered candidate deletions from 63 tasks for F2P construction.
- B Construction of the Guard-and-Go Taxonomy: 684 task–model pairs retained human-deleted code while adding guards that routed execution around it, forming the Guard-and-Go analysis set.Pairs were kept only when classifier evidence recovered quotes for both the retained lines and added guard.
- B Construction of the Guard-and-Go Taxonomy: 23 categories emerged after six rounds covering 30 pairs, with the final two rounds adding no categories or changing definitions.The resulting taxonomy was treated as saturated before consolidation.
- C Construction of Deletion-Focused F2P Tests: Deletion-focused F2P tests target behaviorally important deletions missed by visible tests and fail when the deleted code remains.Each test is repository-native and passes when the intended deletion is performed.
- C.1 Selecting Behaviorally Important Deletions: 69 tasks initially met the criterion that deletions constituted at least 25% of developer-patch changed lines, and 65 were analyzed.Related deleted lines were grouped into deletion units rather than treated as independent targets.
- C.1 Selecting Behaviorally Important Deletions: 166 deletion units across 63 tasks included meaningful removals such as conditions, control-flow statements, calls, signatures, and compound statements.Standalone imports counted only when removed with dependent code; two tasks had no meaningful code deletion.
- C.1 Selecting Behaviorally Important Deletions: 39 of 63 selected deletion units were uncovered by listed visible tests and therefore formed the candidate pool for deletion-focused F2P construction.A model adopted the selected deletion when it removed at least half of that unit’s lines; 24 units were visibly covered.
C.2 Retrofitting and Validating F2P Tests … D.4 Deletion-Compliance Evaluator
The paper builds deletion-focused tests and the CanItDelete benchmark to distinguish genuine removal from retained or disabled code. It defines a structurally challenging 200-task corpus and a deterministic evaluator that checks occurrence-specific deletion compliance without relying on LLM judges.
- C.2 Retrofitting and Validating F2P Tests: 39 source-level oracles fail on the base version and pass on the developer-patched version when the targeted deletion is absent or present, respectively.The oracle compares source states rather than executing project code.
- C.2 Retrofitting and Validating F2P Tests: The tests compare normalized source lines without importing target code, isolating structural edits from initialization, fixtures, framework settings, and runtime side effects.Complete-file checks handle disappearance, while stable enclosing-function checks handle relocation, rewriting, or duplication.
- C.2 Retrofitting and Validating F2P Tests: Host validation retains tests only when they collect, fail on the base with the intended deletion assertion, and pass on the gold version; harness validation additionally runs them in the full SWE-bench environment.Tests that error, skip, fail collection, or cannot distinguish the two versions are discarded.
- D.1 Candidate Mining and Structural Ranking: 79,074 eligible deletion-only file edits remain after filtering Python, JavaScript, JSX, and JavaScript-module files for additive-free, recoverable, nonempty revisions.The selection permits other files in the same commit to contain additions or modifications.
- D.1 Candidate Mining and Structural Ranking: 200 tasks from 35 repositories are selected by structural challenge without model performance, comprising 151 Python and 49 JavaScript-family tasks, including 53 test files.Every task contains at least three separated deletion hunks.
- D.1 Candidate Mining and Structural Ranking: 1,551 (1,266–1,800) median pre-edit LOC, 34 (29–39) deleted lines, 4 (3–5) deletion hunks, and 8 (6–10) instruction words characterize CanItDelete.These are median (IQR) structural distributions across the benchmark.
- D.2 Instruction Generation and Task Validation: The benchmark tasks use generated present-tense deletion requests validated by both an LLM gate and an author gate for nontriviality, grounding, coverage, site consistency, scope, and recoverable boundaries.GPT-5.6 Sol generates requests from the pre-edit file, reference diff, file path, and commit message; mechanical checks reject malformed or patch-oriented requests.
- D.3 Diagnostic Modes: The four diagnostic modes cumulatively add one controlled signal, so failure after exact lines are supplied isolates execution and scope control.Each stage retains prior guidance, allowing changes between stages to identify missing information.
D.5 Complete Diagnostic-Ladder Results
The complete diagnostic ladder shows that exact-line pointers are the only intervention that consistently improves success and sharply reduces incomplete deletion. Remaining failures reveal a boundary-control deficit: models may remove the target while changing too much.
- Experimental setup: Each within-model comparison holds its task set fixed; Claude Opus 4.8 uses 173 paired tasks, while the other four models use all 200.The Opus trajectory therefore differs from the vanilla analysis only in having fewer usable tasks.
- Diagnostic ladder: Explicit deletion changes success by −2.5 to +2.5 points, region pointers by 0.0–7.0 points, and exact lines by 6.5–31.5 points across models.Exact lines are the only intervention improving all five models.
- Diagnostic ladder: Incomplete deletion falls to 0.6–3.0% for four models after exact-line pointers, but Qwen3-235B retains target code in 17.5% of tasks.The intervention sharply reduces omitted deletions, but not uniformly across models.
- Boundary-control deficit: Claude Opus 4.8 reaches 1.7% complete-deletion invalid edits, whereas GPT-5.6 Sol remains at 16.5%.These failures indicate that removing the target can still produce an invalid edit.
- Boundary-control deficit: For Qwen3-235B, complete-deletion invalid edits rise from 20.5% to 26.0% as incomplete deletion falls, exposing a boundary-control trade-off.Exact localization can replace omitted deletions with edits that remove the target but change too much.
E Deletion-Focused Training Data Selection · E.1 File-level examples. · E.2 Repository-level examples.
The paper constructs deletion-focused training data at file and repository scales using generated edits, deterministic or test-based rejection sampling, and task packaging. File-level sampling yields 10,000 accepted one-file responses, while repository-level mining yields 2,821 Python training samples spanning multi-file deletion-only commits.
- E.1 File-level examples.: 10,000 accepted responses result from sampling across repositories, languages, and deletion difficulty with required deletions confined to one file.DeepSeek-V3.2 generates the complete edited file from the instruction and pre-edit file, followed by deterministic rejection sampling.
- E.1 File-level examples.: DeepSeek-V3.2 generates complete edited files from instructions paired with the complete pre-edit file.The file-level examples are designed to vary across repositories, languages, and deletion difficulty rather than retain only structurally difficult edits.
- E.2 Repository-level examples.: 2,821 samples are obtained for training from non-root, non-merge Python commits containing only deletions and touching at least two files.The commits span 3–1,500 changed lines and are mined for repository-level examples.
- E.2 Repository-level examples.: Each repository-level task includes an instruction, an F2P test that fails before and passes after the reference edit, and Harbor-format packaging.The task construction targets commits whose Python changes contain only deletions.
- E.2 Repository-level examples.: Rejection sampling for repository-level examples uses generated F2P tests and MiniMax-M2.7 as the teacher through mini-SWE-agent.This procedure filters the mined commit-based examples before inclusion in the training set.
- E.2 Repository-level examples.: Table 16 reports complete five-model diagnostic-ladder results as counts with percentages, using 200 tasks per mode for four models and 173 for Claude Opus 4.8.The table is associated with the repository-level examples subsection.