PaperlayerView exact revision on arXiv

Source-linked AI summary

When Models Edit Too Much: On the Fidelity of Minimal Code Edits

Tongyao Zhu, Wei Hern Lim, Min-Yen Kan

arXiv:2609.04061v1cs.SEcs.AIcs.CL

TL;DR

Code-repair evaluations often reward passing tests without measuring whether models preserve existing implementations or make unnecessarily large edits. This paper constructs controlled minimal-repair tasks from corrupted BigCodeBench solutions and finds that over-editing is widespread, while preservation prompting and reinforcement learning improve edit fidelity under the study’s settings.

  • Problem

    Correctness-only evaluation does not capture whether code repairs are minimal, reviewable, and faithful to the original implementation.

  • Method

    The study injects localized AST-level corruptions into 400 BigCodeBench reference solutions, defines the reversal as the known minimal repair, and measures correctness, token edit distance, and cognitive complexity.

  • Results

    Over-editing is widespread; preservation prompting reduces excess Levenshtein distance from 0.195 to 0.131, cuts added cognitive complexity by 26.6%, raises Pass@1 by 2.3 points, and reinforcement learning offers the strongest out-of-domain fidelity trade-off.

  • Takeaways & Limitations

    Edit fidelity is a distinct axis of code-repair quality that can be measured and improved through prompting and post-training.

  • Takeaways & Limitations

    The study mainly uses Python tasks and Qwen training experiments, while its injected minimal reversals are simpler than real repository bugs and multi-file changes.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to edit existing code, but correctness alone is not enough: useful repairs should also be minimal, reviewable, and faithful to the original implementation. We study over-editing, the tendency of a model to rewrite code beyond what is required to fix a bug. We construct an evaluation framework from 400 BigCodeBench problems by injecting controlled AST-level corruptions into reference solutions, giving each repair task a known minimal patch. Across frontier LLMs, over-editing is widespread even among strong models like GPT-5.5: high Pass@1 can coexist with unnecessarily large edits and added cognitive complexity. A preservation instruction substantially reduces this behavior, lowering average excess Levenshtein distance from 0.195 to 0.131, reducing added cognitive complexity by 26.6%, and increasing Pass@1 by 2.3 points. However, these gains do not simply follow from a larger reasoning budget or larger models. We next ask whether minimal editing can be learned directly during post-training. We observe that supervised fine-tuning overfits to seen corruption patterns, whereas reinforcement learning gives the best out-of-domain edit-fidelity and performance-retention trade-off. These results position edit fidelity as a distinct axis of code-repair quality and show that it can be measured and learned.

1 Introduction

The paper argues that code repairs should be evaluated not only for functional correctness but also for minimality and fidelity to existing implementations. It introduces controlled minimal-repair tasks and finds that over-editing is widespread, while preservation prompting reduces unnecessary edits and improves repair performance.

  • Problem: Over-editing is a functionally correct repair that changes more code than the minimal fix requires.A one-line boundary correction can be replaced by a much larger rewrite that still passes the same tests.
  • Findings: Median patch size ranges from 2 to 60 inserted lines among models whose five completions all pass the tests.GPT-5.4 adds 60 lines covering validation, dtype coercion, NaN masking, and resampling to a task requiring only a one-line fix.
  • Motivation: Existing code-repair evaluation emphasizes whether tests pass, but useful maintenance patches must also preserve intent and avoid unnecessary complexity.Minimal patches reduce review burden, diff churn, regression risk outside existing tests, and disruption of implicit design choices.
  • Method: The benchmark injects localized AST-level corruptions into 400 BigCodeBench reference solutions, making the minimal repair known by construction.It evaluates functional success, normalized token-level Levenshtein distance, and added cognitive complexity.
  • Findings: A preservation instruction reduces excess Levenshtein distance from 0.195 to 0.131, lowers added cognitive complexity by 26.6%, and raises Pass@1 by 2.3 points.These results support edit fidelity as a distinct, measurable dimension of code-repair quality.
  • Conclusion: The paper reports that over-editing is widespread but reducible through prompting and post-training, with reinforcement learning improving out-of-domain fidelity while preserving broader coding competence better than supervised fine-tuning.Reasoning effects are model-specific, and larger models do not monotonically produce smaller or simpler passing repairs.

2 Related Work

Prior coding benchmarks mainly measure functional success, while recent work increasingly studies editing, patch precision, and minimal repair. This paper differs by evaluating controlled repairs with known minimal reversals across models, prompts, reasoning variants, and post-training methods.

  • Correctness-centered evaluation: Coding benchmarks primarily evaluate executable or repository-level functional success, such as Pass@1, Pass@k, issue resolution, or test-passing repairs.These settings broaden evaluation context but generally do not directly measure whether edits were larger than necessary.
  • Editing and minimal repair: Recent instructed-editing and repair benchmarks examine modifications to existing code, including superfluous changed lines and patch precision.Examples include HumanEvalFix, CanItEdit, CREF, and localization- or preference-learning approaches.
  • Positioning: This work uses controlled BigCodeBench tasks with known minimal reversals and spans frontier models, prompting conditions, reasoning variants, and post-training methods.The controlled construction distinguishes the paper’s evaluation target from prior approaches that infer or approximate patch minimality.
  • Similarity metrics and constraints: Reference-similarity metrics such as CodeBLEU do not directly determine whether a repair changed more than necessary relative to the buggy input and minimal fix.The paper frames this as a code-specific constraint-following problem for repair systems.

3 Evaluating Over-Edit Behavior

The study evaluates whether coding models repair corrupted functions with the smallest faithful patch, rather than merely producing passing code. Across controlled corruptions, over-editing is common but can be reduced by preservation prompting, while reasoning, scale, and bug type do not provide uniformly reliable safeguards.

  • Evaluation setup: The benchmark injects localized AST-level corruptions into 400 BigCodeBench reference solutions, creating known minimal repair targets.The benchmark contains 400 tasks and 568 corruption applications; 91.8% of gold repairs require at most two token edits, and none touches more than two lines.
  • Evaluation setup: Repairs are evaluated by functional success, excess token-level Levenshtein distance, and added cognitive complexity.Edit-fidelity metrics are computed on passing repairs unless stated otherwise.
  • Frontier model behavior: GPT-5.5 High reaches Pass@1 0.823 while its excess distance exceeds four times Claude Opus 4.7’s, showing that passing repairs can still be substantially nonminimal.The results separate correctness from edit fidelity: models may add validation, restructure computations, or alter surrounding control flow.
  • Prompting: A preservation instruction reduces aggregate excess Levenshtein distance from 0.195 to 0.131, lowers added cognitive complexity by 26.6%, and raises Pass@1 by 2.3 percentage points.The instruction raises Pass@1 in 40 of 50 settings, with the largest reductions among heavy over-editors.
  • Reasoning and scale: Reasoning effects are model-specific, and larger models do not monotonically produce smaller repairs.Under the generic prompt, excess distance rises from 0.108 at 14B to 0.127 at 32B in the Qwen2.5-Coder-Instruct series.
  • Bug types: List operations and conditionals trigger the most over-editing, with slice bounds combining the highest Pass@1, 0.874, and the largest excess distance, 0.353.The authors associate this pattern with ambiguity about preconditions, unsafe indexing, or unstable control flow rather than difficulty alone.

4 Learning Minimal Editing

The paper tests whether minimal editing can be learned during post-training, finding that reinforcement learning generalizes edit fidelity better than supervised fine-tuning while preserving broader coding ability.

  • Setup: The study trains Qwen3 models on corrupted code and evaluates minimal-edit methods on both seen and held-out corruption families.It compares SFT, rejection-sampled SFT, DPO, and RL using functional success, edit distance, and cognitive complexity.
  • Results: RL gives the best out-of-domain trade-off, while SFT nearly solves in-domain corruption but generalizes poorly.SFT Pass@1 drops from 0.932 to 0.458 out of domain; RL achieves nearly the same correctness as DPO with substantially smaller patches.
  • Results: Across Qwen3 sizes from 4B to 14B, RL steadily reduces excess Levenshtein distance while preserving or improving Pass@1.This supports minimal editing as a scalable and transferable repair preference.
  • Results: RL is the only training method that preserves or improves broader LiveCodeBench performance, whereas SFT loses 14.9 points.RL reaches 33.2%, a +0.6-point change from the 32.6% base score, while SFT drops to 17.7%.
  • Results: LoRA recovers most full-parameter RL gains, with edit fidelity continuing to improve through rank 64 even after Pass@1 saturates by rank 16.The result suggests minimal editing is a learnable stylistic preference that can be captured with a relatively small adapter.
  • Results: Reward design exposes a correctness–edit-locality trade-off: correctness-only rewards produce working but bulky edits, while the full reward gives the best balance.Adding cognitive complexity to the reward performs worse in this localized repair setting.
  • Results: On Defects4J single-method Java bugs, RL preserves pass rates while reducing excess Levenshtein distance and raw token edits.The minimal-editing preference transfers to realistic bugs in an unseen language, although absolute repair rates remain low.

5 Discussion

The discussion argues that edit size matters especially in brownfield maintenance, where unnecessary rewrites increase review burden and can hide regressions despite passing tests.

  • Transfer to real bugs: RL preserves pass rates while reducing edit size on single-method Defects4J bugs.This cross-domain result connects minimal editing to realistic human-written Java repairs.
  • Why edit size matters: In brownfield maintenance, a correct but unnecessarily large diff raises review cost and can hide regressions outside the tests.The model should fix the error without rewriting unrelated style or structure.
  • Why edit size matters: The paper therefore treats edit size as an important quality dimension alongside functional correctness.This framing distinguishes maintenance repair from greenfield code generation.

6 Conclusion

The paper concludes that over-editing is widespread and that edit fidelity should be evaluated alongside functional correctness because minimal editing can be learned.

  • Conclusion: Over-editing is widespread, but preservation instructions reduce excess edits and post-training can make minimal editing more durable.RL improves out-of-domain edit fidelity while preserving broader coding competence better than standard supervised fine-tuning.
  • Conclusion: Code-repair models should be evaluated on both functional correctness and edit fidelity.The conclusion frames edit fidelity as a distinct axis of code-repair quality.

Limitations

The evaluation uses controlled function-level corruptions that provide clean minimal-patch ground truth, but this setting is simpler than real repository repair. Scope is also limited by the Python-focused tasks, Qwen training experiments, and small human studies.

  • Scope: Controlled function-level corruptions simplify the benchmark relative to real repository bugs and multi-file changes.The known reversal supplies clean patch-size ground truth.
  • Scope: The main evaluation tasks are in Python, while post-training experiments use the Qwen model family.
  • Human studies: Human validation is limited in scale, using three annotators for metric validation and a single-annotator audit of 100 high-excess repairs.The metric-validation study covers over 100 patch pairs.
  • Transfer: Defects4J transfer suggests generalization to real Java bugs, but absolute repair rates remain low for the trained model sizes.Extending evaluation to repository-level edits, more languages, richer rewards, and larger human studies remains future work.

Ethical Considerations

The paper reports no significant ethical risks and describes responsible-research material in the appendix. Its benchmark methodology uses controlled code corruptions and model-query settings.

  • Ethical considerations: The authors do not identify significant ethical risks from using open-source datasets and evaluation protocols.They also report no other ethical considerations worth mentioning.
  • Documentation: The appendix includes responsible-research statements alongside benchmark construction, validation, and post-training documentation.
  • Benchmark methodology: The benchmark samples one or two controlled corruptions from predefined AST-level corruption families for each evaluation example.
  • Benchmark methodology: The corruption families include operator, bound, initialization, arithmetic, guard, indexing, call-substitution, copy, boolean, numeric, slice, conditional, and range-step changes.

A.2 Benchmark Composition Statistics

The benchmark comprises 400 function-level Python repair tasks built from controlled corruptions, with known minimal reversals and documented prompt and evaluation procedures. Its composition tracks corruption applications, task coverage, and minimal patch sizes.

  • Benchmark composition: 400 BigCodeBench problems form the evaluation set, each retaining a reference solution, executable tests, and a known minimal reversal of injected AST corruptions.
  • Benchmark composition: 568 corruption applications span 400 tasks, with 232 tasks receiving one corruption and 168 receiving two.
  • Composition statistics: The benchmark documents function lengths, minimal-patch sizes, corruption-family distributions, and higher-level semantic categories.
  • Patch sizes: Gold repairs are small: 96.8% affect at most three normalized code lines, 98.0% at most four, and every repair affects one or two lines.
  • Prompting: The two prompt conditions share the system prompt, task description, signature and docstring constraints, and test suite; only a small preservation clause differs.The generic and preservation requests are shown explicitly in the benchmark documentation.
  • Model querying: Evaluation queries use provider-default decoding settings, with documented reasoning controls and audit records for supported reasoning models.

B.1 Metric Sanity Check Against CodeBLEU

The metric sanity check compares token-level excess Levenshtein distance with a CodeBLEU-based alternative using human judgments and qualitative cases. The evidence favors token edit distance for identifying minimal repairs because CodeBLEU can reward broad overlap despite unrelated changes.

  • Metric comparison: CodeBLEU can favor broad lexical or structural overlap even when a repair changes unrelated code, or score a patch as smaller than the known minimal reversal.This is difficult to interpret when the intended local repair is known by construction.
  • Metric comparison: In matched-correct cases, token edit distance prefers smaller patches, whereas CodeBLEU may prefer statement rewrites that preserve long n-grams over identifier-renaming patches.Figure 7 illustrates this disagreement with o4-mini and GPT-4.1 repairs.
  • Metric choice: Token-level Levenshtein distance is used as the main edit-size metric because it directly measures reversal of the injected bug rather than broad similarity to the corrupted program.
  • Human validation: A blinded audit found unnecessary edits in 82.3% of 96 determinate high-excess repairs, while 17.7% were alternative local fixes.The audit sampled 100 correct repairs with high excess distance.
  • Taxonomy validation: The over-editing taxonomy is highly specific: 98.3% of near-minimal repairs receive zero labels, and category prevalence rises smoothly with excess distance.
  • Taxonomy validation: Five frontier models show substantial to almost-perfect agreement on four of five taxonomy categories, with no agreement drop after excluding GPT-5.5 outputs.

C.3 Statistical Robustness of the Preservation-Prompt Effect

Robustness checks show that explicit preservation prompting consistently reduces over-editing while maintaining or improving repair correctness across matched settings, repeated samples, and instruction variants. The appendix also documents the training setup and held-out corruption families used for minimal-edit experiments.

  • C.3 Statistical Robustness of the Preservation-Prompt Effect: The preservation effect appears in 40 of 50 matched generic–explicit settings and is significant under both Wilcoxon signed-rank and sign tests.The reported p-values are 9.88 × 10−8 and 9.26 × 10−6, respectively.
  • C.3 Statistical Robustness of the Preservation-Prompt Effect: Repeated temperature-1.0 sampling on 100 tasks reduces over-editing by 19–32% relative and improves Pass@1 for all three evaluated models.Each model and prompt received eight independent samples per task.
  • C.3 Statistical Robustness of the Preservation-Prompt Effect: Three alternative preservation instructions all reduce over-editing and improve Pass@1 relative to the generic prompt.The variants request the smallest possible patch, impose a three-line edit budget, or ask the model to localize before fixing the bug.
  • C.3 Statistical Robustness of the Preservation-Prompt Effect: Qualitative examples compare each bug’s required local repair with additional behavior introduced by passing but over-edited model repairs.The comparison focuses on unnecessary behavior changes despite functional success.
  • D.1 Minimal-Edit Training Setup: The minimal-edit training appendix records configurations for SFT, rSFT, DPO, and RL, including learning rates, rollout settings, and reward coefficients.RL uses 16 rollouts per example, while the reward combines execution and edit-distance terms with λexec = 0.1 and λedit = 1.0.
  • D.2 Held-Out DeepCoder Corruption Families: The out-of-domain evaluation applies held-out corruption families to DeepCoder samples so transfer is tested beyond the BigCodeBench corruption list.Examples include function swaps, argument-order changes, comparison toggles, and comprehension-filter removal.
  • D.2 Held-Out DeepCoder Corruption Families: Additional held-out families include dictionary iteration changes, None-comparison toggles, string-operation variants, sorting-key changes, and collection-cast swaps.These transformations vary both the semantic operation and the syntactic form of the localized corruption.
  • D.2 Held-Out DeepCoder Corruption Families: Further held-out corruptions alter rounding, comprehension filters, string lookup methods, quantifiers, zip argument order, range endpoints, and negative indices.Together, these families broaden the transfer evaluation across localized Python edits.

D.3 Auxiliary Edit Metrics

The auxiliary analyses test whether edit-fidelity conclusions depend on token-level measurement, corruption count, rollout budget, or preference-learning implementation. Across these checks, the reported ordering and qualitative conclusions remain stable, while the study is bounded by synthetic benchmark corruptions and limited deployment coverage.

  • D.3 Auxiliary Edit Metrics: Two auxiliary edit metrics preserve the same model ordering as excess Levenshtein distance on correct out-of-domain repairs.Excess line diff and excess syntax-tree diff correlate with excess Levenshtein at pooled Spearman ρ = 0.909 and ρ = 0.910, respectively.
  • D.4 Corruption-Count Robustness: Exactly one corruption per program reproduces the qualitative training comparison: SFT collapses out of domain, DPO trades correctness for smaller edits, and RL has the best combined outcome.The replication covers both DeepCoder training data and corrupted BigCodeBench evaluation data.
  • D.5 RL Rollout-Budget Ablation: Reducing RL from 16 to 8 rollouts still leaves it with a better edit-fidelity trade-off than rSFT and DPO.The authors therefore identify the learning objective, rather than candidate budget, as the dominant factor, while noting that no comprehensive hyperparameter search was performed.
  • D.6 Comparison with DPOP: The appendix includes a full comparison with AdaPatcher-style DPOP preference learning trained under the DPO protocol.This comparison is reported as a separate out-of-domain preference-learning baseline.
  • Scope and Limitations: The study focuses on public benchmark programs and synthetic reversible corruptions rather than vulnerability discovery, exploit generation, or production-repository deployment.The paper also warns that passing repairs can alter unrelated behavior or hide regressions outside the test suite.
Loading 2609.04061v1…