PaperlayerView exact revision on arXiv

Source-linked AI summary

Diff-XYZ: A Benchmark for Evaluating Diff Understanding

Evgeniy Glukhov, Michele Conti, Egor Bogomolov, Yaroslav Golubev, Alexander Bezzubov

arXiv:2510.12487v2cs.SEcs.LG

TL;DR

End-to-end patch evaluations make it difficult to isolate the effect of diff format from other factors, even though representation can affect generation quality and cost. Diff-XYZ provides controlled tasks for comparing formats, finding that search-replace works best for stronger models while smaller open models benefit little from formatting choices.

  • Problem

    End-to-end evaluations make it difficult to isolate diff-format effects from retrieval, tool use, and distinct patch failure modes.

    Source: §1 · p. 2

  • Method

    Diff-XYZ varies diff representation in a controlled benchmark of apply, anti-apply, and diff-generation tasks using code edits from real commits.

    Source: §1 · p. 2, §8 · p. 9

  • Results

    Search-replace performs best across tasks for stronger models, while smaller open models benefit little from formatting choices.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark offers a reproducible setting for studying model behavior and comparing diff representations to guide representation design.

    Source: §8 · p. 9

    The tasks are simplified proxies, and their performance has no established quantitative link to outcomes in real systems.

    Source: §6 · p. 8

Abstract

from arXiv · show

Reliable handling of code diffs is central to agents that edit and refactor repositories at scale. We introduce Diff-XYZ, a compact benchmark for code-diff understanding with three supervised tasks: apply (old code $+$ diff $\rightarrow$ new code), anti-apply (new code $-$ diff $\rightarrow$ old code), and diff generation (new code $-$ old code $\rightarrow$ diff). Instances in the benchmark are triples $\langle \textit{old code}, \textit{new code}, \textit{diff} \rangle$ drawn from real commits in CommitPackFT, paired with automatic metrics and a clear evaluation protocol. We use the benchmark to do a focused empirical study of the unified diff format and run a cross-format comparison of different diff representations. Our findings reveal that different formats should be used depending on the use case and model size. For example, representing diffs in search-replace format performs best for larger models across most tasks, while structured udiff variants offer similar but slightly weaker performance. In contrast, smaller open models benefit little from any formatting choice. The Diff-XYZ benchmark is a reusable foundation for assessing and improving diff handling in LLMs that can aid future development of diff formats and models editing code. The dataset is published on HuggingFace Hub: https://huggingface.co/datasets/JetBrains-Research/diff-xyz.

1 Introduction

Diff-XYZ isolates diff representation as a factor in code-diff understanding, addressing the difficulty of separating format effects from other factors in end-to-end evaluations. It defines three complementary tasks and reports that search-replace works best for stronger models, while smaller open models remain limited across formats.

1 Introduction

  • Diff representation matters in patch workflows because it can influence generation quality and cost, and models encounter formats such as unified diff, V4A, and search-replace.These formats are used in code-agent and patch-based settings, including tasks where outputs must be applicable and inspectable.Source: §1 · p. 2
  • End-to-end evaluations make format effects hard to isolate because patch outcomes can reflect malformed syntax, incorrect context, or failure to fix the issue.The introduction notes that retrieval and tool use can also affect these evaluations.Source: §1 · p. 2
  • Diff-XYZ varies the diff representation while holding other context fixed, using three tasks that infer the missing element of diff = new code − old code.The tasks test apply, anti-apply, and diff generation as complementary subproblems.Source: §1 · p. 2
  • Search-replace performs best across tasks for stronger models, while smaller open models remain limited regardless of format.Proprietary models consistently outperform open-source models on unified diffs; Claude 4 Sonnet and GPT-4.1 achieve the highest scores.Source: §1 · p. 2

2 Data Collection

Diff-XYZ is built from real code edits in CommitPackFT and represents each example as paired code versions with their diff. Filtering and stratified sampling shape the dataset toward varied edit sizes, hunk counts, languages, and repositories.

2 Data Collection

  • The benchmark contains 1,000 code edits from CommitPackFT, each represented by old code, new code, and the diff between them.The source corpus contains open-source commits with paired before-and-after code and commit metadata.Source: §2 · pp. 2–3
  • Filtering retains single-file commits and excludes binaries, generated or vendor code, and trivial whitespace-only changes; changed files must have 40–1,000 lines in at least one version.The sampling targets 50% single-hunk and 50% multi-hunk edits, with small, medium, and large edits targeted at 40%, 40%, and 20%.Source: §2 · p. 3
  • The dataset has 200 examples each in Python, JavaScript, Java, Kotlin, and Rust.The language selection combines widely used languages with comparatively underrepresented but actively used ones.Source: §2 · p. 3
  • Most edits include both additions and deletions (81.5%); the dataset also includes add-only edits (16.3%) and delete-only edits (2.2%) across 891 repositories.Source: §2 · p. 3

3 Tasks Description

The benchmark tests code reconstruction in both directions and generation of a formatted diff. Its evaluation distinguishes exact and line-level correctness from diff validity and partial correctness.

3.1 Apply Task and Anti-Apply Task

  • Apply generates new code from old code and a diff, while Anti-Apply reconstructs old code from new code and a diff.A substantial score difference between the tasks may indicate that a model overfits one of them.Source: §3.1 · p. 3
  • For Apply and Anti-Apply, evaluation uses whitespace-stripped exact match and line-level IoU.Whitespace-only lines are removed before calculating the metrics.Source: §3.1 · p. 3

3.2 Diff Generation Task

  • Diff Generation produces a specified-format diff from old and new code, and format familiarity may affect performance.The system prompt always describes the format because models may be less familiar with formats beyond unified diff.Source: §3.2 · p. 3
  • Because valid diffs can differ, generated diffs are evaluated through application to old code and with F1-scores over added and deleted lines.Parsing and applying rates, plus exact match and line-level IoU after application, measure validity and resulting-code correctness; addition-line and deletion-line F1 capture partial correctness when application fails.Source: §3.2 · pp. 3–4

4 Unified Diff Evaluation

The unified-diff evaluation tests proprietary and open models with and without explicit format instructions. Strong proprietary models lead, while open models improve with size on applying edits but lag on diff generation.

4 Unified Diff Evaluation

  • The evaluation fixes each task’s user prompt and compares a generic system prompt with one that explicitly defines unified diff format.Source: §4 · p. 4

4.1 Proprietary Models

  • Only the strongest proprietary models solve diff application perfectly; Claude 4 Sonnet makes almost no Apply mistakes, while showing some quality drop on Anti-Apply.GPT-4.1 also performs very well across both tasks, but varies more across prompts.Source: §4.1 · p. 4, Table 1 · p. 5
  • Across proprietary models, Apply Rate nearly matches IoU, indicating that applicable diffs are usually close to the expected result.Source: §4.1 · p. 4, Table 1 · p. 5
  • For proprietary Diff Generation, explicit format instructions narrow outputs and reduce format switching; GPT-4.1 otherwise often emits V4A instead of unified diff.The effect of the system prompt is especially noticeable for GPT-4.1.Source: §4.1 · p. 4, Table 2 · p. 5

4.2 Open Source Models

  • Performance rises with Qwen Coder size: reliable Apply and Anti-Apply results emerge around 7B and the largest approaches GPT-4o, while Diff Generation remains far behind.None of the open-source models achieve comparable performance on Diff Generation.Source: §4.2 · p. 5, Table 3 · p. 6, Table 4 · p. 6

5 Diff Format Exploration

The study compares diff representations under fixed prompts and finds that the strongest format depends on model scale and task. Search-replace leads overall for larger models, while structural choices such as hunk headers and line markers affect reliability.

5 Diff Format Exploration

  • The comparison tests standard udiff, two udiff variants, and search-replace, using a format description and example in each system prompt.Udiff-h relaxes hunk headers; udiff-l replaces single-character markers with explicit tags to address practical generation issues.Source: §5 · p. 6
  • Search-replace is most effective overall, especially for larger models across all three tasks, while smaller open models struggle regardless; the best format also varies by model scale and task step.Structured udiff and udiff-h remain reliable but are consistently outperformed by search-replace.Source: §5 · p. 6, Table 5 · p. 7
  • Search-replace avoids global constraints such as line-number prediction and hunk-length matching, while larger models better locate distinctive anchors and order local edits.Each replacement stands on its own, so an error in one edit does not invalidate the rest.Source: §5 · pp. 6, 8
  • Udiff-l’s explicit tags reduce marker ambiguity, but results suggest its added verbosity increases complexity rather than helping models.Small models may confuse single-character udiff markers with ordinary code characters.Source: §5 · p. 8
  • Standard hunk headers appear to scaffold patch structure and file order; removing their numeric anchors in udiff-h may increase out-of-order hunks and reduce reliability.The application step does not rely on the header numbers, but their presence appears to help models structure patches.Source: §5 · p. 8

6 Limitations and Future Work

The benchmark’s findings have boundaries: its tasks are simplified proxies, its experiments use single-pass greedy decoding, and its representation set covers only popular formats.

6 Limitations and Future Work

  • Diff-XYZ tasks are simplified proxies, and their performance has no established quantitative link to downstream outcomes, so results should not be read as production predictions.Commit message generation, automated bug fixing, and code review are examples of downstream settings left for future study.Source: §6 · p. 8
  • Experiments use non-reasoning models with greedy decoding, excluding tool use, multistep reasoning, sampling, and best-of-n decoding.Reported numbers reflect single-pass behavior and likely are conservative estimates rather than achievable upper bounds.Source: §6 · p. 8
  • The study covers popular edit representations but leaves tree- or AST-based patches, structured search-replace variants, and error-tolerant or partially specified formats unexplored.These alternatives can be integrated into Diff-XYZ without modifying the benchmark.Source: §6 · p. 8

7 Related Work

Existing code-generation and editing benchmarks address synthesis or realistic workflows, but typically do not isolate edit representation. Diff-XYZ complements them by holding task context fixed while studying formats in a lightweight setup.

7 Related Work

  • Code-generation benchmarks test function-level synthesis and correctness but do not address how edits are represented or applied.Examples include HumanEval [9], MBPP [4], and extensions such as HumanEval+ and MBPP+ [19].Source: §7 · p. 8
  • Editing benchmarks capture realistic workflows, but their evaluation mixes retrieval, long-context reasoning, semantic correctness, and patch formatting.SWE-bench evaluates resolving real GitHub issues with patches that apply and pass tests.Source: §7 · p. 8
  • Existing benchmarks generally assume a fixed diff format, while Aider reports format differences without a systematic study of edit representations.Formats in use include normal, context, unified, V4A, and search/replace schemes.Source: §7 · pp. 8–9
  • Diff-XYZ isolates edit representation by holding task context fixed, enabling lightweight evaluation without repository setup or an execution harness.The benchmark targets a capability described as directly impacting downstream systems.Source: §7 · p. 9

8 Conclusion

Diff-XYZ provides focused, specified tasks with automatic metrics for studying LLM code-diff handling. It establishes unified-diff baselines and compares edit representations across models, with trade-offs intended to guide representation design.

8 Conclusion

  • Diff-XYZ pairs focused tasks with specified inputs and targets and automatic metrics, creating a reproducible setting for studying diff-centric workflows.The authors identify downstream tasks, corrupted or partial diff application, and structured code editing as possible extensions.Source: §8 · p. 9
  • Frontier models perform strongly on the provided unified-diff tasks, motivating more challenging instances within the framework.Source: §8 · p. 9
  • The benchmark compares edit representations across models, with the resulting trade-offs intended to guide representation design.The authors say future work will address observed flaws and iterate toward formats that improve faithfulness and applicability.Source: §8 · p. 9

A Inference Details

The appendix contrasts a generic assistant prompt with a prompt that explicitly specifies unified-diff formatting.

A.1 System prompts

A.2 Task Prompts

The appendix shows templates for applying, reversing, and generating diffs, alongside examples of four diff representations used for exploration.

A.2 Task Prompts

A.3 Examples of Diff Formats

  • Figure 6 gives a unified-diff example included in the system prompt for diff exploration.Source: Figure 6 · p. 13
  • Figure 7 gives a udiff-h example included in the system prompt for diff exploration.Source: Figure 7 · p. 13
  • Figure 8 gives a udiff-l example included in the system prompt for diff exploration.Source: Figure 8 · p. 14
  • Figure 9 gives a search-replace example included in the system prompt for diff exploration.Source: Figure 9 · p. 14

B Typical Errors

The examples show GPT-4.1 producing the wrong output type or diff representation under different prompt conditions.

B Typical Errors

  • More than 10% of Apply answers with the format prompt are diffs instead of new code; the answers without that prompt are correct, and Anti-Apply answers are correct with either prompt.Source: Figure 10 · p. 15
  • More than 50% of the illustrated Diff Generation answers use V4A rather than unified diff; the figure contrasts answers without and with the format prompt.Source: Figure 11 · p. 15
Loading 2510.12487v2…