PaperlayerView exact revision on arXiv

Source-linked AI summary

GitGoodBench: A Novel Benchmark For Evaluating Agentic Performance On Git

Tobias Lindenbauer, Egor Bogomolov, Yaroslav Zharov

arXiv:2505.22583v1cs.SEcs.AI

TL;DR

Existing SE benchmarks such as SWE-bench focus on programming and do not benchmark agents’ end-to-end Git interaction. GitGoodBench evaluates three Git scenarios using open-source repositories; its GPT-4o baseline with custom tools achieves a 21.11% solve rate on GitGoodBench Lite.

  • Problem

    Existing SE benchmarks focus on programming proficiency and do not evaluate an agent’s end-to-end capacity to interact with Git.

    Source: §1 · p. 1

  • Method

    GitGoodBench benchmarks Merge Conflict Resolution, Interactive Rebase, and Iterative Committing of Changes using permissively licensed Python, Java, or Kotlin repositories.

    Source: §1 · p. 2

  • Results

    21.11% overall solve rate was achieved by the GPT-4o baseline on GitGoodBench Lite.

    Source: §6 · p. 5

  • Takeaways & Limitations

    The results highlight the need to explicitly consider Git when designing software-engineering agents.

    Source: §6 · p. 5

    The benchmark does not include important Git diagnostic workflows such as git bisect.

    Source: §7 · p. 6

Abstract

from arXiv · show

Benchmarks for Software Engineering (SE) AI agents, most notably SWE-bench, have catalyzed progress in programming capabilities of AI agents. However, they overlook critical developer workflows such as Version Control System (VCS) operations. To address this issue, we present GitGoodBench, a novel benchmark for evaluating AI agent performance on VCS tasks. GitGoodBench covers three core Git scenarios extracted from permissive open-source Python, Java, and Kotlin repositories. Our benchmark provides three datasets: a comprehensive evaluation suite (900 samples), a rapid prototyping version (120 samples), and a training corpus (17,469 samples). We establish baseline performance on the prototyping version of our benchmark using GPT-4o equipped with custom tools, achieving a 21.11% solve rate overall. We expect GitGoodBench to serve as a crucial stepping stone toward truly comprehensive SE agents that go beyond mere programming.

1 Introduction

GitGoodBench addresses the absence of an end-to-end benchmark for agents’ Git proficiency with training and evaluation data covering three Git scenarios. Its GPT-4o baseline achieves a 21.11% solve rate.

1 Introduction

  • Existing end-to-end SE benchmarks assess bug fixing, while Git tasks such as Interactive Rebase require reasoning over commit history and dependencies.The introduction identifies no benchmark for evaluating agents’ end-to-end interaction with Git.Source: §1 · p. 1
  • GitGoodBench provides a training corpus and lite and full evaluation sets for end-to-end evaluation of Git tasks.The benchmark is intended to support SE agents that go beyond programming.Source: §1 · p. 1
  • The benchmark covers Merge Conflict Resolution, Interactive Rebase, and Iterative Committing of Changes.For Merge Conflict Resolution, an agent must reproduce the ground-truth merge commit given a set of conflicts.Source: §1 · p. 2
  • 21.11% solve rate was achieved by the GPT-4o baseline equipped with custom tools.The benchmark draws from permissively licensed Python, Java, and Kotlin repositories.Source: §1 · p. 2

2 Related Work

Existing SE benchmarks may allow Git use but assess bug resolution rather than VCS proficiency; earlier work on Git automation has focused on individual subtasks. GitGoodBench evaluates Git tasks across multiple subtasks.

2 Related Work

  • GitGoodBench explicitly evaluates Git proficiency, unlike benchmarks that permit Git use but measure whether an agent resolves a bug.This distinction allows future research to examine and refine VCS-focused strategies in SE agents.Source: §2 · p. 2
  • Earlier work on merge conflict resolution and commit-message generation or completion focused on specific VCS subtasks, whereas GitGoodBench combines multiple subtasks in one scenario.The combined subtasks include commit-message generation, reasoning across commits, and rebase-plan generation.Source: §2 · p. 2

3 GitGoodBench Datasets

GitGoodBench provides evaluation and training datasets for three Git scenarios drawn from permissively licensed repositories. Its construction stratifies samples by scenario type and programming language, while its metrics define success and solve rates for different task types.

3 GitGoodBench Datasets

  • GitGoodBench has a 900-sample comprehensive evaluation set, a 120-sample rapid-prototyping set, and a 17,469-sample training corpus.The training split is dedicated to collecting trajectories for training Git agents.Source: §3 · p. 2

3.1 Supported Scenarios

  • Merge conflict resolution reproduces ground-truth conflict resolutions; interactive rebase improves commit ordering, while iterative committing builds history from disorganized changes.Interactive rebase can consolidate or modify commits and refine messages; iterative committing organizes changes into a reasonable Git history.Source: §3.1 · p. 3

3.2 Dataset Creation

  • The authors mine permissively licensed repositories for merge and file-commit-chain samples, then stratify and sample by scenario type and programming language.They identify 6,917 merge samples and 11,572 file-commit-chain samples; the held-out sets contain 120 Lite and 900 full-benchmark scenarios, with the remainder used for training.Source: §3.2 · pp. 3–4
  • MCR difficulty ranges from one conflict (easy), to multiple conflicts in one file (medium), to conflicts across files (hard); Lite contains 31 easy, 13 medium, and 16 hard samples.The benchmark’s scenario and programming-language sampling is stratified, and its datasets are mutually disjoint.Source: §3.2 · p. 4, Table 3 · p. 4

3.3 Metrics

  • Success rate counts scenarios without system errors; MCR solve rate uses exact match, while FCC solve rate requires an LLM judge to prefer the agent history in both presentation orders.Table 2 reports success and solve rates by scenario type; its caption characterizes the benchmark as complex despite the strong baseline and custom tools.Source: §3.3 · p. 4, Table 2 · p. 4

4 Environment

The baseline evaluates GPT-4o with custom tools on GitGoodBench Lite, without an agentic reasoning framework. Task-specific context and tools scaffold interactive rebase, iterative committing, and merge conflict resolution.

4 Environment

  • The baseline evaluates GPT-4o on GitGoodBench Lite with custom tools, but without an agentic reasoning framework.It uses the tasks and metrics defined by the paper.Source: §4 · p. 4

4.1 Provided Context

  • Interactive rebase receives commit contents and rebase-todo instructions; iterative committing receives hunks and updated remaining-hunk lists; merge conflict resolution receives commit order, file names, and conflicts.Each task’s initial context also includes few-shot function-calling examples.Source: §4.1 · pp. 4–5

4.2 Provided Tools

  • After minimal terminal-only tooling produced no meaningful progress, the authors added task-specific tools for viewing commits, editing rebase plans, grouping hunks, and resolving conflicts sequentially.For merge conflicts, the agent can inspect conflicts, whole files, or the overall diff, but may patch only the current conflict.Source: §4.2 · p. 5

5 Baseline Results

GPT-4o with custom tools solved 21.11% of GitGoodBench Lite scenarios overall, with lower performance on merge-conflict resolution than on the other scenario types.

5 Baseline Results

  • 21.11% of GitGoodBench Lite scenarios were solved overall, despite an 88% success rate for the baseline.The authors note that the system did not solve the majority of benchmark tasks, even with significant scaffolding.Source: §5 · p. 5
  • The system achieved higher success and solve rates on IR and ICC than on MCR, which the authors partly attribute to stricter scaffolding for IR and ICC.MCR requires generating code to apply at conflict locations; performance deteriorated particularly on medium and hard samples requiring globally consistent choices.Source: §5 · p. 5
  • Easy MCR samples had approximately three times the solve rate of hard samples, while IR and ICC samples had similar success and solve rates.The authors report that the MCR difficulty heuristic tracked solve rate, whereas the IR and ICC difficulty heuristic did not correlate with observed difficulty.Source: §5 · p. 5

6 Conclusions

GitGoodBench benchmarks agent performance on three Git scenarios, and a GPT-4o baseline with Git tools achieved a 21.11% overall solve rate on GitGoodBench Lite.

6 Conclusions

  • GitGoodBench is a benchmark for training and evaluating agents on MCR, IR, and ICC Git scenarios.The baseline equipped GPT-4o with tools for interacting with Git.Source: §6 · p. 5
  • 21.11% was the overall solve rate for the GPT-4o baseline on GitGoodBench Lite.Source: §6 · p. 5
  • The authors conclude that SE agents should be designed to understand Git artifacts and use Git functionality, alongside repository-level reasoning and code generation.They point to poor overall performance and degradation across MCR difficulty levels as reasons to consider Git explicitly.Source: §6 · p. 5

7 Limitations

The authors identify constraints in the baseline tools and evaluation, as well as gaps and distribution concerns in the benchmark data.

7 Limitations

  • MCR tooling cannot change Git-generated hunk boundaries, limiting flexibility when hunks are too coarse.Source: §7 · p. 5
  • ICC tooling is limited to single-file changes, while IR cannot modify commit content; FCC evaluation uses an LLM-based judge that may introduce bias.The authors suggest using an ensemble of LLMs to mitigate judging bias and subjectivity, and note that Git MCP effects were not investigated.Source: §7 · p. 6
  • The authors report remaining dataset limitations despite efforts to ensure diversity, including that the FCC difficulty heuristic failed to correlate with observed performance.Source: §7 · p. 6
  • The FCC difficulty heuristic did not correlate with empirical performance, so FCC sample difficulty may be skewed in the benchmark.The benchmark also omits Git diagnostic workflows such as git bisect and, because it is static, may need updates to counter benchmark saturation and data leakage.Source: §7 · p. 6

A Dataset Details

The appendix describes dataset diversity across repositories and MCR difficulty, noting a language skew and broadly distributed MCR difficulty.

A Dataset Details

  • The appendix discusses repository representation and MCR difficulty distributions; training-set language distribution is also reported, while other datasets use fixed language distributions to ensure diversity.Source: A Dataset Details · p. 7
  • The datasets have a heavy skew toward Python repositories and, to a lesser extent, Java repositories, which the authors say matches expectations about language popularity.Source: A Dataset Details · p. 7
  • MCR difficulty is relatively well-distributed overall, with a difficulty spike in GitGoodBench despite stratification.The authors note that stratifying by other features, such as programming language, can produce such spikes.Source: A Dataset Details · p. 7

A.1 Sample Data

The appendix describes the dataset’s sample structure and diversity across repositories and programming languages, then introduces the scenario field.

A.2 The Scenario Field

  • Each data point has repository metadata and scenario-specific information; scenario-field contents vary by sample type.The appendix introduces further details on the scenario field after describing the sample structure.Source: §A.1 · p. 7, §A.2 · p. 8

A.1 Sample Data

  • GitGoodBench Lite includes 31 easy, 13 medium, and 16 hard MCR samples, plus 15 easy, 22 medium, and 23 hard FCC samples.Table 4 reports success and solve rates by scenario type and difficulty.Source: Table 4 · p. 8
  • Across the mined samples, the datasets include 816 unique repositories overall: 525 Python repositories, 284 Java repositories, and 79 Kotlin repositories.Source: Table 5 · p. 8

A.2.1 Contents For FCC Samples

The appendix details scenario metadata for FCC and merge samples, and defines a purity-based FCC difficulty heuristic whose empirical pattern did not match expectations.

A.2.1 Contents For FCC Samples

  • FCC scenario metadata records the sample source, chain length, and starting and ending commits.Table 11 gives a representative FCC scenario field.Source: §A.2.1 · p. 8, Table 8 · p. 9

A.2.2 Contents For Merge Samples

  • MCR scenario metadata includes the merge commit used as ground truth; its parent commits generate the conflict resolved by that merge.The example is classified as hard because it has multiple conflicts across multiple files.Source: §A.2.2 · p. 8

A.2.3 File-Commit Chain (FCC) Difficulty Heuristic

  • FCC difficulty is defined by purity p, the ratio of changes in the mined file to all changes in the chain; the heuristic assigns purity categories.A Git line with a + or - prefix counts as a change; Table 8 illustrates an FCC sample with purity 0.68 classified as medium.Source: §A.2.3 · p. 8, Table 8 · p. 9
  • FCC easy samples achieved 100% success but had lower success than medium and hard samples, contrary to the expected difficulty ordering.The authors suggest that purity omits the total number of changes, which can increase the context an agent must reason over even when difficulty labels match.Source: §A.2.3 · p. 8
  • Table 6 reports difficulty distributions across GitGoodBench datasets.The appendix notes a spike in MCR difficulty on GitGoodBench despite stratified sampling.Source: Table 6 · p. 9

B Prompts

The appendix documents prompts for the three Git scenarios and the LLM judge, including the information and tool instructions supplied to agents.

B Prompts

  • The appendix introduces prompts for individual scenarios and LLM-as-a-Judge evaluation, referring readers to the repository for further details.Source: B Prompts · p. 9

B.1 Merge Conflict Resolution (MCR) Scenarios

B.2 Interactive Rebase (IR) Scenarios

  • IR prompts provide participating commit information, interactive-rebase instructions, tool examples, and the required rebase-todo JSON schema.Commit information is included to save agent turns spent reading it.Source: §B.2 · p. 10, Figure 5 · p. 13, Figure 6 · p. 14

B.3 Iterative Committing of Changes (ICC) Scenarios

B.4 LLM-as-a-Judge Evaluation

  • The judge prompt specifies history-quality assessment dimensions, example responses, and a response schema.The prompt presents both ground-truth and agent-generated histories for evaluation.Source: §B.4 · p. 10
  • Both judge runs use the same prompt, swapping the histories’ positions to mitigate position bias.Figures 10–11 show the evaluation prompt.Source: §B.4 · p. 10, Figure 10 · p. 16, Figure 11 · p. 17
Loading 2505.22583v1…