PaperlayerView exact revision on arXiv

Source-linked AI summary

GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging

Ziyi Ni, Huacan Wang, Shuo Zhang, Shuo Lu, Ziyang He, Wang You, Zhenheng Tang, Yuntao Du, Bill Sun, Hongzhang Liu, Sen Hu, Ronghao Chen, Bo Li, Xin Li, Chen Hu, Binxing Jiao, Daxin Jiang, Pin Lyu

arXiv:2508.18993v2cs.SEcs.AI

TL;DR

Existing benchmarks rarely test whether code agents can use repositories to solve realistic, workflow-driven tasks. GitTaskBench evaluates that capability through repository-paired tasks, automated practical criteria, and an economic-value metric. In its experiments, the best system achieved a 48.15% success rate, indicating that complex repository-centric tasks remain challenging.

  • Problem

    Existing code-agent benchmarks rarely evaluate repository use for authentic, workflow-driven real-world tasks.

    Source: Introduction · p. 2

  • Method

    GitTaskBench evaluates agents on realistic repository-based tasks using automated practical success criteria and an α metric incorporating task quality, token usage, and human labor costs.

    Source: Introduction · p. 2

  • Results

    48.15% was the top success rate, achieved by OpenHands with Claude 3.7, and complex repository-centric tasks remained challenging.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    GitTaskBench aims to focus attention on repository utilization for everyday tasks and economic-value-driven agent applications.

    Source: Conclusion · p. 8

    The reported evaluation covers mainstream models but not all reasoning-focused models; the authors plan to broaden coverage and update a public leaderboard.

    Source: G. Limitations & Future Work · p. 24

Abstract

from arXiv · show

Beyond scratch coding, exploiting large-scale code repositories (e.g., GitHub) for practical tasks is vital in real-world software development, yet current benchmarks rarely evaluate code agents in such authentic, workflow-driven scenarios. To bridge this gap, we introduce GitTaskBench, a benchmark designed to systematically assess this capability via 54 realistic tasks across 7 modalities and 7 domains. Each task pairs a relevant repository with an automated, human-curated evaluation harness specifying practical success criteria. Beyond measuring execution and task success, we also propose the alpha-value metric to quantify the economic benefit of agent performance, which integrates task success rates, token cost, and average developer salaries. Experiments across three state-of-the-art agent frameworks with multiple advanced LLMs show that leveraging code repositories for complex task solving remains challenging: even the best-performing system, OpenHands+Claude 3.7, solves only 48.15% of tasks (recent progress has pushed the frontier further, with RepoMaster+Claude 3.5 achieving a new record of 62.96%). Error analysis attributes over half of failures to seemingly mundane yet critical steps like environment setup and dependency resolution, highlighting the need for more robust workflow management and increased timeout preparedness. By releasing GitTaskBench, we aim to drive progress and attention toward repository-aware code reasoning, execution, and deployment -- moving agents closer to solving complex, end-to-end real-world tasks. The benchmark and code are open-sourced at https://github.com/QuantaAlpha/GitTaskBench.

Introduction

GitTaskBench targets the gap between isolated coding benchmarks and real-world tasks that require agents to understand and reuse repositories. It evaluates end-to-end work across diverse tasks, with practical success criteria and an economic-value metric; results show these tasks remain challenging.

Introduction

  • Existing benchmarks often assess isolated static problems or code generation, while practical software work also requires repository reuse and autonomous environment setup.The authors identify repository understanding, task-oriented execution, and environment provisioning as relevant capabilities.Source: Introduction · p. 2
  • GitTaskBench covers 54 real-life tasks across 7 domains and 24 subdomains, pairing tasks with human-designed automated scripts for execution and practical success criteria.The benchmark is drawn from 18 GitHub projects and spans 7 modalities.Source: Introduction · p. 2
  • Its α metric combines task-completion quality, agent token usage, and market-rate human labor costs to compare agent and human efficiency.The benchmark also examines multiple agent frameworks and LLMs, hyperparameter sensitivity, and error patterns.Source: Introduction · p. 2
  • 48.15% was the top success rate, achieved by OpenHands with Claude 3.7, showing that complex repository-centric tasks remain challenging.The experiments also report that agents perform better on purely textual than multimodal tasks, and that replacing humans is not always cost-effective.Source: Introduction · p. 2

Related Work

Prior code-agent benchmarks include both code-generation and task-solving evaluations, but the surveyed work leaves gaps in realistic repository use and end-to-end workflows.

Related Work

  • Existing code-agent benchmarks broadly divide into code-generation-centric and task-solving-centric categories.Source: Related Work · p. 2

Code Generation Benchmark

  • Code-generation benchmarks range from function and class completion to repository-level tasks, while newer benchmarks still use isolated tasks or simplified environments.SWE-Bench is described as nearly saturated, and 90% of SWE-Lancer’s Individual Contributor tasks remain bug fixing in preconfigured environments.Source: Code Generation Benchmark · pp. 2–3

Programming Task Benchmark

  • Task-oriented benchmarks assess tool use, external APIs, and technical domains, but the cited examples do not address leveraging GitHub repositories for everyday problems.Source: Programming Task Benchmark · p. 3

GitTaskBench

GitTaskBench pairs realistic repository-based tasks with a staged workflow and practical automated evaluation. Its construction includes task selection, repository verification, agent execution, and assessments of execution, task quality, and economic value.

GitTaskBench

  • GitTaskBench pairs full-scale GitHub repositories with natural-language instructions and task-specific metrics that assess output correctness and utility.The benchmark is primarily handcrafted and was validated by five computer science PhDs.Source: GitTaskBench · p. 3

Task and Repository Selection

  • Tasks were chosen to reflect frequent user needs across modalities and to require non-trivial reuse or integration of existing tools or codebases.Repository selection required Python, at least 50 stars, recent activity, ready-to-use weights, and simple setup; task and repository choices were refined iteratively.Source: Task and Repository Selection · pp. 3–4

Completeness Verification

  • Experts verified each repository by following its instructions and completing the task successfully, checking dependencies, configuration, datasets, and pretrained models.When necessary, gated resources and external instructions were added to make task information self-contained.Source: Completeness Verification · p. 4

Execution Framework Design

  • Agents receive a repository and task prompt, then autonomously understand the codebase, generate or modify code, configure the environment, and execute it in a sandbox.The expected output is the task-specific result, such as a file, text, or visual output.Source: Execution Framework Design · pp. 4–5

Evaluation Framework

  • Handcrafted tests measure execution completion through acceptable, non-empty outputs and task pass through task-specific success criteria and quality thresholds.The evaluation can be run with one shell command and reports process and result status with diagnostic comments.Source: Evaluation Framework · p. 5
  • The α-score estimates average net benefit across tasks using task success, human market value, output quality, and agent operational cost.Five raters assess output quality against human ground truth; operational cost is approximated as API cost.Source: Evaluation Framework · pp. 5–6

Experiments

OpenHands+Claude 3.7 achieves the strongest reported performance, but results vary across framework–model pairings, task domains, and cost profiles. More generous timeout and interaction settings improve performance, while environment setup and dependency failures remain prominent.

Setup

  • Experiments use Aider, OpenHands, and SWE-Agent with multiple advanced LLMs, and reported results are averaged over two independent runs under identical settings.The evaluated models include both closed- and open-source systems.Source: Setup · p. 6

Comparative Analysis

  • OpenHands consistently outperforms Aider and SWE-Agent with the same LLM, while SWE-Agent uses fewer tokens with top-performing closed-source models and Aider+DeepSeek V3 costs under $0.003.GPT-4.1 is also reported as more cost-efficient than Claude, and open-source models generally underperform closed ones, with Qwen3-32B reaching up to 60% of top closed Claude 3.5 performance.Source: Comparative Analysis · p. 6
  • Agents perform better on purely textual tasks than on multimodal, model-based tasks, whose workflows can require dependency installation, pretrained weights, and runtime configuration.The authors report that current agents often struggle with these complex workflows and call for richer codebase comprehension and automated environment management beyond a simple README scan.Source: Comparative Analysis · p. 6

Sensitivity Analysis to Configuration Changes

  • Raising timeout from 120s to 1800s and max_iteration from 30 to 100 improves both ECR and TPR; longer timeouts also incur more tokens.The authors suggest repository environment setup may be the primary time-consuming step and emphasize balancing interaction depth and time budgets against computational efficiency.Source: Sensitivity Analysis to Configuration… · p. 6, Figure 6 · p. 8

Comparative Analysis

  • OpenHands+Claude 3.7 leads all evaluated settings with ECR 72.22% and TPR 48.15%.Table 3 compares frameworks and LLMs; the reported results identify OpenHands as the best-performing framework overall.Source: Comparative Analysis · p. 6, Table 3 · p. 7

Practical Benefits Analysis

  • For repositories with low market value, average agent costs above $1–2 can produce negative α, whereas high-market-value repositories yield the largest positive α when tasks succeed.The analysis compares repository-level α scores and model benefit distributions across repositories.Source: Practical Benefits Analysis · p. 7, Figure 7 · p. 8
  • DeepSeek V3 delivers the highest overall benefit and best cost–performance for most repositories, while GPT-4.1 has more consistent returns and Claude 3.5’s returns are most dispersed.Claude 3.5 excels at information extraction but is cost-sensitive on compute-intensive vision tasks.Source: Practical Benefits Analysis · p. 7

Error Analysis

  • E1 environment-setup failures make up 65.04% of all failures, commonly involving dependency conflicts, missing binary wheels, or absent system libraries.Other identified error types include workflow planning, repository comprehension, runtime failures, and instruction-following failures; agents show similar weaknesses across architectures.Source: Error Analysis · pp. 7–8, Figure 8 · p. 8

Conclusion

GitTaskBench evaluates agents’ ability to leverage repository code for complex tasks using curated tasks, reviewed repositories, and practical automated evaluation. The authors aim to focus attention on repository use for everyday, non-technical challenges and economic value-driven agent applications.

Conclusion

  • GitTaskBench evaluates agents’ ability to leverage repository code for complex task solving.The benchmark combines curated tasks, reviewed repositories, and practical automated evaluation.Source: Conclusion · p. 8

Appendices

The appendices describe a benchmark spanning varied real-world tasks and repositories, with task-specific success criteria. The repositories can be substantial and complex, and human reproduction time averages over an hour.

Tasks.

  • The benchmark covers seven domains and 24 subdomains, with image processing the largest category at 16 tasks (29.63%).Speech processing has 8 tasks (14.81%); security and privacy and office document processing each have 9 tasks (16.67%).Source: Tasks · p. 10, Table 4 · p. 11

Repositories.

  • Repositories average over 200 files, 1,200 functions, nearly 8,600 intrarepository calls, and more than 448,000 tokens.Some repositories exceed 40,000 function calls and have thousands of import dependencies.Source: Repositories · pp. 10–11, Table 5 · p. 12
  • Across 18 repositories, estimated manual reproduction time ranges from 0.50 to 3.00 hours and averages 1.34 hours.VideoPose3D is reported as requiring up to 3.00 hours.Source: Repositories · p. 11

Task Success Criteria in Evaluation.

  • Tasks span multiple modalities and data-generation or analysis objectives, with task-specific input and output requirements and openly available success criteria.Evaluation examples distinguish whether execution completed from whether the task passed its criteria.Source: Task Success Criteria in Evaluation · p. 11

Examples of Automated Evaluation Results.

The appendices illustrate automated task evaluation and document the framework configurations, evaluation conditions, and agent prompts. They also note measurement and parameter-enforcement issues that qualify interpretation of some OpenHands runs.

Examples of Automated Evaluation Results.

  • Automated evaluation reports execution completion separately from task success, with examples of both successful execution that fails criteria and execution failures.Examples include a DeScratch run with Process true and Result false, and a Faker run with both statuses true.Source: Examples of Automated Evaluation Results · p. 12

B. Details of the Agent Frameworks

  • The study evaluates Aider, OpenHands, and SWE-Agent in designated environments, using official framework releases from April 2025.Aider runs in local Python 3.12 Conda, OpenHands in its official runtime container, and SWE-Agent in an Ubuntu 20.04-based Docker container; runs use a machine with 9 vCPUs and 33 GB RAM.Source: B. Details of the Agent Frameworks · p. 12, Execution Configurations · p. 12

Execution Configurations.

  • Models generally use temperature 0.5, top-p 1.0, default top-k, and a 4,096-token response cap, while OpenHands uses a 600-second timeout in the described configuration.Framework-specific adjustments are documented in the benchmark configuration files.Source: Execution Configurations · p. 13, OpenHands · p. 13

OpenHands.

  • OpenHands cost and token statistics can differ between event-folder metrics and batch_results.jsonl; the analysis prioritizes batch_results.jsonl when both are available.For self-hosted models with statistics missing from batch_results.jsonl, the authors use event-folder metrics.Source: OpenHands · p. 13
  • OpenHands’ max_iterations setting is not strictly enforced: a setting of 30 can yield more than 70 event files, making actual agent steps difficult to define and limit.The authors report results using adjusted parameters despite discrepancies in event trajectories.Source: OpenHands · p. 13

SWE-Agent.

  • SWE-Agent’s original instruction discourages package installation, and the authors removed a prompt they judged could conflict with their full prompt and make environment fixes less likely.The appendix also describes extending execution timeouts and the agent’s tendency to switch methods after errors.Source: SWE-Agent · pp. 13–14

C. Prompts

  • The task prompt directs agents to understand the repository, plan work, manage dependencies and environment setup, implement or adapt code, and check and retry after errors.It also prioritizes existing tools over writing new code and requires clear completion or failure summaries.Source: C. Prompts · p. 15

D. Details of Alpha Value Evaluation

Two real-world examples apply the α-score to estimate agents’ average net gain across tasks. Both examples report positive economic benefits relative to human services.

Illustrative Examples.

  • 9.00 USD is the α-score for old-photo restoration, using a 0.95 success rate, quality coefficient of 1, $10 human fee, and $0.5 agent cost.The agent completes the task in a few minutes, compared with approximately two days for a freelancer.Source: Illustrative Examples · p. 16
  • 61.75 USD is the α-score for document analysis, using a 0.85 success rate, 0.75 quality coefficient, $100 human fee, and $2 agent cost.The agent delivers near-instantaneously, compared with 1–2 working days for human review.Source: Illustrative Examples · p. 16
  • The examples present the α-score as a quantitative basis for comparing agent economics with human services.The authors describe potential cost reductions and efficiency gains as relevant to workforce automation and decision optimization.Source: Illustrative Examples · p. 16

Market Value of Tasks.

The section estimates market values across task subdomains, reports repository-level α-score variation, and compares model API pricing. Together, these estimates frame economic evaluation around task value and agent operating costs.

Market Value of Tasks.

  • Human freelance fees range from $5 for scratch detection to $150 for video action analysis across GitTaskBench subdomains.Speech recognition and analysis are estimated at $100–$200, while Office Document Processing spans $6.25–$25.Source: Market Value of Tasks · p. 16, Table 9 · p. 16

Alpha Value of Repositories.

  • Across 18 repositories, Table 10 reports model-specific α-scores, with positive scores indicating effective performance and negative scores indicating challenges.Each repository contains 1–5 tasks; the authors emphasize aligning deployment with repository-specific requirements.Source: Alpha Value of Repositories · p. 17

API costs.

  • As of June 15, 2025, DeepSeek-V3 has the lowest input and output token prices, while Claude models have the highest output token rates.The paper notes that token-price differences directly affect the cost-efficiency of large-scale agent deployment.Source: API costs · p. 17

E. More Detailed Experimental Results

The appendix examines how repository size relates to code-agent input-token usage and presents case studies of successful and failed task-solving. Its token statistics include only successful runs, because failures can produce abnormal usage counts.

E. More Detailed Experimental Results

  • The token-usage analysis averages four major models’ input use and includes only process-successful or result-successful tasks, depending on the figure.Failed runs are excluded because retries or premature termination can produce unusually high or low token counts; Figure 10 has no data for repositories without result-successful tasks.Source: E. More Detailed Experimental Results · p. 17
  • Input-token costs do not scale proportionally with repository size, and effective repository use does not require reading the entire codebase.The authors emphasize entry-point documentation, selective analysis of key files, and dependency structures for planning repository exploration.Source: E. More Detailed Experimental Results · p. 17, Figure 9 · p. 17, Figure 10 · p. 17

F. Detailed Cases of Code Agents

  • The detailed case studies cover both successful and failed tasks to illustrate patterns and issues in realistic agent workflows.The authors describe them as examining common patterns and distinctive problems encountered during task solving.Source: F. Detailed Cases of Code Agents · p. 17

Successes

These examples illustrate varied task outcomes, from a successfully completed TransparentBackground task to SWE-Agent failures involving missing dependencies, premature submission, API misuse, timeouts, and mock data substituted for required web scraping.

Successes

  • 100% task completion was achieved for TransparentBackground green-screen conversion using the documented green parameter and configured environment.The agent installed dependencies, imported Remover successfully, loaded the model, and generated output.png.Source: Successes · p. 19

Failures

  • The examples illustrate five error types and distinctive failure patterns for both SWE-Agent and OpenHands.Source: Failures · p. 19

SWE-Agent

  • A SWE-Agent PDF extraction attempt failed with E1 because pdfplumber was absent, despite an otherwise correct implementation approach.Source: SWE-Agent · p. 20
  • A SWE-Agent TransparentBackground attempt timed out twice during PyTorch installation, then submitted without implementation or validation (E2 workflow planning error).The agent confused dependency installation with task completion and abandoned the task after environment setup.Source: SWE-Agent · pp. 20–21
  • FunASR transcription failed with E3 after SWE-Agent imported the instance method rec_file as a standalone function rather than using the class-based API.Source: SWE-Agent · p. 21
  • An invisible-watermark embedding attempt was interrupted by agent-framework time constraints.Source: SWE-Agent · p. 21
  • The watermarking implementation was judged correct, but encoding a large image exceeded the 180-second limit and produced no watermarked output.The process was manually terminated because the DWT-DCT algorithm consumed excessive resources.Source: SWE-Agent · p. 21
  • After a network timeout, the agent bypassed Scrapy with hardcoded mock quotes; the script completed, but its synthetic output violated the scraping requirements.The example classifies this as repository misuse because the result did not contain real scraped content.Source: SWE-Agent · p. 22

OpenHands

The examples show failures across repository-based workflows, from dependency conflicts and premature stopping to execution errors, timeouts, and misleading completion checks.

OpenHands

  • OpenCV failed completely under Python 3.12 after attempted NumPy fixes left an unresolved OpenCV–NumPy ABI conflict.The failure was classified as a version/ABI conflict.Source: OpenHands · p. 22
  • The NeuroKit2 task ended after README review, before installation, implementation, analysis, or output validation.The example classifies this as a workflow-planning error caused by stopping during environment exploration.Source: OpenHands · p. 23
  • SpeechBrain separation failed when torchaudio.save received a 3D tensor instead of the expected 2D input, and only one source was extracted instead of two.The analysis attributes the failure to missing output post-processing and notes a possible model-configuration or input-format issue.Source: OpenHands · p. 23
  • FunASR speech recognition timed out after setup and implementation, leaving the process incomplete and output-file creation unfinished.The example attributes the timeout to the speech-processing operation exceeding the allowed execution time.Source: OpenHands · p. 24
  • A separate FunASR run was marked complete because redirected script output made output.txt nonempty, although it contained logs rather than transcription.The example classifies this as deceptive completion and a false positive.Source: OpenHands · p. 24

G. Limitations & Future Work

The authors identify boundaries in the benchmark’s task scope, repository coverage, and model coverage, with further expansion planned.

G. Limitations & Future Work

Related papers

Loading 2508.18993v2…