PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models

Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Huaixi Tang, Haoyang Huang, Zhiyi Lai, Zizheng Zhan, Yanan Wu, Chenchen Zhang, Kepeng Lei, Yifan Yao, Xinping Lei, Wenqiang Zhu, Zongxian Feng, Han Li, Junqi Xiong, Dailin Li, Zuchen Gao, Kun Wu, Wen Xiang, Ziqi Zhan, Yuanxing Zhang, Wuxuan Gong, Ziyuan Gao, Guanxiang Wang, Yirong Xue, Mengtong Li, Mengfei Xie, Xiaojiang Zhang, Jinghui Wang, Wenhao Zhuang, Zheng Lin, Huiming Wang, Zhaoxiang Zhang, Yuqun Zhang, Haotian Zhang, Bin Chen, Jiaheng Liu

arXiv:2511.05459v3cs.SEcs.AI

TL;DR

Existing software engineering benchmarks often center on algorithmic problems or Python-centric bug fixing, leaving broader developer workflows and languages undercovered. SWE-Compass brings together repository-derived tasks across multiple dimensions and evaluates models under two agentic frameworks. The experiments reveal difficulty hierarchies across tasks, languages, and scenarios, alongside persistent gaps and failure modes.

  • Problem

    Existing evaluations have narrow task coverage, language bias, and limited alignment with real-world workflows, leaving important software engineering activities underexplored.

    Source: §1 · p. 1

  • Method

    SWE-Compass combines 2,000 verified GitHub-derived instances across eight task types, eight scenarios, and ten languages, evaluated with SWE-Agent and Claude Code.

    Source: §1 · p. 2, §5 · p. 13

  • Results

    Experiments reveal task-difficulty hierarchies, language-specific variability, and dominant failure modes involving requirement misinterpretation and incomplete solutions.

    Source: §5 · p. 13

  • Takeaways & Limitations

    The benchmark provides a structured, reproducible basis for diagnosing agentic coding capabilities across diverse software engineering tasks and settings.

    Source: §1 · p. 2, §5 · p. 13

Abstract

from arXiv · show

Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, SWE-Compass provides a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models.

1 Introduction

Existing coding benchmarks leave much of real-world software engineering underexplored, especially work beyond bug fixing and across diverse scenarios and languages. SWE-Compass addresses this gap with a structured, execution-grounded benchmark and reports persistent capability gaps across tasks, languages, and scenarios.

1 Introduction

  • Prior evaluations often center on single-file tasks, Python-centric bug fixing, or synthetic algorithms, leaving feature work, refactoring, configuration, and optimization underexplored.Repository-grounded benchmarks improve ecological validity but still largely treat bug fixing as the dominant axis, limiting diagnosis across workflows and scenarios.Source: §1 · p. 1
  • SWE-Compass evaluates LLMs across eight task types, eight programming scenarios, and ten languages using 2,000 curated GitHub-PR instances with reproducible execution environments.Its taxonomy includes feature implementation, refactoring, test generation, and deployment, among other activities.Source: §1 · p. 2
  • SWE-Compass structures evaluation around real-world developer activities to support fine-grained capability diagnosis and consistent comparison of agentic coding systems.The framework covers activities such as feature implementation, refactoring, test generation, and deployment.Source: §1 · p. 2
  • Experiments with state-of-the-art models and agentic systems reveal persistent gaps across tasks, languages, and scenarios, highlighting the difficulty of scaling beyond bug fixing.The authors argue for benchmarks reflecting the complexity of real-world software engineering.Source: §1 · p. 2

2 Related Works

Coding LLMs have gained long-context and tool-use capabilities that support autonomous, multi-step work in large codebases. Yet benchmark coverage remains incomplete: single-file tests abstract away repository complexity, while repository-scale benchmarks largely emphasize bug fixing.

Coding LLMs and Agents

  • Modern coding LLMs combine long-context handling and tool use, enabling agentic systems to plan, search, edit, and test across codebases.The cited systems can also use tools such as editors, shells, debuggers, and browsers.Source: §2 · pp. 2–3

Coding Benchmarks

  • Single-file benchmarks abstract away software engineering’s structural, contextual, and environmental complexity, while repository-grounded benchmarks remain overwhelmingly focused on bug fixing.The latter omit broader activities such as feature implementation, refactoring, configuration, performance optimization, and test generation across varied engineering contexts.Source: §2 · p. 3

3 SWE-Compass

SWE-Compass organizes software engineering evaluation across tasks, scenarios, and languages, using repository-derived instances and reproducible environments. Its construction combines iterative taxonomy development, PR collection, task-specific construction, and validation.

3.1 Overview

  • SWE-Compass contains 2,000 instances across eight code-related task types, eight programming scenarios, and ten languages, addressing gaps in existing benchmark coverage.Its scope includes bug fixing and performance optimization, among other software engineering capabilities.Source: §3.1 · p. 3

3.2 Design Principles

  • Four principles guide the benchmark: real-world alignment, balanced coverage, a systematic taxonomy, and evaluation fidelity through executable tests and reproducible environments.Controlled synthesis supplements categories underrepresented in real repositories while retaining consistency with real-world task characteristics.Source: §3.2 · pp. 3–4

3.3 Benchmark Construction

  • Construction proceeds through user analysis, data collection, environment building, task construction, and validation, combining automated processing with expert annotation.The workflow begins with iterative active learning over real-world coding conversations and ends with multi-round filtering and quality validation.Source: §3.3 · pp. 4–5

3.3.1 Step 1: User Analysis

  • An iterative active-learning process labels conversations by task, scenario, and language, yielding eight task types, eight scenarios, and ten languages with defined category distinctions.The task taxonomy distinguishes feature implementation from enhancement, bug fixing, and behavior-preserving refactoring; the scenario taxonomy spans application, database, data, ML/AI, infrastructure, specialized, security, and UI/UX work.Source: §3.3.1 · p. 5

3.3.2 Step 2: Data Collection

  • Data collection supplements mapped existing benchmarks with filtered repositories and merged PRs linked to descriptive issues and identifiable changes; about 50,000 PRs remained after filtering.Repository screening included licensing, maintenance, contributor, activity, and executable-test criteria.Source: §3.3.2 · p. 6

3.3.3 Step 3: Environment Building

  • Containerized environments were built from repository dependencies and validated with native tests; expert-assisted fixes yielded about 4,000 runnable Docker images.Initial automated build success was around 2%; the expert retry process raised overall retention to approximately 8%.Source: §3.3.3 · p. 6

3.3.4 Step 4: Task Building

  • Task construction uses checklist synthesis for code understanding, reverse masking for deployment and test generation, and targeted filtering for patch-based tasks.Performance-optimization seeds require more than 30% runtime improvement while passing unit tests before and after patch application; other patch tasks are classified by intent and validated for consistency and clarity.Source: §3.3.4 · pp. 6–7

3.3.5 Step 5: Data Validation

  • Difficulty screening, balanced sampling, and manual verification produced 2,000 high-quality instances balanced across tasks, scenarios, and languages.Evaluation uses Pass@1 for feature implementation, enhancement, bug fixing, and refactoring; optimization score for performance optimization; line coverage for test generation; and checklist-based LLM judging for code understanding.Source: §3.3.5 · p. 7, §3.4 · pp. 7–8

4 Experiments

Across ten models and two agent workflows, SWE-Compass reveals complementary workflow strengths and clear variation in difficulty across task types, languages, and scenarios. Further analyses connect performance patterns to interaction effort, consistency, and recurring failure modes.

4.1 Evaluated LLMs and Frameworks

  • The evaluation covers executable and non-executable tracks, ten models, and two offline workflows using unified budgets, executors, and single-attempt settings.Networking is disabled and retries are not used; workflow configurations and deviations are documented in Appendix A.4.Source: §4.1 · p. 8

4.2.1 Main Results

  • 32.9% with Claude Code and 31.8% with SWE-Agent: Claude-Sonnet-4-20250514 ranks first under both workflows, while the agents show complementary AVG strengths.Among five overlapping models, two score higher on AVG with Claude Code and three with SWE-Agent.Source: §4.2.1 · p. 8, Table 2 · p. 9
  • Code Understanding is among the strongest task categories, while Feature Implementation and Bug Fixing are harder; Test Case Generation and Performance Optimization remain challenging but not single-digit on average.Configuration & Deployment varies substantially across models; Feature Enhancement and Refactoring occupy a middle tier.Source: §4.2.1 · p. 8, Table 2 · p. 9
  • SWE-Agent’s edit–diff–execute loop favors investigative, multi-file localization but faces more timeout exposure; Claude Code performs strongly on well-scoped deterministic tasks with lower tool overhead.Higher Eval scores often coincide with more interaction turns, with diminishing returns beyond moderate turn counts.Source: §4.2.1 · pp. 8–9

4.2.2 Further Analysis

  • Java, Kotlin, and JavaScript tend to score higher, systems languages are harder, TypeScript is notably lower, and Python is mid-tier overall with dataset-selection effects.Turn distributions show heavier tails for systems languages, especially under SWE-Agent; Python has high variance across environments and workflows.Source: §4.2.2 · p. 9
  • Higher-ranked systems tend to improve more consistently across languages, and higher overall performance visually coincides with lower cross-language/task variability.The authors argue that reporting both central tendency and dispersion avoids overstating gains concentrated in a subset of languages.Source: §4.2.2 · pp. 9–11
  • Scenario difficulty tracks tooling determinism and edit locality: reliable, fast-test settings favor Claude Code, while complex multi-stage stacks often favor SWE-Agent’s iterative localization but raise timeout exposure.Claude Code has a consistent average advantage across scenarios in this setting, concentrated in pipelines with reliable, low-variance signals.Source: Table 3 · p. 11, §4.2.2 · pp. 11–12
  • In 600 failed trajectories per model, Requirement Misinterpretation and Incomplete Solution & Side Effects account for over 60% of failures, while Technical Knowledge Gap is consistently low.The post-hoc SWE-Agent analysis samples three representative systems and uses Claude-Sonnet-4 as an LLM judge; reported rates are 30–34%, 29–42%, and 5–8%, respectively.Source: §4.2.2 · pp. 12–13

5 Conclusion

SWE-Compass provides a broad, reproducible evaluation of coding agents and reveals difficulty hierarchies and language variability alongside recurring requirement-understanding and solution-completeness failures.

5 Conclusion

  • Experiments across ten LLMs and two agent frameworks reveal task-difficulty hierarchies, language-specific variability, and dominant failures in requirement misinterpretation and incomplete solutions.The authors identify requirement grounding, environment reliability, and reasoning consistency as priorities for future progress in automated software engineering.Source: §5 · p. 13

6 Future Works

The authors outline extensions to broaden SWE-Compass and improve the reliability, depth, and accessibility of evaluation.

6 Future Works

  • Explore safe online or incremental tracks, human-adjudication subsets for LLM judging, and continued reproducibility, safety, and accessibility work.Suggested measures include sandboxing, replayable logs, judge reliability audits, released containers, and a lightweight stable-seed subset.Source: §6 · p. 13
  • Expand dataset scale, languages, and repository types while maintaining distribution alignment across task, scenario, language, and difficulty.Examples include mobile stacks, diverse SQL dialects, monorepos, and polyglot services.Source: §6 · p. 13
  • Add harder long-context tasks and richer metrics and protocols, including consistency, variance, timing, coverage, and cost reporting under fixed budgets.Proposed settings include multi-module, cross-process, and build-pipeline tasks that stress architectural coherence and memory.Source: §6 · p. 13

Judge Prompt for Code Understanding Task

The prompt evaluates code-understanding answers against checklist requirements and assigns a score based on satisfied items.

Judge Prompt for Code Understanding Task

  • The prompt supplies a question, checklist requirements, and a possibly truncated answer for evaluation.It requests a natural-language explanation rather than code diffs alone.Source: Judge Prompt for Code Understanding Task · p. 18
  • The evaluator awards 1.0 only when all checklist items are thoroughly satisfied with clear explanations, and scores the fraction of satisfied items.Vague statements and wrong information are penalized; code diffs alone or completely wrong answers receive 0.0.Source: Judge Prompt for Code Understanding Task · p. 18
  • The output format includes a brief rationale, a score, and the satisfied-item identifiers.The expected JSON fields include reasoning, score, and satisfied_items.Source: Judge Prompt for Code Understanding Task · p. 18

A.2 Claude Code: Parallel Tool-Calls System Prompt

An SDK-appended prompt encourages Claude Code to invoke tools in parallel when operations are independent.

A.2 Claude Code: Parallel Tool-Calls System Prompt

  • The prompt encourages parallel tool invocations for independent operations.It is appended via the SDK.Source: §A.2 · p. 19

Claude Code: Parallel Tool-Calls System Prompt

The system prompt directs Claude Code to run independent operations concurrently, using parallel file reads as an example and favoring parallel calls over sequential ones.

Claude Code: Parallel Tool-Calls System Prompt

A.3 Trajectory Failure Analysis Prompt

The appendix specifies a trajectory-failure analysis prompt, executor settings, and an analysis of coverage gaps in open-source SWE benchmarks, alongside the evaluated model list.

Trajectory Failure Analysis Prompt

  • The prompt supplies repository tools for terminal commands, file viewing, searching, editing, and submission, then requests a description, category, and error actions.Source: Trajectory Failure Analysis Prompt · pp. 19–22
  • The failure-analysis prompt treats trajectories as failed when final evaluation fails and directs analysis toward one predefined flaw category rather than agent self-assessment.Categories include requirement misinterpretation, inadequate testing, incomplete solutions or side effects, technical knowledge gaps, tool invocation errors, infinite loops, and other.Source: Trajectory Failure Analysis Prompt · pp. 21–22

A.4 Executor Details and Method-Specific Settings

  • SWE-Agent and Claude Code runs each had a 150-turn maximum; SWE-Agent alone had a 600 s per-tool step timeout, with method-specific parsing, permissions, and tool-call configurations.All runs were strictly offline unless otherwise noted; Claude Code networking was disabled.Source: §A.4 · p. 22
  • The appendix standardizes non-interactive build and test commands and hardens execution with pinned toolchains, offline caches, output controls, and language-specific repairs.It also describes navigation pruning and global job limits.Source: §A.4 · pp. 22–23

A.5 Analysis of Open-Source Benchmark Distributions

  • The appendix identifies open-source SWE benchmarks totaling 2,480 instances across multiple languages and scenarios as the basis for its distribution analysis.Figure 9 presents distributions by task type, programming scenario, and language.Source: §A.5 · p. 23, Figure 9 · p. 24
  • Existing open-source benchmarks cover bug fixing alone, leaving seven listed task types—including feature work, code understanding, and refactoring—absent.Bug-fixing tasks comprise 100% of instances.Source: §A.5 · pp. 23–24
  • Application Development accounts for 32.6% and Infrastructure Development for 22.1%, while Python dominates at 71.7% of instances.UI/UX Engineering, Database Systems, and Security Engineering account for 3.0%, 2.9%, and 7.5%, respectively; Go is 13.5% and JavaScript 8.7%.Source: §A.5 · p. 24

A.6 Model link list

  • The evaluated model set includes closed-source and open-source systems, with the complete model list referenced in Table 5.Source: §A.6 · p. 24, Table 5 · p. 24
Loading 2511.05459v3…