PaperlayerView exact revision on arXiv

Source-linked AI summary

ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development

Jie Yang, Honglin Guo, Li Ji, Jiazheng Zhou, Rui Zheng, Zhikai Lei, Shuo Zhang, Zhiheng Xi, Shichun Liu, Yuxin Wang, Bo Wang, Yining Zheng, Tao Gui, Xipeng Qiu

arXiv:2601.11077v1cs.SEcs.AIcs.CL

TL;DR

Existing benchmarks provide limited evidence about agents’ ability to complete production-like backend workflows that combine coding, configuration, deployment, and end-to-end testing. ABC-Bench addresses this gap with 224 tasks from open-source repositories across 8 languages and 19 frameworks, evaluated through executable service deployment and external API tests. Current systems remain unreliable on these tasks, with environment configuration and deployment identified as persistent bottlenecks.

  • Problem

    Existing benchmarks often emphasize localized code logic and fragmented validation, providing limited coverage of the integrated configuration, deployment, and end-to-end requirements of backend engineering.

  • Method

    ABC-Bench evaluates agents on full-lifecycle backend tasks, while ABC-Pipeline automatically constructs tasks from open-source repositories and verifies their executable environments and tests.

  • Results

    Current systems remain unreliable on full-lifecycle tasks; the top-performing model achieves 63.2% pass@1, with environment configuration and deployment identified as persistent bottlenecks.

  • Takeaways & Limitations

    The benchmark exposes a substantial gap between current model capabilities and the practical demands of real-world backend engineering.

Abstract

from arXiv · show

The evolution of Large Language Models (LLMs) into autonomous agents has expanded the scope of AI coding from localized code generation to complex, repository-level, and execution-driven problem solving. However, current benchmarks predominantly evaluate code logic in static contexts, neglecting the dynamic, full-process requirements of real-world engineering, particularly in backend development which demands rigorous environment configuration and service deployment. To address this gap, we introduce ABC-Bench, a benchmark explicitly designed to evaluate agentic backend coding within a realistic, executable workflow. Using a scalable automated pipeline, we curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories. Distinct from previous evaluations, ABC-Bench require the agents to manage the entire development lifecycle from repository exploration to instantiating containerized services and pass the external end-to-end API tests. Our extensive evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks, highlighting a substantial disparity between current model capabilities and the demands of practical backend engineering. Our code is available at https://github.com/OpenMOSS/ABC-Bench.

1 Introduction

ABC-Bench addresses the gap between localized benchmark tasks and production-like backend engineering by evaluating agents across the full development lifecycle. Its evaluation shows that current systems remain unreliable, with environment configuration and deployment emerging as major bottlenecks.

  • Motivation: Current benchmarks often emphasize isolated code edits and unit-level validation, overlooking environment configuration, deployment, and end-to-end backend workflows.Real-world backend engineering integrates coding, configuration, container orchestration, and functional validation.
  • Evaluation: The evaluation uses a closed loop in which agents analyze repositories and resolve issues before Docker deployment and real HTTP verification.The workflow spans development steps followed by service instantiation and production-like endpoint testing.
  • Benchmark: ABC-Bench requires agents to explore repositories, edit code, configure environments, deploy containerized services, and pass external API-level tests.Credit is awarded only when the deployed service starts correctly and exhibits the expected behavior.
  • Benchmark: 224 curated tasks were constructed from 2,000 open-source repositories across 8 backend programming languages and 19 frameworks.The benchmark is designed to preserve the heterogeneity of real-world backend stacks.
  • Results: 63.2% pass@1 is achieved by the top-performing model, while many systems perform substantially worse on full-lifecycle tasks.Further analysis identifies environment configuration and deployment as persistent barriers to success.
  • Contribution: ABC-Pipeline automates task extraction from open-source repositories, reducing manual effort in constructing realistic backend evaluation datasets.The pipeline supports scalable benchmark construction.

2 Related Work

Prior work has moved software-engineering evaluation toward repository-level, tool-using agents and executable feedback. ABC-Bench is situated within this broader shift while emphasizing diverse backend software stacks.

  • Agentic Paradigms: Agentic software-engineering systems extend code-focused LLMs from isolated programming problems to repository navigation, tool use, and multi-step development tasks.This shift enables models to operate in real-world codebases and development environments.
  • Evaluation of Coding Capability: Coding benchmarks increasingly incorporate executable feedback, natural prompts, multi-stage development, and code editing rather than only function-level generation.Repository-level evaluations also assess issue resolution and feature implementation on real-world repositories.
  • Benchmark Scope: ABC-Bench’s dataset composition spans 8 programming languages and 19 backend frameworks, covering a wide range of software stacks.The framework distribution is presented as evidence of ecosystem diversity.

3 ABC-Bench

ABC-Bench evaluates backend agents on repository-level tasks drawn from diverse ecosystems, including both logic implementation and autonomous operational setup. Agents must independently modify repositories and produce deployed services that pass external API checks.

  • Dataset: ABC-Bench contains 224 tasks spanning 8 programming languages and 19 backend frameworks.The tasks are intended to represent modern backend ecosystems.
  • Dataset: 132 tasks focus on logic implementation in pre-provisioned runtimes, while 92 require environment configuration and containerized service startup.The latter subset directly tests end-to-end operational capability.
  • Evaluation Setup: Agents operate autonomously in an isolated sandbox, exploring repositories, modifying code, installing dependencies, and updating Docker configurations.The development loop proceeds without human intervention.
  • Evaluation Setup: Success is determined exclusively by external functional API requests against the deployed backend service.Static code artifacts are insufficient unless the service launches and demonstrates the expected behavior.

4 ABC-Pipeline

ABC-Pipeline converts open-source backend repositories into verified full-lifecycle benchmark tasks through repository exploration, environment synthesis, and task instantiation. Two validation stages filter unreliable environments and ineffective masks before balancing the final benchmark.

  • Overview: ABC-Pipeline is an automated workflow that converts open-source backend repositories into full-lifecycle development tasks.Its process is organized into three phases.
  • Repository Exploration: Repository Exploration filters 2,000 MIT-licensed repositories, identifies functional API groups, and generates verification suites for connectivity and logic.The pipeline does not rely solely on potentially incomplete or outdated existing tests.
  • Environment Setup & Verification: Environment Synthesis resolves dependencies, generates container configuration files, and verifies that services build, launch, and listen on expected ports.This phase establishes deployable infrastructure for the selected API targets.
  • Task Instantiation: Task Instantiation masks target endpoint logic, creates natural-language instructions, and packages the repository, environment files, and verification suite.Environment-configuration tasks additionally remove synthesized setup files and provide corresponding instructions.
  • Verification: The two-stage verification protocol first requires intact repositories to pass tests, then requires masked repositories to fail them.This filters unstable runtime configurations and tests that cannot detect removed functionality.
  • Dataset Construction: 600 candidate tasks were filtered into 224 final tasks spanning 8 programming languages and 19 web frameworks, including 92 environment-configuration tasks.The final selection balances language and framework distributions.

5 Experiments

ABC-Bench evaluates diverse models and agents on full-lifecycle backend tasks, revealing substantial variation across models, language stacks, environment setup, and execution depth.

  • 5 Experiments: The evaluation covers diverse open-source and proprietary models, including general, coding-specialized, and agent-oriented systems.The suite includes Qwen3, DeepSeek-V3.2, GLM 4.7, Nex-N1, GPT-5, Gemini 2.5 Pro, and Claude Sonnet 4.5.
  • Full-Lifecycle Tasks Remain Challenging: Claude Sonnet 4.5 achieves the highest overall pass@1 at 63.2%, while DeepSeek-V3.2 reaches around 50% and Qwen3-8B remains below 10%.The benchmark reports average pass@1 under three independent attempts per task using OpenHands.
  • Models Lack Multilingual Robustness: Performance varies substantially by language stack, with Rust scoring 0.0% for most models while Claude Sonnet 4.5 and GPT-5 exceed 30%.Python, Go, and JavaScript generally show higher success rates than less common or more complex language stacks.
  • Correlation Between Interaction Depth and Success: Task success correlates positively with interaction depth at r = 0.87, with Claude Sonnet 4.5 averaging more than 60 turns and Qwen3-8B approximately 10.Top-performing models execute longer trajectories, while weaker models tend to terminate prematurely.

6 Analysis

ABC-Bench analysis shows that performance depends strongly on environment configuration, interaction depth, agent framework, task domain, and model scale. Failures shift from basic setup and syntax issues in smaller models toward more complex logic errors in larger models.

  • Agent Frameworks: OpenHands enables approximately 50% performance for DeepSeek-V3.2 and GPT-5, while mini-SWE-agent reduces GPT-5 below 20%.The framework’s interaction strategy substantially affects holistic-system performance.
  • Environment Configuration: Figure 4 compares Build Success (S1), Conditional End-to-End Success (S2), and Average Pass@1 across 92 environment-related tasks.The comparison includes Claude Sonnet 4.5, GPT-5, DeepSeek-V3.2, and Qwen3-8B.
  • Task Categories: Claude Sonnet 4.5 reaches 86.7% in Analytics, while GPT-5 surpasses it in Identity at 73.3% versus 60.0%.The heatmap reveals substantial variation across application domains.
  • Error Analysis: Environment configuration remains a universal bottleneck, with Qwen3-8B recording 76 Path Missing errors versus 19 for GPT-5.Larger models show stronger syntax and compilation robustness, but their failures contain more Logic Errors: 30 for GPT-5 and 49 for Qwen3-Coder-480B-A35B-Instruct.

7 Conclusion

ABC-Bench evaluates full-lifecycle backend engineering across diverse repositories, languages, and frameworks. The conclusion reports that current systems remain unreliable, with environment configuration and deployment often blocking validation of code logic.

  • Conclusion: ABC-Bench contains 224 tasks sourced from 2,000 GitHub repositories, spanning 8 programming languages and 19 frameworks.The benchmark targets full-lifecycle backend software engineering.
  • Conclusion: Current systems remain far from reliable on complex backend tasks, with environment configuration and deployment often preventing code logic validation.The authors position these findings as a gap between model capabilities and practical backend-engineering demands.

A Experiment Detail

The experiments standardize model access, inference settings, containerized evaluation, and repeated runs to support controlled comparisons. They also include agent-framework evaluation and Qwen3 fine-tuning under specified computational settings.

  • Experiment Setup: Inference uses temperature 1.0 for reasoning-oriented models and 0.7 for non-reasoning models across model-framework combinations.The settings are held consistent for controlled comparisons.
  • Model Configurations: Closed-source models and several open models use official APIs, while Qwen models are deployed on up to 128 NVIDIA H100 GPUs with SGLang.YARN-based RoPE scaling extends the Qwen3 Instruct context window.
  • Evaluation: Evaluations run in containerized environments managed by Terminal-Bench, comparing OpenHands, mini-SWE-agent, and Claude code under consistent task constraints.The setup provides Docker runtime isolation and standardized timeouts, concurrency, and environment settings.
  • Reliability: Each model is evaluated three times, with mean performance reported and error distributions analyzed to reduce sensitivity to stochasticity and outliers.The procedure also examines run-to-run variability.
  • Training: Qwen3 models at 8B and 32B scales are fine-tuned for 3 epochs on 128 NVIDIA H100 GPUs using 128K sequences and a global batch size of 128.Training uses Megatron-LM, Adam optimization, cosine learning-rate scheduling, and warmup.

B Metric Detail

ABC-Bench separates environment construction from functional correctness using two sequential success stages. S1 measures whether services build and start, while S2 measures functional tests only among S1 successes.

  • Metric Definition: The 92 environment-related tasks are decomposed into Environment Build (S1) and Functional Execution (S2).This separates environment configuration outcomes from functional logic correctness.
  • Metric Definition: S1 requires a valid Dockerfile, successful image construction, and container initialization without fatal errors.It measures whether the service environment can be constructed and started.
  • Metric Definition: S2 measures the functional-test pass rate among tasks that successfully pass S1.It evaluates functional logic within the established environment.
  • Metric Definition: When Nbuild = 0, Success RateS2 is defined as 0.This conditional formulation decouples S2 from the environment bottleneck identified in S1.

C Dataset Detail

ABC-Bench is built from licensed public repositories and focuses on technical backend artifacts while excluding sensitive credentials and personal information. Its 224 tasks span diverse languages, frameworks, and task categories, with primarily automated construction and internal quality verification.

  • Licensing: ABC-Bench uses public GitHub repositories explicitly released under the MIT license and excludes proprietary or unclear-license code.The authors state that use remains consistent with the original artifacts' open-source licensing.
  • Data handling: The benchmark uses repository snapshots and automated filtering to detect and redact access tokens, API keys, and private keys.The dataset avoids logs and other user-generated content containing personal information.
  • Dataset scope: 224 tasks are derived from real-world backend repositories spanning diverse programming languages, frameworks, and task categories.Dataset statistics document distributions over languages, frameworks, and task types.

D Responsible NLP Research Statements

The paper describes a primarily automated, license-conscious construction process with privacy screening and internal quality checks. ABC-Bench task instructions cover backend code implementation, API requirements, and, for environment tasks, Docker-based service setup.

  • Licensing: ABC-Bench and ABC-Pipeline use code from public GitHub repositories explicitly released under the MIT license.The paper states that proprietary code and repositories with unclear licensing are excluded.
  • Quality assurance: The construction process is primarily automated, with voluntary internal manual verification of task titles and descriptions by the research team.The process did not recruit external human subjects or crowdworkers.
  • Privacy: Automated screening removes or redacts dataset items that identify individuals or contain offensive material before release.The screening is intended to protect privacy and anonymize distributed data.
  • Task instructions: ABC-Bench task instructions combine task descriptions, API endpoint specifications, and important notes for backend implementation and environment setup.The benchmark provides separate templates for tasks with and without environment-configuration subtasks.
  • Environment setup: Environment-configuration tasks require agents to inspect a backend project, determine its language and architecture, implement the requested changes, and create a runnable Dockerfile.The Dockerfile instructions require dependencies inside the image, successful build and execution, and placement in the project root.
Loading 2601.11077v1…