PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Coding Agents for System-Level Performance Optimization

Yanze Zhang, Maggie Yongqi Guan, Shudong Liu, Qi Hao, Kanye Ye Wang, Meng Qu, Xiaobo Zhou

arXiv:2610.29695v1cs.SE

TL;DR

Existing coding benchmarks largely assess function- or repository-level tasks, leaving agents’ system-level performance optimization capabilities less explored. MicroPerf-Bench evaluates this capability through controlled anti-pattern injection, and the best configuration achieves an average normalized score of 42.0 out of 100.

  • Problem

    Existing benchmarks largely evaluate function- or repository-level tasks, leaving system-level optimization across interacting microservices less explored.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    MicroPerf-Bench injects empirically grounded performance anti-patterns into real-world microservice systems while preserving functionality and measuring mixed-workload performance.

    Source: §1 · p. 2

  • Results

    42.0 out of 100 was the best configuration’s average normalized score, and functionally correct changes often produced no measurable end-to-end performance gain.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Identifying true runtime bottlenecks remains a central challenge for coding agents optimizing system-level performance.

    Source: §1 · p. 2, §6 · p. 10

    Profiling benefits were not uniform across systems, and the reported single-run findings require further replication.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Coding agents are becoming an increasingly important paradigm for automating software engineering tasks. However, existing coding benchmarks primarily evaluate agents at the function or repository level, leaving their capabilities in more complex distributed software systems largely unexplored. In practice, many modern applications adopt microservice architectures, where end-to-end performance emerges from interactions among multiple independently running services, requiring agents to reason beyond local code changes. To study this capability, we introduce MicroPerf-Bench, a benchmark for evaluating coding agents on end-to-end performance optimization in microservice systems. Constructing such a benchmark is challenging because reproducible system-level performance fixes are sparse and difficult to attribute. We address this challenge by injecting empirically grounded performance anti-patterns into real-world open-source systems while preserving functionality and ensuring measurable performance degradation. The resulting benchmark contains 65 optimization instances across seven systems and nine anti-patterns, each with an executable environment, functional tests, and mixed-workload evaluation. Across 1,755 runs covering nine agent-model configurations, the best configuration recovers only 42.0% of the injected performance degradation. The dominant failure is not invalid code, but functionally correct and constraint-compliant changes that fail to improve end-to-end performance, highlighting system-level bottleneck localization as a key challenge for current coding agents. The source code and data are available at https://github.com/zyzcjnb/microperf.

1 INTRODUCTION

Existing coding-agent benchmarks emphasize functional correctness and performance within functions or repositories, leaving end-to-end optimization across interacting services less explored. MicroPerf-Bench evaluates this system-level capability, and current agents often make functionally correct changes without improving end-to-end performance.

1 INTRODUCTION

  • Existing performance benchmarks largely assess functions or repositories, leaving optimization across interacting components and end-to-end system behavior less explored.System performance depends on workload, execution environment, and runtime interactions.Source: §1 · pp. 1–2
  • MicroPerf-Bench injects empirically grounded anti-patterns into real-world microservice systems while preserving functionality and creating measurable degradation.It contains 65 instances across seven systems and nine anti-patterns, with executable environments, functional tests, and mixed-workload evaluation.Source: §1 · p. 2
  • 42.0 out of 100 was the best configuration’s average normalized score, and functionally correct modifications often yielded no measurable end-to-end performance gain.The evaluation covered nine agent–model configurations; the authors identify bottleneck localization as a central challenge.Source: §1 · p. 2

2 BACKGROUND AND MOTIVATION

Microservice performance emerges from runtime interactions across services, which existing function- and repository-level benchmarks do not fully model. Empirically grounded performance anti-patterns offer a way to construct realistic, attributable system-level optimization tasks.

2 BACKGROUND AND MOTIVATION

  • End-to-end latency in microservices depends on interactions across service execution paths, including communication overhead, dependency topology, serialization, and resource contention.A user request may trigger inter-service calls, asynchronous messages, and shared database transactions.Source: §2 · p. 2
  • Existing benchmarks typically treat a self-contained program or repository as the unit of optimization, leaving cross-service structural changes and deployed multi-service evaluation unmodeled.Local code efficiency may not yield system-level gains when downstream bottlenecks or concurrency dynamics change under load.Source: §2 · p. 3
  • Recurring performance anti-patterns provide realistic, actionable bugs whose performance recovery can be attributed under controlled experimental conditions.Examples include excessive fine-grained RPC calls and multiple services coupled directly to one database.Source: §2 · p. 3

3 MICROPERF-BENCH

MicroPerf-Bench provides deployable microservice optimization tasks built through controlled injection of performance anti-patterns into real systems. Its task formulation constrains shortcuts, while its evaluation combines functional gates with mixed-workload latency scoring.

3 MICROPERF-BENCH

  • MicroPerf-Bench comprises 65 system-level optimization instances from seven open-source microservice applications and nine anti-patterns, each with a deployable environment, tests, and profiling scripts.The benchmark covers systems including Robot Shop, Pitstop, and three DeathStarBench suites.Source: §3 · p. 3

3.1 TASK FORMULATION

  • Agents receive an under-optimized multi-service codebase, deployment configurations, mixed-workload scripts, and task constraints, then return modified code or configuration files.The harness rebuilds affected images, redeploys the system, and runs correctness tests and workload profiling.Source: §3.1 · pp. 3–4

3.2 DATA CONSTRUCTION

  • Because mined fixes are sparse and entangled, benchmark instances use anti-pattern injections screened for measurable degradation and actionable remediation.The construction preserves API contracts and functional completeness; single-pattern injections require at least 20% end-to-end latency degradation relative to the clean reference.Source: §3.2 · pp. 4–5
  • Three experts independently validated all 65 instances, diagnosing the injected pattern from runtime behavior alone in 81.5% of cases.A blinded study found 47.2% accuracy in distinguishing injected instances from real parent-commit defects.Source: §3.2 · p. 5

3.3 BENCHMARK EVALUATION PROTOCOL AND METRICS

  • Evaluation uses average end-to-end latency under mixed workloads, requiring functional correctness and fairness compliance before scoring recovery of injected degradation.Runs include a 90-second warm-up and three averaged measurements; the normalized score ranges from 0 for no improvement or a gate failure to 100 for full recovery.Source: §3.3 · p. 5

4 EXPERIMENTS

Across 65 instances, baseline agents recover only part of the injected degradation, and profiling results vary by system. Experiments compare agent configurations, classify failures, and examine representative optimization trajectories.

4 EXPERIMENTS

  • The experiments evaluate agent capabilities across models and scaffolds, analyze baseline performance and failure modes, test runtime profiling, and present case studies.The primary evaluation comprises 1,755 optimization trials.Source: §4 · p. 6, §4.1 · p. 6

4.1 EVALUATION SETUP

  • Nine configurations combine three agent scaffolds and three base models; each was run three times on all 65 instances under a two-hour per-instance budget.Evaluation used functional and fairness gates, mixed workloads, and three averaged end-to-end latency measurements in a dedicated server environment.Source: §4.1 · p. 6

4.2 OVERALL BASELINE PERFORMANCE

  • 42.0 was the highest baseline average score, while partial-score outcomes (27.4% of trials) outnumbered full-score outcomes (16.7%).The positive-score rate averaged 44.0% across configurations; Table 1 reports the nine configurations’ three-run means over 65 instances.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 EFFECTS OF BASE MODEL AND AGENT SCAFFOLD

  • Across scaffolds, GLM outscored DeepSeek by 12.7 points and Qwen outscored DeepSeek by 9.3; scaffold comparisons were not statistically significant.OpenHands led OpenCode and Claude Code by 3.2 and 3.3 points, respectively, but neither comparison reached the significance threshold.Source: §4.3 · p. 7

4.4 FAILURE MODE ANALYSIS

  • Within zero-score trials, “no performance gain” was the dominant failure mode, accounting for 48.7% of all failures; agents often wrote functionally correct, compilable code that met constraints but did not improve end-to-end latency.“No gain” constituted 67.6% of GLM’s failures. RobotShop and Pitstop failed almost exclusively due to “no gain” (67.7% and 80.0%, respectively), whereas PetClinic and Social Network had more functional correctness errors.Source: §4.4 · pp. 7–8, Figure 2 · p. 8

4.5 PROFILING-TOOL AUGMENTATION

  • The profiling-augmented Claude Code + Qwen run scored 47.7, above its baseline mean of 39.4 and the highest baseline configuration’s 42.0.Improvements varied by system, with no change on Martian and decreases on PetClinic and Pitstop; the augmented evaluation was a single run.Source: §4.5 · p. 8, Figure 3 · p. 8

4.6 CASE STUDIES

  • Case studies show that correct bottleneck localization can enable full recovery, while static reasoning can leave severe runtime bottlenecks untouched.OpenHands + GLM recovered the Martian instance to a score of 100; in the hotel case, profiling helped Claude Code + Qwen identify the shared-DB path and score 88.8.Source: §4.6 · p. 9

5 RELATED WORK

Prior coding-agent performance benchmarks have progressed from function-level to repository-level tasks, while system-level optimization remains a distinct setting. MicroPerf-Bench evaluates end-to-end optimization across interacting microservices while preserving functionality.

5 RELATED WORK

  • MicroPerf-Bench extends performance optimization benchmarks from function and repository levels to system-level behavior emerging from independently running services.Prior microservice benchmarks mainly studied system performance, resource efficiency, reliability, testing, and operation; more recent work evaluated end-to-end code generation.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents MicroPerf-Bench as a benchmark for end-to-end microservice optimization and reports that current agents struggle to recover injected degradation. Its results point to runtime bottleneck identification as a challenge and suggest that profiling evidence can help.

6 CONCLUSION

  • Across nine configurations, the best average normalized score was 42.0 out of 100, and many correct attempts produced no measurable end-to-end improvement.The benchmark contains 65 instances across seven systems and nine injected anti-patterns; performance also varies across host systems and repeated runs.Source: §6 · p. 10

AI USE STATEMENT

The authors report that generative AI tools were not used for research tasks requiring mandatory disclosure, including research design and implementation. They used such tools for language editing and limited assistance with simple experimental scripts, and say authors reviewed and verified all AI-assisted content.

AI USE STATEMENT

  • The authors report no generative AI use for research tasks requiring mandatory disclosure, such as designing experiments, processing datasets, or interpreting results.Source: AI USE STATEMENT · p. 10
  • Generative AI was used for language editing and limited help with simple experimental scripts, not for research methods, benchmark artifacts, or evaluation logic.The authors state that core technical artifacts, including anti-pattern injections and manual optimizations, were produced without generative AI assistance.Source: AI USE STATEMENT · p. 10
  • The authors say they reviewed and verified AI-assisted content, including manually checking, executing, and testing AI-assisted code.They take responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The authors release benchmark materials and per-run results so the evaluation can be reproduced and re-aggregated under alternative protocols.

REPRODUCIBILITY STATEMENT

  • The release includes benchmark instances, evaluation materials, all 1,755 per-run scores, and zero-score failure labels for recomputing aggregates under alternative protocols.Degraded-baseline latency was constant across the three runs for every instance, making repetitions directly comparable.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The authors describe voluntary expert participation and a pre-study showing that attributable system-level performance fixes are sparse, motivating injection-based construction. They map injected anti-patterns to real-world evidence while noting that injected instances may not capture the full complexity of organically emerging problems.

ETHICS STATEMENT

  • The authors recruited experts through personal networks; participation was voluntary.They respected the open-source systems’ original licenses and sanitized injected variants to mitigate benchmark contamination.Source: ETHICS STATEMENT · pp. 10–11

A PRE-STUDY: MAPPING ANTI-PATTERNS TO REAL-WORLD PERFORMANCE ISSUES

  • 15 of more than 10,000 screened commits (0.15%) met the system-level performance-fix criteria, motivating injection-based construction.The qualifying commits were concentrated in repositories maintained by corporate performance teams.Source: A PRE-STUDY: MAPPING ANTI-PATTERNS TO… · p. 13
  • Seven of nine anti-patterns are corroborated by merged commits or documented industrial cases; the other two draw on an empirical catalog and related evidence.Empty messages traces to web-service anti-pattern literature and detected refactorings; ESB misuse is corroborated by removal of Istio’s centralized Mixer component.Source: A PRE-STUDY: MAPPING ANTI-PATTERNS TO… · p. 13
  • Injected instances may not capture the full complexity of open-ended, organically emerging performance problems, despite grounding operators in a catalog and corroborating them with evidence.Some anti-patterns, such as Empty messages, are persistent design choices rather than one-shot fixes, so their evidence is industrial rather than commit-level.Source: A PRE-STUDY: MAPPING ANTI-PATTERNS TO… · p. 13

B TASK SPECIFICATION PROVIDED TO AGENTS

The agent task specification constrains optimization to code logic, architecture, or topology while preserving functionality and safeguards. Expert validation assesses diagnosability and apparent provenance, and configurations use shared models and controlled, independent runs.

B TASK SPECIFICATION PROVIDED TO AGENTS

  • Agents may optimize code logic, service architecture, or topology, but cannot tune configuration or add resources, replicas, or instances.Changes must preserve APIs and basic functionality, avoid modifying generated files or test scripts, and retain safeguards such as tracing, durability, security, and correctness semantics.Source: B TASK SPECIFICATION PROVIDED TO AGENTS · p. 15

C EXPERT VALIDATION

  • Injected anti-patterns were correctly diagnosed from runtime behavior in 81.5% of the 65 benchmark instances.The assessment tested diagnosability, not equivalence between injected and naturally occurring defects.Source: C EXPERT VALIDATION · p. 16
  • Reviewers identified defect provenance with 47.2% accuracy across a blinded sample of 12 injected and 12 naturally occurring defects.This descriptive result does not establish identical runtime behavior or optimization difficulty between the two groups.Source: C EXPERT VALIDATION · pp. 16–17

D AGENT CONFIGURATIONS

  • All three scaffolds use the same three model endpoints, temperature, and thinking mode in independent, scripted, non-interactive invocations.The configurations receive identical task specifications and containerized environments, with a two-hour budget per instance and no manual intervention.Source: D AGENT CONFIGURATIONS · p. 17, Table 4 · p. 17

E CASE STUDIES

The case studies show that measured recovery can depend on localizing the dominant bottleneck, while anti-pattern difficulty varies substantially by pattern and workload. Profiling can reorder optimization priorities, and some structural changes remain difficult or error-prone.

E CASE STUDIES

  • In Case A, topology restoration returned latency to the reference run-to-run range, while the sibling nano-only variant scored 0 at 47.68 ms.The combined injection’s measured latency was 9.04 ms versus a 9.22 ms reference and 13.81 ms injected latency; this is convergence to reference-level performance, not a demonstrated improvement beyond it.Source: E CASE STUDIES · pp. 17–18
  • Case B’s edits passed both gates but raised latency from a 4,260.99 ms degraded baseline to 5,951.80 ms, producing a score of 0.The agent batched secondary calls but left the dominant serial memcached reads inside a global mutex; nineteen other runs of the variant scored between 58.6 and 100.Source: E CASE STUDIES · p. 18
  • Profiling-guided optimization in Case C reached 8.05 ms and a score of 88.8 after measurements reprioritized targets toward shared-database serialization and a full-collection scan.The same configuration’s two static-code runs scored 0, while its other baseline run scored 75.3; profiling included at most two measurement cycles.Source: E CASE STUDIES · pp. 18–19
  • No selected system contained an ESB component, so the benchmark has no injected ESB misuse instances.Source: E CASE STUDIES · p. 19

F RESULTS BY ANTI-PATTERN

  • The hub-like dependency pattern averaged 21.6 points with an 8.3% full-score rate, and its scores ranged from 4.3 on Martian and Social to 56.6 on Pitstop.The pattern was hardest on average; its workload-dependent difficulty reflects whether the hub lies on the mixed workload’s critical path.Source: F RESULTS BY ANTI-PATTERN · p. 19
  • On-line only and Chatty service averaged 47.9 and 42.5 points, while 43.3% of On-line only zero-score trials involved correctness errors.Empty messages had a 61.7% positive-score rate but averaged 34.9 points; 87.1% of its zero-score trials were no-gain outcomes.Source: F RESULTS BY ANTI-PATTERN · p. 19
  • The single Mega service instance was never solved; merging more than ten services broke the build in 51.9% of its zero-score trials.The instance was on Social, and the paper attributes the difficulty to large-scale structural consolidation within the two-hour budget.Source: F RESULTS BY ANTI-PATTERN · p. 19

G FAILURE ANALYSIS DETAILS

Zero-score trials are most often no-gain outcomes, but failure profiles vary across systems and models. System difficulty and run-to-run variability also shape how configuration scores should be interpreted.

G FAILURE ANALYSIS DETAILS

  • 48.7% of zero-score trials are no-gain outcomes; correctness errors account for 20.5%, fairness violations 15.3%, and timeouts 1.1%.Build and deploy/startup failures are reported jointly as 14.4%.Source: G FAILURE ANALYSIS DETAILS · p. 19
  • Martian failures split between no-gain outcomes (46.9%) and fairness violations (48.5%), while PetClinic has a 26.4% deploy/startup share and Social has 44.2% correctness errors and 21.2% build failures.RobotShop and Pitstop failures are dominated by no-gain outcomes (67.7% and 80.0%).Source: G FAILURE ANALYSIS DETAILS · p. 20, Table 8 · p. 20
  • Failure profiles vary more by model than scaffold: GLM-5.2 has 67.6% no-gain failures, Qwen3.7-Max has 25.1% fairness violations, and DeepSeek-V4-Pro has the most build and deploy/startup failures.OpenHands had no timeouts; OpenCode accounted for most timeouts (2.1%).Source: G FAILURE ANALYSIS DETAILS · p. 20, Figure 4 · p. 20

H PER-SYSTEM SCORES BY CONFIGURATION

  • System difficulty is consistent across configurations, but no configuration dominates every system; OpenHands + GLM is the most balanced and has no near-zero system.Martian, Social, and RobotShop have no score cell above 21, while Hotel, PetClinic, and Pitstop are tractable for most configurations.Source: H PER-SYSTEM SCORES BY CONFIGURATION · pp. 20–21, Figure 5 · p. 21

I SINGLE-PATTERN VS. COMBINED-PATTERN INSTANCES

  • Combined-pattern instances score 35.9 versus 34.0 for single-pattern instances, but the +4.3-point within-system difference is not significant (p = 0.170).System difficulty transfers strongly between subsets (Pearson r = 0.96); the comparison is descriptive because pattern selection is confounded.Source: I SINGLE-PATTERN VS. COMBINED-PATTERN… · p. 21, Figure 6 · p. 22

J RUN-TO-RUN STABILITY

  • Across three runs, configuration standard deviations range from 1.0 to 5.9, and 23.1%–50.8% of instances are flaky.Pooled Best@3 is 53.7 versus an overall mean of 34.8; single-run evaluations can underestimate capability and reverse configuration rankings.Source: J RUN-TO-RUN STABILITY · pp. 21–22, Table 9 · p. 22

K COST AND TIME ANALYSIS

Token costs vary primarily by scaffold and do not track performance, while all configurations finish well within the time budget. The evaluation uses fixed baselines and workloads, and its success criterion remains robust under stricter thresholds.

K COST AND TIME ANALYSIS

  • Claude Code consumes 2.8× more billing-equivalent tokens than OpenCode on average (1.31M vs. 0.47M), and 2.3× more than OpenHands (0.58M).Model averages are closer: GLM 0.70M, Qwen 0.81M, and DeepSeek 0.84M tokens per instance.Source: K COST AND TIME ANALYSIS · pp. 22–23, Table 10 · p. 23
  • Token cost and average score are negligibly correlated (Pearson r = −0.20, p = 0.60), while score-per-cost varies 6× across configurations.OpenCode + DeepSeek yields 84.2 score points per 1M billing-equivalent tokens; Claude Code + DeepSeek yields 13.9 and is the most expensive, worst-scoring configuration.Source: K COST AND TIME ANALYSIS · p. 23, Figure 7 · p. 23
  • Mean session durations of 7.7–18.0 minutes use only 6.4–15.0% of the two-hour budget across configurations.Early termination, rather than budget exhaustion, is the norm.Source: K COST AND TIME ANALYSIS · p. 23, Figure 7 · p. 23
  • The reported cost figures use plan-specific billing weights and cover only the nine baseline configurations, excluding profiling-tool augmentation.Absolute costs can shift with pricing, though the passage reports the scaffold ranking as independent of the exact weights.Source: K COST AND TIME ANALYSIS · p. 23

L MEASUREMENT PROTOCOL AND SCORE ROBUSTNESS

  • The evaluation reuses frozen reference and degraded-baseline latency averages across trials and applies the same fixed-rate workload, endpoint mix, and input dataset.Each deployment is measured over three workload iterations; resource limits are fixed, and service merging or splitting cannot exceed the original system’s aggregate CPU and memory budget.Source: L MEASUREMENT PROTOCOL AND SCORE… · p. 24
  • Under a 10% minimum-improvement threshold, the positive rate falls to 39.9%, while model-level conclusions remain unchanged.Only 1.2% of positive trials improve latency by less than 1%; the median improvement is 62.8%.Source: L MEASUREMENT PROTOCOL AND SCORE… · p. 24
  • Among partial-score trials, 29.6% are already within 1.2 times reference latency; the median relative improvement is 51.5%.Overall, 19.8% of trials reach LO ≤ 1.1 LR and 24.6% reach LO ≤ 1.2 LR.Source: L MEASUREMENT PROTOCOL AND SCORE… · p. 24

Related papers

Loading 2610.29695v1…