PaperlayerView exact revision on arXiv

Source-linked AI summary

ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

Ayush Nangia, Shikhar Mishra, Aman Gokrani, Paras Chopra

arXiv:2602.19594v1cs.LG

TL;DR

Real-world inference optimization requires agents to improve performance while modifying the correct bottleneck, a capability existing evaluations do not fully measure. ISO-Bench addresses this with 54 vLLM and SGLang tasks and combined hard and soft metrics. The results show that hard metrics can overestimate performance, execution is a larger failure mode than understanding, and agent behavior varies across codebases and scaffolds.

  • Problem

    Existing benchmarks do not target the specific challenges of GPU-based inference serving optimization, and hard metrics alone can miss whether agents addressed the intended bottleneck.

  • Method

    ISO-Bench evaluates agents on 54 real optimization tasks from vLLM and SGLang using execution-based metrics, soft bottleneck-targeting judgments, and correctness validation.

  • Results

    Hard metrics alone overestimate agent performance by about 10 - 20 percent; agents often identify the right target but fail to implement working solutions, and rankings vary across codebases.

  • Takeaways & Limitations

    Complete evaluation of inference optimization agents requires both performance outcomes and evidence that the intended bottleneck was correctly targeted.

  • Takeaways & Limitations

    ISO-Bench covers only two codebases and excludes commits modifying more than 10 files, leaving coordinated multi-module and broader systems-level optimizations untested.

Abstract

from arXiv · show

We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the most popular LLM serving frameworks. Each task provides an agent with a codebase and bottleneck description, whereby the agent must produce an optimization patch evaluated against expert human solutions. We curated 54 tasks from merged pull requests with measurable performance improvements. While existing benchmarks heavily use runtime-based metrics, such approaches can be gamed to pass tests without capturing the actual intent of the code changes. Therefore, we combine both hard (execution-based) and soft (LLM-based) metrics to show that both are necessary for complete evaluation. While evaluating both closed and open-source coding agents, we find no single agent dominates across codebases. Surprisingly, agents often identify correct bottlenecks but fail to execute working solutions. We also show that agents with identical underlying models differ substantially, suggesting scaffolding is as important as the model.

1 Introduction

LLM inference optimization is increasingly important, but coding agents that perform well on general software tasks still struggle with real-world optimization. ISO-Bench addresses this gap with 54 tasks and dual evaluation of performance and bottleneck targeting.

  • Motivation: LLM inference engines require increasingly sophisticated systems-level optimization as models and architectures evolve.The relevant work spans memory management, kernel development, and scheduling.
  • Motivation: Existing coding agents perform well on general software-engineering benchmarks but struggle with optimization tasks.KernelBench reports frontier models matching GPU kernel baselines in under 20% of cases, while GSO reports success rates below 5%.
  • ISO-Bench: ISO-Bench introduces 54 real-world optimization tasks drawn from vLLM and SGLang.Each task evaluates an agent patch against a production inference-code setting.
  • ISO-Bench: The benchmark evaluates both performance improvements and whether agents target the correct bottleneck using hard and soft metrics.This distinguishes execution outcomes from the agent’s optimization approach.
  • Findings: Agent performance varies substantially across codebases, and execution failures remain distinct from identifying the correct optimization target.These behavioral findings motivate evaluating both what agents understand and what they successfully implement.

2 Related Work

Prior benchmarks separately evaluate code correctness, kernel efficiency, or repository-level optimization, while coding-agent scaffolding and LLM-based judging provide complementary evaluation perspectives. ISO-Bench builds on these directions for GPU inference serving optimization.

  • Correctness-driven benchmarks: Code-generation benchmarks progressed from standalone functional correctness toward repository-scale tasks requiring navigation and multi-file edits.HumanEval emphasizes pass@k, whereas SWE-bench uses real GitHub issues and executable tests.
  • Efficiency-driven benchmarks: Efficiency benchmarks evaluate actual speedup in addition to correctness, including GPU-kernel and Triton workloads.KernelBench reports fewer than 20% success even for strong models, while TritonBench reports correctness and GPU efficiency.
  • Efficiency-driven benchmarks: Repository-level optimization benchmarks compare agent patches with tests, speedups, or human expert improvements.SWE-Perf, GSO, and SWE-fficiency differ in evaluation design but all target performance-oriented code changes.
  • Coding-agent architectures: Coding-agent interfaces and scaffolding substantially affect software-engineering performance.Prior systems include SWE-Agent, OpenHands, TRAE-Agent, Claude Code, and Codex CLI.
  • LLM-as-a-Judge: LLM-based code judges offer scalable evaluation but remain subject to biases such as favoring longer responses.This motivates careful use of LLM-based assessment alongside execution-based measures.

3 ISO-Bench

ISO-Bench evaluates coding agents on 54 production-grade GPU inference optimization tasks using execution-based and soft assessments. Its framework separates correct, effective optimization from failed execution and accidental speedups while validating functional correctness.

  • 3 ISO-Bench: ISO-Bench targets a gap in existing benchmarks by evaluating GPU-based inference serving optimization in real-world codebases.Tasks are drawn from widely used vLLM and SGLang systems.
  • Task formulation: Each task gives an agent a pre-optimization repository and bottleneck description, requiring a patch that improves the specified benchmark.Patches are evaluated against human expert solutions from the original pull requests.
  • Benchmark construction: 54 benchmark instances comprise 39 vLLM tasks and 15 SGLang tasks with verified benchmarks, model specifications, and evaluation commands.The dataset is constructed through commit extraction, manual curation, and pull-request analysis.
  • Evaluation metrics: Hard metrics measure TTFT and throughput against human baselines, while soft metrics assess whether patches target the human-identified optimization region and approach.The benchmark uses the projects’ original benchmark tools and a 5% threshold for measurement noise.
  • Quadrant framework: The quadrant framework classifies attempts as True Success, Good Intent, Lucky Win, or Complete Failure by combining performance with bottleneck targeting.True Success requires both correct targeting and competitive performance; Hard Success includes Q1 and Q3.
  • Functional correctness: Functional correctness validation checks whether speedups preserve accuracy, especially for Lucky Win cases that may alter model behavior.Accuracy is compared between the unoptimized baseline and agent patch using the LM Evaluation Harness.

4 Experimental Setup

ISO-Bench evaluates coding agents on inference-optimization tasks using isolated, time-limited environments and both execution-based and LLM-based patch assessment. The setup compares agent scaffolds and measures performance against unoptimized baselines, human solutions, and target bottlenecks.

  • Agent Scaffolding: Three coding agents are evaluated: Claude Code, Codex CLI, and TRAE-Agent with Claude Sonnet 4.5 or GPT-5.TRAE-Agent is reported as TRAE (Sonnet) and TRAE (GPT-5) according to its underlying model.
  • Execution Environment: Agents work in isolated git worktrees for 120 minutes per task, using Docker containers to explore, modify, test, and commit patches.They receive a structured optimization prompt that withholds the solution.
  • Evaluation: Hard metrics measure TTFT and throughput on NVIDIA H100 GPUs against unoptimized baselines and human solutions.Functional and performance evaluation is executed through benchmark commands.
  • Evaluation: Soft metrics use LLM-based judging to compare agent and human patches for bottleneck targeting and implementation approach.This complements execution-based measurement by assessing whether the intended code regions and strategy were used.

5 Results

Across 54 vLLM and SGLang tasks, agent performance varies sharply by codebase, and hard metrics can overstate success when patches miss the specified bottleneck. Agents commonly understand the target but fail during execution, while scaffolding and strategy strongly influence outcomes.

  • Overall Results: 54 tasks span 39 vLLM tasks and 15 SGLang tasks, with True Success requiring both correct bottleneck identification and measurable improvement.True Success is reported by project in Table 3.
  • Overall Results: Claude Code reaches 46.2% True Success on vLLM, whereas TRAE (GPT-5) reaches 86.7% on SGLang and other agents outperform Claude Code there.Claude Code reaches 26.7% on SGLang.
  • Hard Versus True Success: Claude Code’s SGLang Hard Success rate of 46.7% drops to 26.7% True Success, a 20% gap caused by improvements that miss the specified bottleneck.Across other configurations, vLLM gaps range from 2.6% to 12.8%.
  • Understanding and Execution: Agents frequently identify the correct bottleneck but fail to implement a working optimization, producing the Good Intent Bad Execution outcome.On vLLM, three of four agents have their highest outcome count in Q2.
  • Understanding and Execution: On SGLang, all agents except Claude Code identify the correct target in all 15 tasks, while Claude Code fails to execute 8 of 12 correct targets.The understanding-to-execution gap is narrower on SGLang than on vLLM.
  • Cross-Codebase Generalization: Agent rankings reverse across vLLM and SGLang, with strategies that succeed on one codebase failing on the other.Claude Code’s alternative approaches work on vLLM but fail on SGLang, while reference-matching approaches show the opposite pattern for other agents.
  • Scaffolding Effects: Claude Code achieves 46.2% True Success on vLLM versus 28.2% for TRAE (Sonnet), but TRAE (Sonnet) reaches 80.0% on SGLang versus 26.7% for Claude Code.Both agents use Claude Sonnet 4.5, while their scaffolding differs in exploration, decomposition, and stopping decisions.
  • Correctness: A Lucky Win can combine strong hard metrics with the wrong target and catastrophic correctness loss, as accuracy falls from 32% to 0% in one example.The patch hardcodes tensor dimensions and produces garbage outputs, showing why hard metrics alone are insufficient.

6 Discussion

ISO-Bench shows that inference-optimization agents often fail despite understanding the target, while results vary substantially across codebases and evaluation choices. Its scope and measurement design also leave several important boundaries for interpretation.

  • Open-Source Model Failures: None of the evaluated open-source models produced a working optimization, with failures clustering into recurring patterns.The observed patterns included failing to attempt the task, interacting without completion, misunderstanding the environment, and failing during execution.
  • Key Findings: Agents often identify the correct optimization target but fail to implement working solutions.The paper characterizes this understanding-execution gap as a primary failure mode.
  • Key Findings: Hard metrics alone can overestimate performance by about 10 - 20 percent because agents may obtain speedups without fixing the true bottleneck.Soft metrics help identify these lucky wins by checking bottleneck targeting and implementation approach.
  • Key Findings: Performance does not transfer cleanly across codebases, and scaffolding choices matter as much as the underlying model.The paper reports that rankings flip between vLLM and SGLang.
  • Limitations: ISO-Bench covers only 54 tasks across two codebases and excludes commits modifying more than 10 files.This biases the benchmark toward localized patch-sized optimizations and leaves coordinated multi-module speedups untested.
  • Limitations: The benchmark also faces contamination risk, single-judge soft-metric validation, and hardware limitations from using one NVIDIA H100 GPU.The authors note that public pull requests may have appeared in training, soft metrics lack human-annotator validation, and multi-GPU or non-NVIDIA optimization is excluded.

A Reproducibility

The authors release the code, dataset, and licensing information needed to reproduce and extend ISO-Bench.

  • Released Artifacts: The complete evaluation harness, agent configurations, and analysis scripts are publicly available.The code is hosted in the ISO-Bench GitHub repository.
  • Released Artifacts: The ISO-Bench benchmark dataset is publicly available on Hugging Face.The dataset is provided as a separate downloadable artifact.
  • Licensing: The code uses the Apache 2.0 license, while the dataset uses the CC BY 4.0 license.The two released artifacts therefore have different reuse terms.

B Task Collection

ISO-Bench tasks were extracted from performance-related commits in vLLM and SGLang, filtered automatically, then manually curated into fair, solution-hidden optimization tasks.

  • Candidate Collection: The benchmark starts from merged pull requests in vLLM and SGLang that demonstrate measurable performance improvements.Candidates are collected through a multi-stage filtering pipeline inspired by GSO.
  • Automated Filtering: The automated pipeline filters commit histories by performance-related keywords, scope, and GPT-5-mini classification.The classifier distinguishes genuine GPU inference optimization from bug fixes, refactoring, and documentation changes.
  • Automated Filtering: Commits modifying more than 10 files are excluded because they are usually system-wide rather than isolated optimization tasks.This scope filter favors patch-sized tasks.
  • Manual Curation: Manual review verifies optimization quality, extracts benchmarking configurations and performance claims, and writes a bottleneck description without revealing the solution.The description is designed to provide enough context for the target while avoiding trivial hints.
  • Final Dataset: 186 vLLM candidates became 39 final tasks, while 201 SGLang candidates became 15 final tasks.Further reductions removed candidates requiring multi-GPU support, very old commits, or unavailable Docker images.

C Task Setup and Agent Configurations

ISO-Bench gives agents identical structured tasks and reproducible execution environments, then compares configurations and analyzes representative successes and failures.

  • Task Specification: Each task specifies repository metadata, runner requirements, target files, and optimization constraints.The metadata includes the human and parent commits, GPU and software versions, and optimization details.
  • Execution Environment: Agents run in isolated Docker containers and git worktrees with a single NVIDIA H100 GPU, reproducible dependencies, and a 120-minute task limit.Each run is captured as a git commit for later analysis.
  • Agent Configurations: The evaluation uses four agent configurations and compares scaffolding effects by running TRAE-Agent with Claude Sonnet 4.5 and GPT-5.TRAE-Agent provides full trajectory visibility, while Claude Code uses opaque proprietary scaffolding.
  • Q1 True Success: +21.09% throughput improvement was achieved in a FlashAttention task by moving tensor reshapes outside the custom operation.TRAE (Sonnet) matched the human optimization strategy for reducing CPU overhead outside CUDA-graph regions.
  • Q2 Good Intent, Bad Execution: A Qwen3 parser patch targeted the correct file and approach but failed during benchmarking.The case illustrates correct bottleneck understanding without executable implementation.
  • Q3 Lucky Win: A Bamba-9B optimization increased hard metrics but reduced exact-match accuracy from 32% to 0%.The regression followed replacement of dynamic dimension preservation with hardcoded dimensions.
  • Q4 Complete Failure: A prefix-caching task failed because the agent modified tests and reimplemented allocator logic without addressing the bottleneck.The patch achieved no optimization classification.
  • Open-Source Failure Patterns: MiniMax-M2.1 generated 81,782 output tokens over 477 seconds but made zero tool invocations.The model verbalized plans for 75 steps without executing them.

D.7 Case Study: GLM-4.7 Task Completion Failure

The case studies show that coding agents can target real inference-optimization bottlenecks yet fail through non-executing code, incorrect outputs, missed targets, or workflow errors. Across examples, successful optimization required both correct bottleneck identification and effective task execution.

  • True Success: +21.09% throughput was achieved when TRAE correctly moved tensor reshapes outside the FlashAttention custom operation.The change reduced non-CUDA-graph CPU overhead while preserving the intended optimization target.
  • Good Intent, Bad Execution: TRAE identified the Qwen3 parser bottleneck but failed to produce executable code for the proposed regex-to-string-operation optimization.The intended change was to replace re.compile() and findall() with str.partition() for faster extraction.
  • Lucky Win: TRAE produced a speedup for the Bamba mixer while causing a 100% accuracy regression.The case is labeled a lucky win because the agent reduced tensor-operation overhead without preserving functional correctness.
  • Complete Failure: TRAE completely missed the BlockManagerV2 prefix-caching warmup target despite producing complex code.The intended fix was to track touched blocks and mark cache-hit blocks as computed after batch scheduling.
  • Execution Failures: MiniMax-M2.1 made zero tool calls during 75 reasoning steps, while GPT-OSS-120B created mock dependencies instead of modifying the scheduler.These examples illustrate failures to translate reasoning into repository-level execution and to understand the target environment.

Case Study: GLM-4.7 Task Completion Failure (OSS Failure)

GLM-4.7 made substantial valid edits to the target files but failed to complete the task workflow. It exhausted the execution budget without calling the finish command.

  • Valid Edits: 386 tool calls included 59 successful str replace edits to the target files.The model therefore executed valid code modifications before the task-completion failure.
  • Task Completion Failure: The case separates implementation progress from workflow completion: successful optimizations did not yield a completed task.The execution trace records valid edits followed by failure to recognize when to finalize.
Loading 2602.19594v1…