Source-linked AI summary
Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
Yipeng Ouyang, Xin Huang, Bingjie Liu, Zhongchun Zheng, Yuhao Gu, Xianwei Zhang
TL;DR
Existing benchmarks provide limited evidence about agent performance in long-horizon, stateful, dependency-driven production workflows. RAMP addresses this gap with unified runtime infrastructure, realistic compiler-construction workloads, staged recovery, and multi-dimensional metrics, while its evaluation reveals substantial gaps between benchmark performance and production robustness.
Problem
Existing evaluations often lack serial dependencies, persistent execution state, process observability, and recovery analysis needed to assess production-level agent behavior.
Method
RAMP decouples workload specification from execution and evaluates heterogeneous models using YatCC workflows, resurrection-based failure decomposition, and multi-dimensional runtime metrics.
Results
RAMP’s findings reveal substantial gaps between benchmark performance and production robustness across task outcomes, execution behavior, artifact continuity, resource efficiency, and downstream recoverability.
Takeaways & Limitations
Practical agent utility cannot be judged by reward alone; assessment must examine time, cost, context use, failure handling, and progress across dependent engineering stages.
Takeaways & Limitations
RAMP focuses on compiler construction, covers 15 model tests, and uses the OpenHands backend, limiting generalization and fine-grained statistical conclusions.
Abstract
from arXiv · showhide
LLM agents are rapidly evolving from coding assistants into autonomous software engineering systems. However, existing evaluation methodologies remain largely centered on static, isolated, and short-horizon benchmarks that fail to capture the dynamic complexity of real-world production workflows. As a result, benchmark performance may poorly reflect practical capability under realistic runtime environments involving long execution chains, tool interactions, dependency management, and iterative feedback loops. We thus present RAMP, a production-grounded infrastructure for assessing long-horizon software engineering agents. Built upon the YatCC integrated platform, RAMP provides a unified runtime assessment architecture through standardized orchestration and execution interfaces. RAMP introduces realistic compiler-construction workloads with serial dependencies and complex toolchain interactions, together with a staged recovery mechanism for analyzing execution behavior under partial workflow failure. The framework further incorporates utility-oriented multi-dimensional metrics that jointly evaluate outcome quality and process efficiency. We conduct runtime assessments across 15 mainstream models and observe substantial capability degradation that remains largely invisible to conventional isolated benchmarks. Task completion rates progressively collapse across serial workflows, dropping from 100% in the initial stage to only 20% in the final stage, while none of the evaluated models successfully completes the entire pipeline. Runtime analysis reveals systematic failure propagation and significant resource inefficiencies, with computational costs differing by up to three orders of magnitude among comparable models. These findings suggest RAMP advances agentic model evaluation toward continuous, runtime-observable, and production-grounded assessment.
1 Introduction
Existing evaluations often isolate short coding tasks, while production engineering requires sustained execution across dependencies, tools, runtime states, and failures. RAMP addresses this gap with production-grounded workflows, recovery analysis, and utility-oriented metrics.
- Motivation: Static benchmarks measure isolated task completion but omit long-horizon dependencies, runtime instability, cascading failures, and adaptive recovery.These omissions can make strong benchmark performance a poor indicator of practical engineering utility.
- RAMP Infrastructure: RAMP provides a unified architecture for evaluating heterogeneous models and agent SDKs through standardized interfaces and orchestration services.The infrastructure is built upon the YatCC integrated serving platform.
- RAMP Workloads: RAMP uses realistic compiler-construction workloads organized as serial dependency chains to expose scaffolding effects and execution complexity.The workflows reflect practical software engineering and infrastructure-development processes.
- Recovery and Metrics: RAMP’s resurrection mechanism restores controlled intermediate states to decompose cascading failures into recoverable diagnostic stages.This makes downstream execution behavior observable when early-stage failures would otherwise conceal it.
- Recovery and Metrics: RAMP introduces utility-oriented metrics covering outcomes, efficiency, runtime cost, workflow robustness, recovery capability, and deployment-oriented behavior.The metrics assess how systems behave throughout execution rather than only whether tasks succeed.
2 Background and Motivation
YatCC supplies sequential compiler-construction workloads and a managed execution environment that model production engineering conditions. These properties expose limitations in evaluations that lack persistent state, intermediate-artifact verification, process observability, and heterogeneous runtime support.
- YatCC Tasks: YatCC compiler workloads contain six sequential stages whose intermediate artifacts become dependencies for subsequent compilation tasks.The stages progress from environment verification and tokenization through AST/ASG generation, LLVM IR, optimization, and RV64 assembly.
- YatCC Tasks: Compiler development models production engineering through large codebases, multi-stage pipelines, strict correctness constraints, coupled dependencies, and iterative tool-assisted debugging.Intermediate artifacts must remain semantically and functionally consistent across the pipeline.
- YatCC Platform: YatCC provides isolated, containerized environments integrating workflow runtime, external toolchains, resource management, deployment, execution, and monitoring.Unified runtime abstractions support development and execution across heterogeneous environments.
- Motivation and Challenges: Existing evaluation approaches lack serial dependency and persistent state, process-level metrics, and infrastructure for systematic cross-model and cross-agent comparison.These limitations prevent analysis of cumulative error propagation, execution dynamics, failure behavior, and recovery capability.
- Requirements: A production-oriented assessment must combine realistic workloads, unified runtime infrastructure, and observability of execution, tools, resources, latency, failures, and recovery.Benchmark comparisons frame these dimensions as requirements for evaluating practical agent capabilities.
3 Design and Implementation
RAMP separates workload specification from execution through a standardized API layer, enabling scalable assessment across heterogeneous models and agent frameworks. Its framework, workloads, and metrics jointly treat outcomes and process dynamics as first-class evaluation objects.
- Architecture: RAMP decouples workload specification from execution through a standardized API layer for heterogeneous model providers and agent frameworks.This design enables scalable tests across different execution configurations.
- Architecture: RAMP is organized around framework, workloads, and metrics as a flexible foundation for end-to-end production-level assessment.The system analyzes both task outcomes and process dynamics.
3.1 Assessment Framework
RAMP provides a unified, reproducible runtime framework for evaluating heterogeneous agentic models and backends in production-oriented workflows. Its layered infrastructure standardizes integration, isolates execution environments, orchestrates serial tasks, and preserves detailed assessment artifacts.
- RAMP organizes runtime assessment into framework, environment, and execution layers for unified, reproducible, and extensible evaluation.
- The framework abstracts heterogeneous agent backends through shared task contexts and standardized execution outputs, enabling direct cross-framework comparisons.
- An OpenAI-compatible API gateway normalizes access to proprietary and open-weight models, reducing backend-specific integration effects on comparisons.
- Each assessment runs in a fresh isolated workspace and container with dependencies pre-installed, preventing cross-run contamination.The environment includes ANTLR, LLVM 14, and pybind11.
- The orchestrator tracks serial task-chain state and pass/fail outcomes, triggers resurrection below a predefined threshold such as 60%, and aggregates scores, logs, trajectories, and process metrics.These artifacts support reproducibility and fine-grained behavioral analysis.
3.2 Long-horizon Workloads
RAMP evaluates agents on continuous compiler-construction workflows whose sequential tasks preserve repository state, context, and artifacts. Its resurrection protocol isolates downstream capability from failures that would otherwise cascade through the pipeline.
- RAMP uses long-horizon compiler-construction workloads arranged as serial dependency chains with preserved context, intermediate artifacts, and runtime state.
- The representative workload contains six sequential tasks requiring agents to complete scaffolded functionality within an existing codebase.
- Persistent execution continuity avoids context resets and exposes continuous development, dependency handling, and cumulative error propagation.
- Resurrection replaces a failed intermediate artifact with a golden artifact and resumes execution from the next task after a score falls below a threshold such as 60%.
- The protocol separates inability to complete the current task from downstream capability given valid prerequisites, improving node-level diagnostic granularity.
- Mode 1 uses resurrection for diagnostic coverage, whereas Mode 2 propagates failed artifacts to measure zero-shot end-to-end pipeline depth.
3.3 Multi-dimensional Metrics
RAMP combines outcome, process, failure, and composite utility metrics to assess both agent performance and runtime efficiency. Its AEI normalizes multiple dimensions so strong task scores cannot fully offset poor resource utilization.
- RAMP combines outcome metrics, process metrics, failure analysis, and composite utility measurements for long-horizon assessment.
- Outcome and Process Metrics: Mean Reward computes a weighted average of task scores with resurrection-aware bonuses, rewarding correctness and autonomous pipeline continuity.Tasks completed without resurrection receive b_i = 1.2; resurrected tasks use b_i = 1.0.
- Outcome and Process Metrics: Process metrics record tokens, interaction turns, command executions, retries, and elapsed wall-clock time at task and end-to-end levels.
- Failure Analysis: RAMP classifies trace-observable failures into reasoning, planning, context, tooling and integration, and infrastructure categories.Only the primary root cause is labeled when multiple failure symptoms co-occur.
- Overall Utility: The Agent Efficiency Index jointly measures task effectiveness and resource efficiency across normalized stage, reward, time, cost, and token dimensions.Each dimension is normalized against the maximum observed value across evaluated models.
- Overall Utility: Equal weighting in AEI prevents strong task scores alone from compensating for poor time, cost, or token efficiency.
4 Results and Analysis
RAMP exposes sharp capability, efficiency, and failure-pattern differences that emerge during sustained serial engineering workflows. No assessed model completes the full six-stage pipeline, while costs and diagnostic bottlenecks vary substantially.
- Capability boundaries: None of the 15 assessed models completes all six stages; Opus-4.7 leads with MR 93.39 but stalls at IR optimization.Opus-4.7 scores 68.4 on T4 despite perfect completion on five tasks.
- Capability boundaries: 100% (T0), 46.7% (T1), 26.7% (T2), 13.3% (T3), 0% (T4), and 20% (T5) are the stage-wise task completion rates.The higher T5 rate shows that task difficulty is not monotonically increasing; accumulated modifications and propagated errors also shape decline.
- Capability boundaries: Average stage-wise scores fall from 100 (T0) to 20.63 (T5), with intermediate values of 76.75, 44.85, 27.95, and 37.60.The sequence includes a nonmonotonic increase at T4 while retaining an overall downward pattern.
- Process efficiency: Total inference costs range from $0.05 for Qwen3-Coder to $126.24 for Opus-4.7, a 2,525x difference.Opus-4.7’s MR of 93.39 costs 14.5x more than DS-v4-Pro’s MR of 85.34, while Qwen3-Coder offers strong resource scores at low absolute cost.
- Failure analysis: Runtime failures are categorized into Reasoning, Planning, Context, Tooling & Integration, and Infrastructure Failure using execution traces and explicit criteria.Context Failure is the most prevalent hardstop failure, affecting 9 of 15 models (60.0%) and concentrating at T2–T3.
- Failure analysis: Planning Failure appears in 8 of 15 models (53.3%), with some agents skipping difficult stages to conserve budget rather than adaptively scheduling work.The skip-first strategy reduces resource consumption but sacrifices end-to-end pipeline completion.
5 Related Work
Prior work spans execution-grounded coding benchmarks, interactive environments, paired interventions, and long-horizon agent evaluation. RAMP combines serial dependency, artifact continuity, resurrection-based failure decomposition, and deployment-oriented process metrics.
- Existing evaluation paradigms: Existing evaluation work includes repository-level coding benchmarks such as SWE-bench and interactive benchmarks such as AgentBench, GAIA, OSWorld, and WebArena.These lines extend evaluation beyond isolated function synthesis toward execution-based verification and tool-mediated interaction.
- Paired evaluation: Paired and augmentation-oriented studies compare agent behavior under controlled conditions, primarily to measure whether external skills improve performance.RAMP instead uses paired serial execution with and without resurrection to decompose hidden downstream failure.
- Long-horizon evaluation: Long-horizon evaluation includes Frontier-Eng, RE-Bench, PaperBench, Voyager, and ProgramBench, which study complex research, engineering, or open-ended settings.These works address increasingly extended agent behavior but pursue different evaluation objectives.
- RAMP’s positioning: RAMP uniquely centers serial dependency, intermediate artifact continuity, resurrection-based failure decomposition, and deployment-oriented process metrics together.Its goal extends beyond determining whether an agent solves a task to observing runtime behavior and workflow execution.
6 Discussion on Limitations and Future Work
The paper identifies scope, sample, framework, and contamination limitations that constrain how broadly RAMP’s findings should be interpreted. Future work targets broader evaluation coverage and improved utility measurement.
- Limitations: RAMP focuses on compiler construction, so its empirical findings should not be assumed to generalize across all software-engineering domains.The methodology is described as transferable, but domains such as machine learning engineering and DevOps remain outside the current scope.
- Limitations: The experiments cover 15 model tests, limiting fine-grained statistical conclusions across model families, agent scaffolds, or repeated runs.The limitation concerns the strength of behavioral-pattern and comparative inferences.
- Limitations: All experiments use the OpenHands backend, so observed planning, tool-use, context-management, and recovery behavior may differ with other agent frameworks.The fixed backend improves internal consistency while narrowing framework-level generality.
- Limitations: YatCC’s public availability creates a potential contamination risk for frontier models that may have encountered parts of its codebase during training.The paper notes that RAMP stresses serial continuity and artifact integration rather than isolated-solution memorization.
- Future work: Future work proposes more models, agent frameworks, and repeated controlled runs, alongside improved AEI weighting or Pareto-front analysis.These directions aim to separate model capability from scaffold and runtime-orchestration effects and make AEI more objective.
7 Conclusion
RAMP argues that static, isolated benchmarks are insufficient for evaluating agentic models in realistic software-development workflows. Its runtime assessment instead examines outcomes, execution processes, resource use, failure handling, and continuity across dependent stages.
- Static and isolated benchmarks inadequately capture long-horizon, stateful, dependency-driven, and resource-constrained software workflows.
- RAMP evaluates agentic systems through serial workflow execution, resurrection-based failure decomposition, and multi-dimensional runtime metrics.
- RAMP assesses task outcomes alongside execution behavior, artifact continuity, resource efficiency, and downstream recoverability.
- Practical utility cannot be judged by reward alone; assessment must examine how agents spend time, cost, and context and handle failures across dependent stages.