PaperlayerView exact revision on arXiv

Source-linked AI summary

ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents

Jiawei He, Jie Jia, Chenbo Liu, Chaoyi Xue, Yapeng Song, Xikai Yang, Dong Sun

arXiv:2605.20251v4cs.SEcs.AI

TL;DR

Outcome-based coding-agent benchmarks provide limited visibility into execution defects. ProcCtrlBench evaluates standardized agent trajectories with a reusable defect ontology, calibrated scorecards, and control-preservation measures. Across its evaluation cases, it reports useful reliability and process-level insights beyond outcome-based assessment.

  • Problem

    Existing coding-agent benchmarks primarily assess final outcomes, providing limited visibility into defects that arise during execution.

    Source: §1 · p. 1

  • Method

    ProcCtrlBench standardizes agent trajectories, evaluates them against an ontology of 11 execution-defect types, and reports calibrated process scorecards incorporating control preservation.

    Source: §1 · pp. 1–2

  • Results

    Across 200 annotated cases, ProcCtrlBench showed useful reliability, improved interpretability relative to direct thresholding, and diagnostic insight beyond outcome-based evaluation.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Process-aware scorecards provide diagnostic insight beyond conventional outcome-based evaluation.

    Source: §5 · p. 10

    The study focuses on coding agents, and additional validation is needed before extending strong conclusions to other agent domains.

    Source: §5 · p. 10

Abstract

from arXiv · show

Existing benchmarks for LLM coding agents primarily evaluate final outcomes. While useful for measuring overall capability, these metrics provide limited visibility and often miss defects that arise during execution. We present ProcCtrlBench, a benchmark for execution-process evaluation in LLM coding agents. ProcCtrlBench organizes recurrent execution defects into a reusable ontology covering 11 defect types in 4 categories, and evaluates agent trajectories through standardized process evidence rather than final outcomes alone. To support comparison across heterogeneous agents, ProcCtrlBench standardizes raw logs into a unified trajectory representation and reports calibrated scorecards over process-level findings. In addition, ProcCtrlBench uses control preservation as a way to quantify execution-process quality, capturing whether execution remains interpretable, interruptible, correctable, reversible, and able to hand back authority when needed. We evaluate ProcCtrlBench on 200 cases sampled from three benchmarks: AndroidBench, TerminalBench, and SWE-bench-Verified. Results show that ProcCtrlBench can be instantiated with useful reliability, provides more stable semantics than direct thresholding, and reveals meaningful differences in execution quality that are often overlooked by conventional outcome-based evaluation.

1 INTRODUCTION

Outcome-based benchmarks measure task success but provide limited visibility into execution defects that can affect supervision and usability. ProcCtrlBench evaluates complete trajectories with a reusable defect ontology, standardized evidence, calibrated scorecards, and control-preservation measures; tests on three benchmarks show useful reliability and process differences beyond outcome-based evaluation.

1 INTRODUCTION

  • Existing benchmarks focus on final outcomes, limiting visibility into execution defects that may persist even when an agent eventually solves a task.Trajectories can be repetitive, inefficient, weakly structured, or difficult to supervise.Source: §1 · p. 1
  • ProcCtrlBench organizes recurrent execution defects into a reusable ontology of 11 types across four categories and evaluates complete trajectories using standardized process evidence.The categories are context management, tool-use efficiency, workflow architecture, and tool-ecosystem consistency.Source: §1 · pp. 1–2
  • Control preservation quantifies whether execution remains interpretable, interruptible, correctable, reversible, and able to hand back authority when needed.The benchmark complements rather than replaces outcome-based evaluation.Source: §1 · p. 2
  • The benchmark standardizes heterogeneous logs into a unified trajectory representation and reports calibrated scorecards over process-level findings.The scorecard includes defect-level findings, dimension-level process-quality scores, control-preservation scores, and a compact summary score.Source: §1 · p. 2
  • ProcCtrlBench’s evaluation on 200 cases shows useful reliability, more stable semantics than direct thresholding, and execution-quality differences often missed by outcome-based evaluation.The cases were sampled from AndroidBench, TerminalBench, and SWE-bench-Verified.Source: §1 · p. 2

2 RELATED WORK

Related work has increasingly examined process-sensitive evaluation because endpoint success does not reveal how multi-step systems reach their results. ProcCtrlBench brings reusable process-failure benchmarking, calibration of trajectory findings, and control preservation together for coding agents.

2 RELATED WORK

  • Process-sensitive evaluations such as ProcBench and TRACE move beyond endpoint scores, while ProcCtrlBench focuses specifically on execution-process defects and control preservation in coding agents.The cited work discusses procedural inference and preserving reasoning-related behavior under continual learning.Source: §2 · p. 2
  • Prior process-sensitive work often centers on qualitative diagnosis, localized analysis, or task-specific case studies; ProcCtrlBench instead seeks reusable benchmark points for structured cross-system comparison.Related efforts examine planning traces, tool-use trajectories, error recovery, and failure analysis.Source: §2 · p. 2
  • Existing calibration methods typically address single-turn responses or final predictions, whereas ProcCtrlBench calibrates process-level findings across defect types and task settings.Its aim is stable, comparable semantics for findings from sequential execution trajectories.Source: §2 · p. 3
  • Oversight research emphasizes understandable, interruptible, correctable behavior and safe hand-back; ProcCtrlBench uses control preservation to quantify execution quality and combine it with defect findings.These concerns are particularly relevant to long-horizon coding agents.Source: §2 · p. 3

3 METHODS

ProcCtrlBench evaluates complete coding-agent trajectories through a standardized representation, defect evidence, calibrated risks, and a structured scorecard. It treats control preservation as a distinct dimension of execution quality, alongside defect burden.

3.1 PROCCTRLBENCH OVERVIEW

  • The overview standardizes raw executions, separates evidence extraction, risk estimation, and reporting, then presents a full scorecard as the primary output and a scalar only for coarse comparison.The scorecard includes process-defect burden, dimension-level quality, control preservation, and an overall summary.Source: §3.1 · pp. 3–4

3.2 STANDARDIZED TRAJECTORY REPRESENTATION

  • A deterministic mapping converts heterogeneous logs into ordered standardized events, a common interface for downstream evidence extraction using features such as counts, similarity, dependency structure, and context transitions.Events capture type, text, tool invocation and return, external operation, context state, and structural dependencies.Source: §3.2 · p. 4

3.3 DEFECT TAXONOMY AND BENCHMARK INSTANTIATION

  • The reusable ontology covers 11 defect types in four categories; each class receives structured, continuous evidence, with exemptions for contextually appropriate patterns and a distinction between the ontology and its current detector-and-scoring instantiation.Exemptions include intended validation reruns and necessary long-lived instruction text.Source: §3.3 · p. 4

3.4 CALIBRATED PROCESS SCORING

  • Because raw detector scores can have different defect likelihoods across contexts, ProcCtrlBench calibrates posterior risks, maps them to warning/error/none reporting bands, and aggregates risks into dimension-level and overall quality scores.The semantic labels are reporting aids, while the full scorecard remains more important than a single scalar.Source: §3.4 · pp. 4–5

3.5 CONTROL PRESERVATION AND BENCHMARK OUTPUT

  • Control preservation covers interpretability, interruptibility, correctability, reversibility, and authority handoff; it is not simply the inverse of defect burden, and the full scorecard is primary over compact scalar comparison.Calibrated findings and control state may also illustrate execution-time governance uses such as confirmation, rollback, or handoff, but these uses are not part of the benchmark definition.Source: §3.5 · p. 5

4 EXPERIMENTS

The experiments assess ProcCtrlBench’s reliability, calibration, cross-system comparisons, and robustness across task scenarios. They find that it adds process-level diagnostic detail while generally preserving broad system rankings.

4.1 EXPERIMENTAL SETUP

  • The study samples 200 tasks, annotates process defects, splits cases for development, calibration, and evaluation, standardizes full logs, and reports detection, calibration, process, and control metrics.The evaluation spans SWE-bench-Verified, AndroidBench, and TerminalBench, with cases stratified by benchmark source and task success across the splits.Source: §4.1 · p. 6

4.2 RQ1: INSTANTIATION RELIABILITY

  • Locally observable defects achieve stronger detection performance, while higher-level structural defects are harder to instantiate; observability varies by defect type.The more difficult examples include Wrapper Workflow, Context Coupling, and Weak Tool.Source: §4.2 · p. 6
  • Long Chain and Dead Step show the strongest positive correlations with downstream task failure, followed by Context Window Thrashing and Context Coupling.Duplicate Step and Ghost Context also exhibit meaningful positive correlations.Source: §4.2 · p. 6, Figure 2 · p. 7

4.3 RQ2: CALIBRATION QUALITY

  • Across all four defect dimensions, hierarchical Bayesian calibration yields lower ECE than direct hard-threshold interpretation, with the largest gains in less observable dimensions.The authors caution that these conclusions reflect the current annotation scale and lack broader out-of-distribution validation.Source: §4.3 · pp. 7–8, Table 1 · p. 7

4.4 RQ3: CROSS-SYSTEM BENCHMARKING BEYOND OUTCOME METRICS

  • ProcCtrlBench generally preserves coarse stronger-versus-weaker system ordering while adding diagnostic separation when endpoint performance is similar.The comparison positions it as complementary to, rather than a replacement for, outcome-based metrics.Source: Table 2 · p. 8, §4.4 · p. 9
  • Claude Code with Claude Sonnet 4.6 has the highest overall summary score; Codex CLI is competitive in tool-use efficiency, while OpenCode variants are weaker on several process dimensions.The reported OpenCode weaknesses include workflow quality, tool-ecosystem consistency, control preservation, and fragile-success rate.Source: §4.4 · p. 9

4.5 RQ4: ROBUSTNESS ACROSS TASK SCENARIOS

  • Claude Code with Sonnet 4.6 performs strongest on AndroidBench and SWE-bench-Verified, while Codex CLI with GPT-5.4 performs strongest on TerminalBench.Strong overall systems generally remain competitive across sources, though their margins differ.Source: §4.5 · p. 9

5 LIMITATIONS AND CONCLUSION

ProcCtrlBench evaluates coding-agent execution beyond final outcomes, but its current implementation has limits in coverage, trace observability, annotation scale, and domain scope. Experiments support useful reliability and diagnostic value while motivating broader validation.

5 LIMITATIONS AND CONCLUSION

  • The 11 defect classes are a practical starting point, not an exhaustive taxonomy of coding-agent failures.Some higher-level structural defects are only partially observable from execution traces.Source: §5 · p. 9
  • The annotated set is limited relative to the breadth of behaviors agents may exhibit, and calibrated risk estimates may be sensitive to distribution shifts.The stated shifts include task styles, tool ecosystems, and workflow regimes.Source: §5 · p. 10
  • Control preservation is an operational proxy for governability, not a complete theory of human oversight.Source: §5 · p. 10
  • The study focuses on coding agents; stronger conclusions about other agent domains require additional validation.Source: §5 · p. 10
  • Across 200 cases from three benchmarks, ProcCtrlBench showed useful reliability, more interpretable calibration than direct thresholding, and diagnostic insight beyond outcome-based evaluation.The authors also conclude that coding-agent evaluation should consider how execution unfolds, not only whether tasks succeed.Source: §5 · p. 10

A DETAILED BENCHMARK DEFINITIONS

The appendix defines recurring process defects in terms of their operational boundaries, distinguishing meaningful pathologies from activity or structure that is merely long, repeated, or complex.

A.1 CONTEXT MANAGEMENT

  • Ghost Context is redundant or outdated material retained despite low downstream use, while Oversized Rules are persistent static prompt burdens rather than dynamically retained content.Necessary long-lived rules, summaries, and persistent memory are excluded from Ghost Context.Source: §A.1 · p. 13
  • Context Window Thrashing is repeated loading, saturation, and compression that discards information before a stable reasoning chain forms.Its consequence can include rediscovery and weak continuity, not simply a large context window.Source: §A.1 · p. 13

A.2 TOOL-USE EFFICIENCY

  • Duplicate Step captures highly similar tool calls without substantive state change, while Tool Call Chain captures cyclic or oscillatory sequences.Validation reruns, time-varying commands, and intended batch processing are exempt from Duplicate Step.Source: §A.2 · p. 13
  • Dead Step is an executed action with no downstream influence; Long Chain is disproportionate trajectory growth without effective decomposition, stage control, or checkpointing.Long trajectories are not inherently defective; the signal is expansion without corresponding progress consolidation.Source: §A.2 · pp. 13–14

A.3 WORKFLOW ARCHITECTURE

  • Wrapper Workflow targets pass-through structure without operational benefit, while Context Coupling concerns deep sharing that blurs unit boundaries and impairs local correction.Modularity itself is not the defect, and thin wrappers may be acceptable when they enforce policy or standardize interfaces.Source: §A.3 · p. 14

A.4 TOOL-ECOSYSTEM CONSISTENCY

  • Inconsistent Tool Interface concerns mismatched conventions among similar tools; Weak Tool concerns a technically available tool that is difficult to discover and use reliably.Weak Tool measures ecosystem legibility and accessibility rather than intrinsic tool quality alone.Source: §A.4 · p. 14

B EXPERIMENTAL SETUP DETAILS

The annotation protocol judges process defects in task context rather than treating surface repetition or failure as sufficient evidence. Reviewers record evidence and exemptions and resolve disagreements through review and adjudication.

B.1 ANNOTATION PROTOCOL

  • Reviewers with AI-coding or agent-tool experience assess 200 sampled cases across four benchmark dimensions, recording triggering evidence and exemption rationales.Disagreements are handled through dual review and adjudication.Source: §B.1 · p. 15
  • Annotators judge whether a suspicious pattern is a genuine defect under its task context, rather than labeling repetition or failure alone.For example, rerunning validation after intervening edits is not automatically a Duplicate Step.Source: §B.1 · p. 15
  • Each trajectory is reviewed as a causal process, with presence, absence, or exemption judgments supported by evidence notes that preserve local semantic context.Source: §B.1 · p. 15

B.2 DATASET AND ANNOTATION STATISTICS

The validation subset samples varied task horizons, tool environments, and defect dimensions from three benchmarks, reflecting that process defects differ across execution regimes.

B.2 DATASET AND ANNOTATION STATISTICS

  • The 200-case subset spans AndroidBench, TerminalBench, and SWE-bench-Verified, with short-, medium-, and long-horizon tasks and tool-light and tool-rich environments.Sampling also ensures representation across all four benchmark dimensions.Source: §B.2 · p. 15
  • Compact terminal tasks tend to expose local redundancy and inefficient tool use, while longer software-engineering tasks more often surface context accumulation, structural drift, and workflow coupling.The authors describe cross-benchmark sampling as reducing the risk of overfitting to one execution regime.Source: §B.2 · p. 15
  • Outcome Score may use task-success rate, test-pass rate, or a normalized outcome composite, depending on logging availability.Source: §B.2 · p. 16
  • For each defect type, annotators assess presence, absence, or exemption against process-level human judgments rather than evaluating detectors solely by final outcomes.Source: §B.2 · p. 16

B.3 HYPERPARAMETER AND IMPLEMENTATION NOTES

Detector settings are refined against annotated trajectories because appropriate thresholds vary with execution context. Exemptions are applied before posterior risk estimation to retain evidence while reducing false positives.

B.3 HYPERPARAMETER AND IMPLEMENTATION NOTES

  • Thresholds begin with heuristic rules and pilot inspection, then are refined against annotations to balance precision and recall.A threshold suitable for short terminal trajectories may be too aggressive for repository-scale debugging, where longer chains or repeated validation can be appropriate.Source: §B.3 · p. 16
  • Warning and error thresholds preserve monotonic separation among low-, medium-, and high-risk findings on the annotated set.Risk estimates use Bayesian or frequency-smoothed updates for low-dimensional count-based detectors and held-out calibrated mappings for higher-dimensional detectors.Source: §B.3 · p. 16
  • Exemptions are applied after score computation but before posterior-risk estimation, preserving raw evidence while reducing false positives for contextually appropriate behavior.Real coding-agent trajectories can contain superficially suspicious but appropriate patterns.Source: §B.3 · p. 16

B.4 ANNOTATION AGREEMENT AND DATA SPLITS

The annotated subset uses independent dual review, adjudication, and stratified development, calibration, and evaluation splits. Results show scenario-specific strengths, meaningful calibrated severity bands, and process differences not captured by endpoint outcomes alone.

B.4 ANNOTATION AGREEMENT AND DATA SPLITS

  • Two annotators label each trajectory, with third-reviewer adjudication when needed; the 200-case subset is split into 80 development, 40 calibration, and 80 held-out evaluation cases.The split is stratified by benchmark source and endpoint task success or failure; agreement is higher for locally observable defects than for structurally higher-level ones.Source: §B.4 · pp. 16–17, Table 5 · p. 17

C.1 SCENARIO-SPECIFIC SCORES

  • Claude Code with Sonnet 4.6 is strongest overall and leads on AndroidBench and SWE-bench-Verified, while Codex CLI with GPT-5 leads on TerminalBench.Strong systems generally remain competitive across benchmarks, though their advantage varies by scenario.Source: §C.1 · p. 17

C.2 CALIBRATION SEMANTICS

  • Error-level findings have the highest empirical defect frequency, warning-level findings an intermediate frequency, and non-triggered findings the lowest.The calibrated severity labels therefore provide an interpretable reporting layer over calibrated risk.Source: §C.2 · p. 17

C.3 EXEMPTION RULES AND AUXILIARY UTILITY

  • Exemption rules reduce false positives particularly for Duplicate Step and Ghost Context; findings may also support uncertainty-guided execution in high-risk trajectories.The latter is described as auxiliary downstream utility, not the study's primary focus.Source: §C.3 · p. 17

C.4 OUTCOME-BASED RANKING VS. PROCCTRLBENCH RANKING

  • ProcCtrlBench preserves the top three outcome-ranked systems but distinguishes similar-success systems by process quality and control preservation; Figure 4 visualizes rank shifts.Some competitively scoring systems also show elevated fragile-success rates.Source: Figure 4 · p. 18, §C.4 · p. 18

C.5 COMPARISON TO A SIMPLER OUTCOME-ONLY BASELINE

Compared with endpoint-only ranking, ProcCtrlBench retains broad leaderboard alignment while adding process-level distinctions. It also penalizes fragile-success and weaker workflow/control profiles among systems with competitive outcome scores.

C.5 COMPARISON TO A SIMPLER OUTCOME-ONLY BASELINE

  • ProcCtrlBench does not radically reorder the leaderboard; it separates systems with similar outcome scores but different process profiles.Its stated role is to identify execution differences endpoint metrics alone do not expose, not to replace endpoint evaluation.Source: §C.5 · p. 18
  • ProcCtrlBench preserves the top three systems under outcome-only ranking, indicating broad coarse-level alignment between endpoint performance and process quality.The comparison does not imply identical rankings across systems.Source: §C.5 · p. 18
  • OpenCode / GPT-5.4 is the clearest example of a system penalized for elevated fragile-success rates and weaker workflow/control characteristics despite competitive outcome scores.Source: §C.5 · p. 18

C.6 BOOTSTRAP STABILITY ANALYSIS

Bootstrap analyses find stable coarse distinctions but caution against overreading near ties. Further analyses show partially related defects, better-calibrated risk interpretations, sensitivity to score weighting among middle ranks, and distinct control-preservation profiles.

C.6 BOOTSTRAP STABILITY ANALYSIS

  • Coarse distinctions between stronger and weaker systems are stable under stratified bootstrap, while several middle-ranked systems remain statistically close.Small differences between near-tied systems warrant caution.Source: §C.6 · p. 19

C.7 INTER-DEFECT CORRELATION STRUCTURE

  • Long Chain and Dead Step show moderate positive association, but cross-dimension defect correlations are generally weaker, supporting a multi-defect scorecard rather than a raw defect count.The matrix covers defect-level posterior risks on the mixed evaluation set.Source: Figure 6 · p. 20, §C.7 · p. 20

C.8 CALIBRATION RELIABILITY DIAGRAMS

  • Calibrated risk curves track empirical defect frequency more closely and preserve low-, medium-, and high-risk ordering, especially for less observable dimensions.The improvement is particularly visible in Workflow Architecture and Tool-Ecosystem Consistency.Source: §C.8 · p. 20, Figure 7 · p. 21

C.10 CONTROL-PRESERVATION SUBDIMENSION BREAKDOWN

  • Control preservation is not simply the inverse of defect burden: systems can differ in interpretability and interruptibility even when their process efficiency or quality profiles differ.Table 7 reports Interpretability, Interruptibility, Correctability, Reversibility, and Authority Handoff scores across 11 systems.Source: §C.10 · p. 21, Table 7 · p. 22

C.9 SENSITIVITY TO THE WEIGHTING OF PROCESS QUALITY AND CONTROL PRESERVATION

  • The strongest systems remain competitive across a broad range of η, while some middle-ranked systems exchange relative position as process quality and control preservation weights change.The full scorecard remains the primary interpretation surface rather than the aggregate score alone.Source: §C.9 · p. 21, Figure 8 · p. 22
Loading 2605.20251v4…