PaperlayerView exact revision on arXiv

Source-linked AI summary

Terminal Agents: A Survey of AI Agents in Command-Line Environments

Yi Bin, Xiaoyang Yuan, Haoxi Zeng, Wencheng Ye, Wenqi Shao, Chen Qian, Wei Ye, Yujuan Ding, Zheng Wang, Pengpeng Zeng, Jingkuan Song, Heng Tao Shen

arXiv:2608.20485v1cs.AIcs.SE

TL;DR

Terminal-mediated behavior remains dispersed across adjacent research areas without a common synthesis of scope, systems, acquisition, and evaluation. This survey organizes the evidence through a seven-dimensional competence profile and finds that behavior and measured performance depend jointly on system configuration and benchmark conditions.

  • Problem

    Terminal-mediated behavior remains dispersed across adjacent research areas without a common synthesis of scope, system architecture, competence acquisition, and evaluation.

    Source: §1 · p. 2

  • Method

    The survey connects architecture, competence acquisition, and evaluation through a seven-dimensional terminal competence profile and bounded fixed-condition diagnostics.

    Source: §1 · p. 3

  • Results

    Terminal-agent behavior is jointly shaped by the model, interface, harness, runtime, and environment, while measured performance depends on benchmark conditions and system configuration.

    Source: §6.3 · p. 20, §8 · p. 23

  • Takeaways & Limitations

    Progress requires cross-domain acquisition and evaluation, replayable process evidence, governable runtimes, and controlled model–harness attribution.

    Source: §8 · p. 23

    Component-level attribution remains difficult because optimized harnesses can contribute gains through context management, observation shaping, retries, policies, or procedural knowledge.

    Source: §3.4 · p. 10

Abstract

from arXiv · show

Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research. We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction. Using terminal-mediated execution as an organizing lens, this survey establishes workload-level boundaries and connects system architecture, competence acquisition, and evaluation through a seven-dimensional terminal competence profile. Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment. Executable trajectories ground learning in action consequences, verification, and recovery, whereas prevailing evaluations emphasize final outcomes and expose process quality, recovery, and governance unevenly. Bounded fixed-condition diagnostics illustrate two implications: benchmark families expose different process signals, and matched system comparisons reveal benchmark-dependent performance and limits of component attribution. These findings motivate explicit reporting of system and runtime conditions, supported by replayable traces and process-level evidence. The framework provides a unified basis for studying terminal-mediated agency across software engineering and emerging application domains.

1 Introduction

This survey defines terminal agents by progress-bearing action–observation loops mediated by command execution, textual feedback, and stateful environments, treating the terminal as part of the agent system rather than a neutral interface. It uses terminal-mediated execution and a seven-dimensional competence profile to unify scope, architecture, acquisition, evaluation, and diagnostics across fragmented evidence.

Operational scope and boundaries

  • Terminal agents are systems whose dominant progress-bearing loop uses command execution, textual feedback, and stateful environment interaction, excluding surface-level CLI access and incidental command use.The loop couples reasoning, execution, observation, and verification through mutable runtimes containing files, dependencies, processes, tests, logs, and tools.Source: §1 · p. 2, Figure 1 · p. 2
  • Terminal-mediated traces make planning, memory, adaptation, and verification jointly inspectable through executable action sequences, persistent state, external feedback, and substrate-level checks.This makes terminal agents a concrete setting for studying interactive, environment-grounded intelligence.Source: §1 · p. 2

Integrated analytical framework

  • The survey addresses fragmented coverage by connecting system architecture, competence acquisition, and evaluation through a seven-dimensional terminal competence profile.The profile provides a common language for comparing system responsibilities, learning signals, and observable evidence.Source: §1 · pp. 2–3

Cross-domain evidence

  • Software engineering supplies the densest empirical foundation, while terminal-mediated interaction also appears in operations, data and scientific workflows, cybersecurity, and cloud management with fragmented evidence.The survey distinguishes established findings from emerging practices and reports its review protocol, corpus construction, and evidence-calibration procedure in supplementary material.Source: §1 · pp. 2–3

Survey organization

  • The survey organizes subsequent analysis around operational foundations, system architecture, competence acquisition, evaluation and diagnostics, and a resulting research agenda.Figure 2 summarizes this progression across Sections 2–7 before the concluding section.Source: Figure 2 · p. 3, §1 · p. 3

Cross-cutting synthesis and diagnostics

  • The synthesis shows that model, interface, harness, runtime, and environment jointly shape behavior; executable trajectories support competence acquisition, while evaluations expose process behavior unevenly.Bounded fixed-condition diagnostics further illustrate benchmark-dependent process exposure and limits of component attribution.Source: §1 · p. 3

2 Background and Scope of Terminal Agents

This section defines terminal agents by workload-level dependence on command execution, textual feedback, and stateful environments, rather than by interface surface. It introduces a seven-dimensional competence profile for analyzing terminal-mediated execution across architecture, acquisition, and evaluation.

2.1 Terminal as a Command-Execution Agent Substrate

  • Terminal execution is distinctive because compositional commands mutate persistent state and produce textual evidence such as outputs, errors, diffs, logs, and traces for iterative verification.This supports a reasoning, execution, observation, and verification loop rather than a one-way tool-call channel.Source: §2.1 · p. 4

2.2 Substrate-Based Scope and Boundaries

  • Terminal agents are scoped by workload-level dependence on command execution, textual feedback, and stateful environment interaction, regardless of whether access occurs through a CLI, IDE, web interface, or platform runtime.The scope excludes auxiliary terminal use, one-shot generation, GUI-primary workflows, and CLI surfaces that merely forward requests.Source: §2.2 · pp. 4–5
  • The operational boundary uses three workload tests: terminal execution is the primary progress substrate, iterative command feedback shapes later actions, and removing terminal access materially changes core behavior.“Dominant” permits non-command actions when task progress remains causally dependent on terminal-mediated state changes and feedback.Source: §2.2 · pp. 4–5

2.3 Terminal Competence as a Capability Profile

  • The seven-dimensional competence profile covers action formulation, feedback interpretation, runtime management, state and context tracking, progress verification, recovery and adaptation, and governance with side-effect control.These responsibilities are distributed across the model, interface, harness, runtime, and environment, and are analytically separable but operationally interdependent.Source: §2.3 · p. 6
  • The profile organizes terminal competence around trace-observable obligations, including persistent state mutation, textual evidence across steps, same-substrate verification, and authorization requirements arising from broad operational reach.It provides a common basis for comparing architecture, acquisition, and evaluation while preserving terminal-mediated execution’s distinctive demands.Source: §2.3 · p. 6

3 Terminal-Agent System Design and Architectures

Terminal-agent architectures have shifted from tool-augmented prompting toward explicit interfaces, persistent runtimes, recovery and governance, and harness-centered context management. Their trade-offs and attribution consequences arise because terminal competence is distributed across model, interface, runtime, control mechanisms, and harness.

3.1 Shifts in Design Emphasis toward Terminal-Mediated Agency

  • Terminal-agent design has shifted from external tool use to structured actions, first-class terminal interaction, and runtime- or harness-centered control of context, workspaces, recovery, and governance.These shifts include TALM [111], ReAct [170], Toolformer [123], CodeAct [149], SWE-agent [165], OpenHands [150], Terminal-Bench [101], CLI-Gym [89], and related systems.Source: §3.1 · pp. 7–8

3.2 Architectural Layers of Terminal Agents

  • Four architectural layers—interface and observation; runtime and workspace; control, verification, recovery, and governance; and harness and context—jointly shape terminal competence and long-horizon persistence.Persistence depends jointly on state and context tracking, runtime persistence, verification, and recovery.Source: §3.2 · p. 8

3.3 Recurring Architectural Patterns and Responsibility Allocation

  • Architectural patterns allocate responsibility differently: direct access favors expressiveness, mediation favors reliability, persistent runtimes broaden task surfaces, and context optimization supports coherence while complicating attribution.Architectures are therefore better compared by responsibility allocation across the model, interface, runtime, control mechanisms, and harness than by a single family.Source: §3.3 · p. 9

3.4 Synthesis: Architectural Trade-offs and Attribution Consequences

  • Raw access expands action space but weakens recoverability, platform runtimes increase generality but reduce task discipline, and approvals improve inspectability while adding latency and interrupting autonomy.These tensions concern dimensions 1, 3, 4, 6, and 7, with inspectability remaining among the least systematically addressed.Source: §3.4 · p. 10
  • Harness gains may reflect context management, observation shaping, retries, permission policy, or procedural knowledge rather than stronger reasoning, making complete model–harness–runtime configurations essential for comparison.Controlled skill injection can add substantial token overhead without improving pass rate [56], while Agentless can rival interactive agents on some repository-repair tasks.Source: §3.4 · p. 10

3.5 Evidence Landscape and Boundary Comparators

  • Boundary comparators such as Agentless, CGM [137], OSWorld [159], and CLI-packaged assistants distinguish progress-bearing terminal interaction from systems sharing only selected architectural components.The evidence landscape spans established systems, harness studies, deployment reports, training environments, and emerging operational settings.Source: §3.5 · p. 10
  • Architectural choices determine executable actions, observations, state changes, interventions, and trajectory failures, thereby shaping both learnable supervision and evaluable evidence.Section 4 examines executable interactions as learning signals, while Section 5 examines how resulting behavior becomes measurable evidence.Source: §3.5 · p. 10

4 Terminal Competence Acquisition and Adaptation

Terminal competence acquisition treats stateful executable trajectories—not isolated prompt–response pairs—as the unit connecting data collection, learning, and runtime adaptation. The section shows that environments and retained failures determine which competencies become observable, while transfer remains limited by successful repository-centric data and weak evaluation evidence.

4.1 Data Sources and Collection Settings for Terminal Competence

  • Collection settings expose different terminal competencies: short rollouts teach action and feedback, long trajectories stress state tracking, executable checks support verification, and retained failures expose diagnosis and recovery.Governance requires settings where authorization, containment, and side effects are visible.Source: §4.1 · p. 12

4.2 Trajectory Construction, Filtering, and Failure Data

  • Successful-trace filtering can remove diagnosis and recovery supervision, whereas retaining misdiagnosis, rollback, and repair provides direct learning signals for observed failure modes [33] [69] [180].Trajectory selection, validation, filtering, and relabeling shape both data quality and the behaviors preserved for learning [33] [69] [114] [155] [162] [189].Source: §4.2 · p. 12

4.3 Learning Signals and Adaptation Approaches

  • Acquisition approaches divide supervision across workflows, executable rewards, intermediate verification, synthetic coverage, failure-conditioned recovery, and runtime memory, but must avoid overfitting to tasks, environments, or harness rewards.SFT teaches common workflows, executable RL favors completion, process-aware methods supervise intermediate decisions, and runtime memory sustains long-horizon behavior; each has distinct blind spots [33] [43] [110] [114] [128] [131] [153].Source: §4.3 · pp. 12–13

4.4 Emerging Practices and Remaining Gaps

  • Current corpora remain concentrated on successful repository traces, leaving limited coverage of failed setup, environment drift, unsafe intermediate actions, and long repair loops [33] [69] [162].Repository-repair training may produce task-specific heuristics rather than general terminal competence, so evaluation must distinguish transferable skills from adaptation to particular repositories, harnesses, environments, or reward channels.Source: §4.4 · pp. 13–14

5 Benchmarks, Metrics, and Evaluation

The section organizes terminal-agent evaluation around benchmark emphases, evidence layers, competence observability, and protocol validity. It shows that evaluation increasingly exposes process and environment interaction, but remains fragmented across task domains, trace standards, freshness, safety, and governance.

5 Benchmarks, Metrics, and Evaluation

  • Terminal-agent evaluation should distinguish repository repair from broader terminal competence by separating benchmark targets, recorded evidence, observable dimensions, and protocol validity.These four axes clarify what agents are asked to do, what evidence is captured, which competencies become visible, and how scores can be interpreted.Source: §5 · p. 14

5.1 From Repository-Level Evaluation to Terminal-Native Benchmarks

  • Repository benchmarks provide executable issue-resolution tests, whereas terminal-native benchmarks directly expose command use, feedback interpretation, state inspection, and persistent workflows.SWE-bench remains a repository-repair paradigm, while terminal-native designs broaden interaction coverage; transfer across task types remains insufficiently established.Source: §5.1 · p. 15

5.2 Process-Aware, Environment-Aware, and Long-Horizon Evaluation

  • Process-, environment-, and long-horizon evaluations reveal intermediate quality, setup defects, recovery failures, and degradation under sustained multi-file or repeated editing.SWE-EVO [139] reports 65–73% on SWE-Bench Verified versus 21–25% on multi-file evolution, while newer settings assess sustained state coherence, progress tracking, and alignment over time.Source: §5.2 · p. 15

5.3 Protocol Validity and Harness Effects

  • Evaluation comparisons reflect the coupled model, harness, interface, runtime, permissions, and budget configuration rather than model capability alone.Contamination, reward hacking, harness-mediated variance, environment conditions, retries, and timeouts all affect score interpretation; an audit found 16% of tasks across five terminal-agent benchmarks reward-hackable.Source: §5.3 · p. 16

5.4 Evaluation Layers and Metrics Beyond Binary Correctness

  • Outcome metrics remain standardized anchors, but layered process, environment, trace, and governance evidence is heterogeneous and captures command economy, recovery, diagnosis, state tracking, and unsafe actions.Freshness cuts across the evidence layers as a validity condition rather than a behavioral layer.Source: §5.4 · pp. 16–17, Table 3 · p. 17

5.5 Benchmark Observability of Terminal Competence Dimensions

  • Benchmark observability is strongest for command formulation and feedback interpretation, while verification, recovery, runtime management, and governance are unevenly or indirectly exposed.SetupBench [7] explicitly measures runtime management, debugging benchmarks strengthen verification and recovery, and governance is explicit mainly in BashArena [70] and ClawSafety [152].Source: §5.5 · p. 17

5.6 Remaining Evaluation Gaps

  • Major gaps persist in domain coverage, shared process traces, freshness mechanisms, and integrated safety and governance evaluation.Current suites distribute workflow realism, setup, process quality, trace visibility, long-horizon persistence, freshness, safety, and governance across separate protocols, limiting attribution to individual components.Source: §5.6 · p. 18

6 Framework-Guided Diagnostics of Process Observability and Attribution Limits

Bounded fixed-condition diagnostics show that benchmark families expose different terminal-process signals, while matched system comparisons reveal benchmark-dependent outcome separation and limits on component attribution. Trace cases explain these aggregate differences through execution-level mechanisms and motivate combining outcomes with process evidence and explicit system conditions.

6 Framework-Guided Diagnostics of Process Observability and Attribution Limits

  • Together, the diagnostics show that realized trajectories and outcomes depend on benchmark conditions and complete system configurations, supporting evaluation with process evidence and explicit runtime reporting.The matched comparison uses identical task identifiers, and each condition represents a complete model, interface, harness, and runtime configuration.Source: §6 · pp. 18–19, §6.4 · p. 21

6.1 Diagnostic Design and Measures

  • The diagnostics use seven trace-derived indicators alongside benchmark-native outcomes, with deterministic measures for invocation failure, environment exits, final-window verification, and governance triggers, and judge-assisted rates for feedback, errors, and recovery.All formal runs fix decoding settings, task identifiers, and benchmark-native evaluators, while retaining each system’s model, interface, harness, and runtime configuration.Source: §6.1 · p. 19

6.2 Benchmark-Exposure Diagnostic

  • Under the fixed mini-SWE-agent and DeepSeek-V4-Flash configuration, benchmark families expose different process limitations: P1 is 0.0% across all four, while P2 ranges from 76.3% to 82.1% and final-window verification from 23.8%–36.6% versus 66.2% for BashArena.SetupBench foregrounds environment bootstrap, LongCLI-Bench combines low outcome with long-horizon demands, and BashArena makes state errors and governance-relevant actions more visible; LongCLI-Bench estimates are directional because of its smaller task set and lower semantic coverage.Source: §6.2 · p. 20, Table 4 · p. 20

6.3 Matched Complete-System Diagnostic

  • SWE-agent has the highest resolved rate in every benchmark–model block, but benchmark choice changes system separation: its largest gap is 21.25 points on Claw-SWE-Bench Lite versus at most 7.00 on SWE-bench Lite.OpenHands ranks above mini-SWE-agent on Claw-SWE-Bench Lite but below it on SWE-bench Lite; Pro-minus-Flash differences never exceed 2.50 points, with no significant directional model-variant difference after Holm adjustment.Source: §6.3 · pp. 20–21, Table 5 · p. 21

6.4 Illustrative Trace Cases

  • Trace cases show why aggregate outcomes differ: an unresolved magsac-install trajectory repeats dependency and build failures at a 38.5% environment-command non-zero-exit rate without closing verification or recovery.The case links runtime management, verification, and recovery demands within a Terminal-Bench 2.1 run using mini-SWE-agent and DeepSeek-V4-Flash.Source: §6.4 · p. 21
  • On rubocop__rubocop-13560, SWE-agent resolves the task while mini-SWE-agent and OpenHands do not because their differing patch scopes violate or broaden the required argument distinction.SWE-agent preserves the distinction and passes the official evaluator, whereas mini-SWE-agent changes implementation and an existing test and OpenHands exempts all method-call arguments.Source: §6.4 · pp. 21–22

7 Challenges and Future Directions

The section identifies four connected gaps—domain concentration, fragmented process evaluation, weak runtime governance, and difficult component attribution—and proposes priorities for addressing them. These priorities span cross-domain competence, fresh and replayable process evaluation, joint safety assessment, and controlled model–harness comparison.

7.1 Cross-Domain Terminal Competence

  • Cross-domain research should test whether terminal competence transfers beyond software engineering, distinguishing reusable substrate capabilities from workload-specific workflows.Operations, data engineering, scientific workflows, cloud management, and cybersecurity impose distinct runtime and governance demands [23] [68] [70] [72] [130] [144], while existing training and evaluation remain concentrated in software engineering [110] [165].Source: §7.1 · p. 22

7.2 Fresh, Replayable, Process-Level Evaluation

  • Combining live or regenerated tasks with replayable traces would connect fresh task distributions to evidence about state preservation, recovery, interventions, and verifier use.Replayable traces preserve commands, observations, state changes, interventions, and verifier calls [9] [14] [16], addressing limits of final-outcome evaluation [31] [57] [74].Source: §7.2 · p. 22

7.3 Runtime Governability and Safety

  • Evaluation should assess task success jointly with governance controls covering authorization, containment, reversibility, destructive-action prevention, and external side effects.Terminal agents can modify filesystems, install packages, launch processes, access networks, and operate around credentials; existing governance is distributed across safety and policy protocols [5] [27] [66] [70] [78] [152].Source: §7.3 · p. 22

7.4 Controlled Model and Harness Attribution

  • Factorial designs or portable protocols should vary models, interfaces, harnesses, or runtimes under fixed task conditions to separate transferable capability from configuration-specific interactions.Measured performance can reflect the model, interface, runtime, context policy, retry budget, or verifier access [79] [93] [150] [165], and harness and LM–CLI choices alter quality, efficiency, and failure behavior [25] [145].Source: §7.4 · p. 23

8 Conclusion … C Empirical Diagnostic Study: Protocol and Extended Results

The survey defines terminal agents through terminal-mediated, state-changing execution and unifies architecture, acquisition, evaluation, and governance with a seven-dimensional competence profile. Its supplementary protocol makes the synthesis traceable through scoped corpus construction, calibrated evidence, explicit system conditions, and replayable process evidence.

8 Conclusion

  • Terminal-agent behavior depends jointly on the model, interface, harness, runtime, and environment, while current evaluations expose process quality, recovery, and governance unevenly.The survey therefore calls for explicit runtime conditions, task outcomes, process evidence, trace provenance, governable runtimes, and controlled model–harness attribution.Source: §8 · p. 23

A Review Protocol and Corpus Construction; A.1 Corpus Scope and Composition; A.3 Corpus Reconciliation and Update Protocol

  • The review uses a structured narrative protocol with workload-level scope rules, entry-level coding, passage-level calibration, and work-level reconciliation across a corpus of 186 research entries.The corpus also includes six deployment-facing or boundary references outside research statistics, with analytical-use counts treated as non-exclusive.Source: A Review Protocol and Corpus… · p. 38, §A.1 · p. 38, §A.3 · p. 41

A.2 Search, Screening, and Work-Level Reconciliation; 3. Does the system produce real or simulated environment state changes through execution?

  • Corpus updates combine keyword and venue searches with backward and forward chaining, retain the most complete work version, and consolidate versions representing the same contribution.Subsequent updates record source-specific dates, query forms, and hit counts before reconciliation, while screening tests terminal execution, feedback-driven action, state change, and analytical relevance.Source: §A.2 · pp. 38–39, §3 · p. 39

A.3 Corpus Reconciliation and Update Protocol

  • Reconciliation adds works only when they satisfy screening and support an analytical role, replaces incomplete versions, and removes works that no longer support synthesis claims.These rules keep corpus composition, analytical coding, and manuscript claims synchronized.Source: §A.3 · p. 41

A.4 Coding Dimensions and Evidence Calibration

  • Coding distinguishes entry-level metadata from subset-specific architecture, acquisition, evaluation, and diagnostic fields, and calibrates claims by passage-level evidence strength rather than whole-paper maturity.Converging evidence supports stronger claims, whereas limited, single-study, indirect, deployment, or boundary evidence receives bounded or provisional treatment.Source: §A.4 · p. 41

A.5 Terminology and System-Layer Roles; 3. Does the system produce real or simulated environment state changes through execution?

  • A terminal agent is characterized by a dominant loop of command execution, textual feedback, and stateful environment interaction, with the command-execution runtime grounding actions, observations, and verification.The terminology separates terminals, shells, CLI tools, harnesses, and runtimes by their distinct roles in access, mediation, execution, and policy.Source: §A.5 · pp. 41–42

A.6 Construction and Boundaries of the Seven-Dimension Profile

  • The seven-dimensional profile covers action formulation, feedback interpretation, runtime management, state tracking, verification, recovery, and governance, distinguishing executable conditions from task-state memory and checks from adaptation.Long-horizon persistence is treated as cross-dimensional, while governance spans model behavior, harness policy, and runtime containment.Source: §A.6 · pp. 42–43

B Extended Scope and Taxonomy; C Empirical Diagnostic Study: Protocol and Extended Results

  • The taxonomy contrasts direct-command, mediated, platform, role-structured, scaffold-centric, memory, and rollout architectures, while evaluation spans outcome, process, environment, trace, governance, and freshness evidence.Terminal-native acquisition exposes feedback and recovery, repository workflows support verification, synthetic tasks risk pattern overfit, and failure traces remain sparse or inconsistently labeled.Source: B Extended Scope and Taxonomy · p. 44

C.1 Experimental Conditions and Evidence Chain · C.2 Operational Definitions of P1–P7 · C.3 Rule-Constrained LLM-as-a-Judge Configuration and Targeted Human Audit

The study fixes benchmark, system, model, runtime, and trace-processing conditions to connect official outcomes with reproducible process evidence. It operationalizes seven indicators and uses constrained semantic judging with targeted human review while keeping uncertain labels and manual adjudications separate from formal rates.

C.1 Experimental Conditions and Evidence Chain

  • Two diagnostics vary benchmark family or compare systems on aligned tasks, with each cell representing a complete model, interface, harness, and runtime configuration.The benchmark-exposure diagnostic uses 241 Terminal-Bench 2.1, 93 SetupBench, 21 LongCLI-Bench, and 640 BashArena tasks; the matched-system diagnostic uses 80 Claw-SWE-Bench Lite and 300 SWE-bench Lite tasks.Source: §C.1 · pp. 44–45
  • The evidence chain links official evaluator outcomes to complete traces, normalized events, task indicators, and benchmark summaries under fixed sampling, budget, seed, and task-ceiling conditions.Benchmark-exposure runs use 80 steps and 3,600 seconds, whereas matched-system runs use 200 steps or calls and 7,200 seconds; provider and harness limitations remain evaluated conditions.Source: §C.1 · p. 45
  • Official outcomes remain separate from trace-derived indicators, while deterministic rules define P1, P3, P5, and P7 and constrained LLM judgments define semantic P2, P4, and P6.Candidate episodes are extracted by rules, then judged in bounded evidence windows with evidence-step identifiers, evidence statements, confidence values, retries for technical failures, and missing values for ineligible semantic evidence.Source: §C.1 · p. 45

C.2 Operational Definitions of P1–P7

  • P1–P7 cover command and environment friction, feedback use, state-tracking errors, final-window verification, recovery success, and governance-review triggers using task-level numerator and denominator rules.P1, P3, and P7 are deterministic rates; P2, P4, and P6 are confidence-filtered semantic rates; P5 records verification within the final five actions or final 20% of the trace, using the larger window.Source: §C.2 · pp. 45–46
  • Reported indicator values use task-level macro-averaging where applicable, with missing evidence excluded rather than treated as zero and a 0.7 confidence threshold applied as an engineering filter.P5 is averaged over all tasks; semantic indicators require eligible, non-uncertain episodes and their specified positive conditions.Source: §C.2 · pp. 45–46

C.3 Rule-Constrained LLM-as-a-Judge Configuration and Targeted Human Audit

  • The rule-constrained judge uses target-specific prompts for P2, P4, and P6, restricting decisions to visible episode windows and requiring schema-valid JSON, visible evidence identifiers, evidence summaries, rationales, and confidence.P2 examines helpful feedback use, P4 consequential state or context errors, and P6 task-relevant recovery; context windows are target-specific and exclude hidden final outcomes unless visible in the episode.Source: §C.3 · pp. 46–47
  • Calibration trajectories and targeted human audits qualitatively checked P2, P4, and P6 label behavior, but manual adjudications remain separate from formal benchmark-level rates.Formal aggregation retains the prespecified confidence and uncertainty filters, and these indicators are treated as auxiliary process measures.Source: §C.3 · pp. 47–48

C.4 Semantic-Label Coverage for the Benchmark-Exposure Diagnostic

The diagnostic reports task eligibility alongside episode-level semantic-label coverage and uncertainty across four benchmarks. LongCLI-Bench has the lowest accepted coverage and highest uncertainty, so its auxiliary indicators are interpreted directionally, while P5 and P7 capture verification and governance-related events.

C.4 Semantic-Label Coverage for the Benchmark-Exposure Diagnostic

  • 241, 93, 21, and 640 tasks are reported for Terminal-Bench 2.1, SetupBench, LongCLI-Bench, and BashArena, respectively, with episode-level coverage of 88.0%, 86.7%, 68.7%, and 88.7%.Eligible-task counts for P2, P4, and P6 are 167/145/162, 76/50/71, 21/16/21, and 471/444/468, respectively; corresponding uncertainties are 9.6%, 11.2%, 28.2%, and 8.5%.Source: §C.4 · p. 48
  • LongCLI-Bench has the smallest task set and lowest accepted semantic-label coverage, making its auxiliary semantic indicators directional rather than definitive.Coverage and uncertainty are episode-level rates over extracted semantic candidates, whereas eligibility is task-level.Source: §C.4 · p. 48
  • P5 records the presence and timing of rule-matched late-trace verification, while P7 records command events that trigger contextual governance review.Source: §C.4 · p. 48

C.5 Matched-System Outcome and Efficiency Matrix

The matrix reports harness-specific action and time efficiency across Claw Lite and SWE-bench Lite, while mini-SWE-agent token totals quantify realized API traffic rather than intrinsic cross-system efficiency.

Efficiency values

  • Action counts and runtimes differ across systems, providers, and benchmarks, but harness-specific schemas prevent treating actions as cross-system units.The reported averages include medians and intervals for mini-SWE-agent, SWE-agent, and OpenHands under Flash and Pro.Source: §C.5 · pp. 48–49
  • Mini-SWE-agent uses 2.153/20.811 input millions/output thousands of tokens on Claw Lite with Flash, versus 2.223/18.116 with Pro.On SWE-bench Lite, the corresponding totals are 1.285/16.427 with Flash and 1.391/15.298 with Pro.Source: §C.5 · p. 49
  • These mini-SWE-agent token totals include repeated context and step count, so they characterize realized API traffic rather than intrinsic cross-system efficiency.Source: §C.5 · p. 49

C.6 Paired Model-Variant Analysis

Across six paired model-variant contrasts, no statistically significant resolved-rate difference appears under the reported conditions.

C.6 Paired Model-Variant Analysis

  • None of the six paired model-variant contrasts is statistically significant after Holm adjustment, and all paired bootstrap intervals include zero.The pairing controls task identity, while each cell remains a single formal run per task.Source: §C.6 · p. 49

C.7 Paired Cross-System Analysis

Paired analyses reject equal resolved rates among the three matched systems across all four benchmark–model blocks. Corrected pairwise differences are benchmark-specific rather than supporting a benchmark-independent system ordering.

C.7 Paired Cross-System Analysis

  • Cochran’s Q rejects equal resolved rates in all four benchmark–model blocks, with Q = 10.57–16.00 and p = 0.0051–0.0003.Table 12 localizes these omnibus differences through task-paired transitions.Source: §C.7 · p. 49, Table 12 · p. 50
  • After Holm correction, SWE-agent differs from mini-SWE-agent on Claw Lite and from OpenHands on SWE-bench Lite for both model variants.All remaining pairwise contrasts stay below the corrected significance threshold.Source: §C.7 · p. 49
  • The findings describe task-level differences within matched execution snapshots, not a benchmark-independent ranking of systems.Table 12 compares discordant task pairs using exact McNemar tests with Holm adjustment across 12 system contrasts.Source: §C.7 · p. 49, Table 12 · p. 50

D Extended Empirical Case Notes · E Extended Research Roadmap

The extended cases show that identical outcomes can arise from different terminal process paths, making outcome, trace indicators, and runtime mechanisms jointly necessary for interpretation. The roadmap translates these findings into priorities for cross-domain evaluation, fresh replayable traces, governable runtimes, and model–harness attribution.

D Extended Empirical Case Notes

  • Each case separates task outcome, trace-derived process indicators, and mechanism-level interpretation while preserving their connections to runtime state and task requirements.The common sequence first establishes what occurred, then locates the relevant episode behavior, and finally explains it mechanistically.Source: D Extended Empirical Case Notes · p. 50
  • Identical outcome scores can reflect different process paths, while similar trace signals depend on task authorization and runtime context.Reading outcomes, indicators, and mechanisms together reveals where progress stalls, how recovery proceeds, and which system layer shapes behavior.Source: D Extended Empirical Case Notes · p. 50
  • Environment-construction cases failed despite repeated setup attempts because dependency friction, unstable runtime strategies, and incomplete late-stage verification prevented convergence.The magsac-install case had P5 final-window verification rate 0.0% and P6 successful task-relevant recovery rate 41.4%; prometheus-bd5b2ea had P5 0.0% and P6 18.8%.Source: D Extended Empirical Case Notes · pp. 50–51
  • Long-horizon tasks require persistent state tracking and final verification beyond local feedback use, as shown by productive exploration that still failed to maintain unresolved-defect and test-state awareness.LongCLI-Bench cmu15_445_p0 reported P2 helpful feedback utilization 58.3%, P4 consequential state-tracking error rate 18.2%, and P5 final-window verification rate 0.0%.Source: D Extended Empirical Case Notes · p. 51
  • Mutable-runtime recovery can become self-referential: after BashArena’s destructive state change, the agent needed to repair the execution substrate required to perform the repair.The case reported P4 consequential state-tracking error rate 73.1%, P6 successful task-relevant recovery rate 5.8%, and repeated libc.so.6 loading failures.Source: D Extended Empirical Case Notes · p. 51
  • Governance-review signals are contextual rather than direct unauthorized-action rates, because authorized secret removal produced P7 governance-review-trigger rate 25.9% while passing.Terminal-Bench sanitize-git-repo passed with official score 1.0, and its broad secret search was within the task’s authorization scope.Source: D Extended Empirical Case Notes · p. 51
  • Matched-system comparison exposed configuration-dependent patch scopes and outcomes: SWE-agent resolved the task, whereas mini-SWE-agent and OpenHands remained unresolved.Mini-SWE-agent edited an existing test, OpenHands exempted all method-call arguments, and SWE-agent changed production logic only before passing the official evaluator.Source: D Extended Empirical Case Notes · p. 51

E Extended Research Roadmap

  • The research roadmap prioritizes cross-domain competence, fresh replayable evaluation, runtime governability, and controlled model–harness attribution.It calls for matched systems, versioned runtimes and trace schemas, authorization and containment reporting, and factorial or portable protocols with explicit system conditions.Source: E Extended Research Roadmap · p. 52

Related papers

Loading 2608.20485v1…