PaperlayerView exact revision on arXiv

Source-linked AI summary

Terminal Agents: A Survey of AI Agents in Command-Line Environments

Yi Bin, Xiaoyang Yuan, Haoxi Zeng, Wencheng Ye, Wenqi Shao, Chen Qian, Wei Ye, Yujuan Ding, Zheng Wang, Pengpeng Zeng, Jingkuan Song, Heng Tao Shen

arXiv:2608.20485v1cs.AIcs.SE

TL;DR

Existing research disperses terminal-mediated behavior across several agent research areas, leaving scope, competence, and evaluation without a common synthesis. This survey uses terminal-mediated execution to define workload boundaries and a seven-dimensional profile, finding that behavior depends jointly on system components and that evaluations expose process evidence unevenly. The resulting framework supports explicit, replayable reporting of system conditions and process evidence while identifying limits on transfer and component attribution.

  • Problem

    Terminal-mediated behavior is dispersed across software engineering, tool-use, computer-use, and evaluation research, leaving four connected gaps in transferable competence, process evaluation, runtime governance, and component attribution.

  • Method

    The survey organizes terminal-agent research around terminal-mediated execution, workload-level scope tests, and a seven-dimensional competence profile linking architecture, acquisition, and evaluation.

  • Results

    Terminal-agent behavior is jointly shaped by the model, interface, harness, runtime, and environment, while evaluations emphasize final outcomes and expose process quality unevenly.

  • Takeaways & Limitations

    Progress requires explicit system and runtime conditions together with task outcomes, process evidence, trace provenance, fresh replayable traces, governable runtimes, and controlled model–harness attribution.

  • Takeaways & Limitations

    Evidence remains concentrated in repository workflows, while cross-domain transfer, environment drift, failed setup, unsafe intermediate actions, and long repair loops remain weakly represented.

Abstract

from arXiv · show

Large language model agents increasingly act through terminals, yet existing surveys disperse terminal-mediated behavior across software engineering, tool use, and computer-use research. We regard terminal agents as systems whose dominant progress-bearing action--observation loop is mediated by terminal command execution, textual feedback, and stateful environment interaction. Using terminal-mediated execution as an organizing lens, this survey establishes workload-level boundaries and connects system architecture, competence acquisition, and evaluation through a seven-dimensional terminal competence profile. Our synthesis shows that realized behavior is jointly shaped by the model, interface, harness, runtime, and environment. Executable trajectories ground learning in action consequences, verification, and recovery, whereas prevailing evaluations emphasize final outcomes and expose process quality, recovery, and governance unevenly. Bounded fixed-condition diagnostics illustrate two implications: benchmark families expose different process signals, and matched system comparisons reveal benchmark-dependent performance and limits of component attribution. These findings motivate explicit reporting of system and runtime conditions, supported by replayable traces and process-level evidence. The framework provides a unified basis for studying terminal-mediated agency across software engineering and emerging application domains.

1 Introduction

Terminal agents are defined by progress-bearing command execution, textual feedback, and stateful environment interaction. The survey unifies fragmented evidence through a seven-dimensional competence profile and examines how architecture, learning, and evaluation shape observed behavior.

  • Operational scope: Terminal agents use command execution, textual feedback, and stateful environment interaction as their dominant progress-bearing loop.The terminal couples reasoning, execution, observation, and verification through mutable runtimes.
  • Motivation: Existing surveys distribute terminal-mediated behavior across general agents, software engineering, computer use, and evaluation research.This fragmentation motivates a common synthesis of scope, system responsibilities, and observable evidence.
  • Evidence base: Evidence is densest in software engineering, while terminal-mediated work in operations, data engineering, scientific workflows, cybersecurity, and cloud management remains fragmented.The survey distinguishes established findings from emerging practices.
  • Contributions: The survey provides a substrate-centered characterization and workload-level tests separating terminal execution from surface-level CLI access and incidental command use.Its organizing lens is terminal-mediated execution rather than the visible interface through which a system is accessed.
  • Contributions: A seven-dimensional terminal competence profile connects system architecture, competence acquisition, and evaluation through system responsibilities, learning signals, and observable evidence.The profile supplies a common analytical language for comparing terminal-agent systems.
  • Synthesis and diagnostics: The framework synthesizes how the model, interface, harness, runtime, and environment jointly shape terminal-agent behavior.It also examines executable trajectories, uneven process evaluation, benchmark-dependent process exposure, and limits of component attribution.

2 Background and Scope of Terminal Agents

The survey defines terminal agents by a substrate-based, execution-grounded workload boundary and analyzes them through seven interdependent competence dimensions. Terminal interaction is treated as a stateful loop in which commands produce textual evidence that shapes later actions and verification.

  • Terminal substrate: Terminal-mediated execution combines compact textual interaction, compositional commands, persistent state, and broad operational reach.Outputs include stdout, stderr, exit codes, diffs, stack traces, logs, and process outputs.
  • Substrate-based scope: Terminal-mediated interaction can occur without a visible terminal emulator or persistent pseudo-terminal when command execution and textual state evidence carry task progress.CLI-exposed products remain outside scope when the CLI merely forwards requests.
  • Substrate-based scope: A terminal agent is in scope when command execution drives workload progress, feedback shapes later actions, and removing terminal access materially changes core behavior.The tests apply to workloads rather than automatically to entire products or platforms.
  • Boundary comparators: GUI environments remain boundary comparators because they include state and executable verification while visual interaction remains the progress-bearing substrate.This distinguishes terminal-mediated agency from systems whose primary feedback is visual or DOM-based.
  • Competence profile: The seven competence dimensions cover action formulation, feedback interpretation, runtime management, state and context tracking, verification, recovery and adaptation, and governance.They separate runtime construction from state tracking, verification from recovery, and authorization control from ordinary command execution.
  • Competence profile: These dimensions are analytically separable but operationally interdependent across model, interface, harness, runtime, and environment responsibilities.Recovery depends on feedback interpretation, state tracking, and verification, while long-horizon persistence combines state tracking, verification, and recovery.

3 Terminal-Agent System Design and Architectures

Terminal-agent architectures increasingly make interfaces, workspaces, recovery, governance, and context management explicit design surfaces. Their behavior is distributed across model, interface, runtime, control mechanisms, and harness, creating trade-offs in expressiveness, reliability, generality, inspectability, and attribution.

  • Architectural shifts: Terminal-agent design shifts from tool-augmented prompting toward structured executable actions, first-class terminal agency, and runtime- or harness-centered systems.These shifts increasingly treat action interfaces, mutable workspaces, recovery, governance, and context management as explicit design surfaces.
  • Architectural layers: Four architectural layers organize terminal-agent responsibilities: interface and observation; runtime and workspace; control, verification, recovery, and governance; and harness and context.The layers locate design choices within the terminal-mediated loop and connect them to competence dimensions.
  • Responsibility allocation: Terminal competence is distributed across the model, interface, runtime, control mechanisms, and harness, so long-horizon persistence depends on joint state and context tracking, verification, and recovery.Persistent runtimes enable cumulative progress but can propagate erroneous intermediate state, while context optimization supports coherence but complicates attribution.
  • Trade-offs: Raw or weakly mediated access expands action space but increases trajectory noise and rollback difficulty, whereas interface mediation improves reliability while potentially restricting cross-task generality.This expressiveness–recoverability trade-off primarily concerns command action and recovery, with verification supplying evidence for recovery.
  • Trade-offs: Permission gates, approvals, and sandboxes improve auditability but add latency and interrupt autonomous execution, leaving governance among the least systematically addressed dimensions.This automation–inspectability tension centers on governance and execution oversight.
  • Attribution: Optimized harness gains may reflect context management, observation shaping, retries, permission policy, or injected procedural knowledge rather than stronger model reasoning.The complete model–harness–runtime configuration matters for system comparison, while component attribution requires separating these contributions.
  • System consequences: Architectural choices determine executable actions, observations, state changes, interventions, and failures, thereby shaping both acquisition supervision and evaluation evidence.Executable interactions become learning signals, while resulting behavior becomes measurable evidence through evaluation protocols.

4 Terminal Competence Acquisition and Adaptation

Terminal competence acquisition treats executable interaction as stateful trajectories that combine actions, observations, state changes, verification, and recovery. Data construction, parameter optimization, and runtime adaptation provide complementary signals, but coverage remains limited for failures, long repair loops, environment drift, and transfer beyond repository-centric tasks.

  • Acquisition framework: Competence acquisition spans parameter learning and runtime adaptation, using stateful trajectories rather than isolated prompt–response pairs as the relevant learning unit.These trajectories include actions, observations, state changes, verification, and recovery decisions.
  • Collection environments: Collection environments determine which actions, observations, state changes, and failures become available as learning evidence across competence dimensions.Short rollouts expose action and feedback interpretation; setup, long-horizon, executable-check, failure, and governance settings expose additional dimensions.
  • Trajectory construction: Trajectory selection, validation, filtering, and relabeling shape the data quality and behaviors preserved for learning.Dockerized generation, CLEANER, and AgentHER exemplify generation, filtering, validation, and hindsight relabeling operations.
  • Recovery: Recoverable failures remain underrepresented because successful-trace filtering can remove interactions needed to learn diagnosis, adaptation, rollback, and repair.Retaining misdiagnosis, rollback, and repair provides direct supervision for recovery under observed failure modes.
  • Learning signals: Successful traces teach workflows, executable rewards favor completion, verifiers and hindsight expose intermediate decisions, synthetic tasks broaden coverage, failure-conditioned data supports recovery, and runtime memory sustains long-horizon behavior.These signals are complementary, but combining them risks overfitting to benchmark tasks, generated environments, or harness-specific rewards.
  • Coverage gaps: Current acquisition pipelines remain concentrated on successful repository-centric traces, with limited coverage of failed setup, long repair loops, environment drift, and unsafe interventions.Sourcing, rollout generation, filtering, replay, and curriculum design are increasingly treated as distinct pipeline levers.
  • Transfer: Transfer remains unresolved because repository-repair training may produce task-specific heuristics rather than general terminal competence.Reusable transfer depends on preserving action, observation, reward, intermediate reasoning, and execution evidence across environments.
  • Evaluation implications: Evaluation evidence is needed to distinguish transferable terminal competence from adaptation to particular repositories, harnesses, environments, or reward channels.Final success alone does not reveal state preservation, failure recovery, completion verification, or adherence to execution constraints.

5 Benchmarks, Metrics, and Evaluation

The survey organizes terminal-agent evaluation around benchmark emphasis, evidence layers, observability, and protocol validity. It finds broad command and feedback observability but persistent gaps in domain coverage, process standards, governance integration, and component attribution.

  • Evaluation framework: Evaluation distinguishes task demands, recorded evidence, observable competence dimensions, and the conclusions supported by protocol design.These four analytical axes separate what benchmarks ask agents to do from what they record and how scores should be interpreted.
  • Benchmark families: Terminal-native benchmarks directly expose command use, output interpretation, state inspection, and workflow persistence.CLI-centered designs make command-mediated interaction part of the task rather than treating the terminal as a passive backend.
  • Process evaluation: Process-aware evaluations extend beyond final completion to assess intermediate quality, error propagation, control preservation, state coherence, and sustained alignment.Representative benchmarks add step-level assessment, defect ontologies, recovery-related analysis, and long-horizon state tracking.
  • Protocol validity: 16% of tasks across five terminal-agent benchmarks were reported as reward-hackable, making freshness, contamination control, regeneration, and anti-hacking central to validity.Static pools are vulnerable to memorization, leakage, stale distributions, and public exposure of task materials.
  • Protocol validity: Benchmark results characterize a coupled configuration including the model, harness, runtime, sandbox, interfaces, execution budget, and trace-release policy.Harness settings such as observation formatting, approvals, retries, context truncation, and recovery affordances can alter performance under a fixed model.
  • Remaining gaps: Four gaps remain: narrow terminal-native workflow coverage, immature trace standards, separated governance evidence, and limited joint integration of evaluation requirements.Most benchmarks remain repository- or coding-centric, while no common trace schema covers commands, observations, failures, retries, state changes, and interventions.

6 Framework-Guided Diagnostics of Process Observability and Attribution Limits

Bounded fixed-condition diagnostics test which process signals benchmark families expose and how matched systems differ on identical tasks. The results show benchmark-dependent process visibility and system separation, while model variants remain similar under the reported conditions.

  • Diagnostic design: The diagnostics fix agent, model, task identifiers, decoding, and evaluators while varying benchmark families or complete systems.The benchmark-exposure study uses mini-SWE-agent with DeepSeek-V4-Flash; the matched comparison evaluates three systems on identical tasks.
  • Measures: Seven trace-derived indicators capture selected competence-related signals, combining deterministic rates with bounded rule-constrained LLM judgments.P1, P3, P5, and P7 are deterministic; P2, P4, and P6 are judge-assisted task-level macro averages over eligible evidence.
  • Benchmark exposure: 0.0% is the P1 invocation-failure rate across all four benchmarks, indicating that the narrow rule-matched signal captures few command failures.Broader action-formulation problems appear through semantic or state-dependent behavior outside this signal.
  • Benchmark exposure: P2 ranges from 76.3% to 82.1%, while final-window verification ranges from 23.8% to 36.6% on three benchmarks versus 66.2% for BashArena.Local feedback use and end-to-end completion therefore expose different aspects of terminal behavior.
  • Implications: The diagnostics show that benchmark conditions change visible process demands, while system configuration changes trajectories and outcomes on matched tasks.This supports combining task outcomes with process evidence and explicit system conditions.
  • Matched systems: SWE-agent has the highest resolved rate in all four benchmark–model blocks, but benchmark choice changes the separation and can reverse OpenHands versus mini-SWE-agent ordering.The largest gap is 21.25 percentage points on Claw-SWE-Bench Lite and at most 7.00 points on SWE-bench Lite.
  • Matched systems: The absolute Pro-minus-Flash difference never exceeds 2.50 points, with no adjusted directional model-variant difference detected.Pro has higher observed mean wall-clock time in five of six system–benchmark pairs and uses more mean input tokens under mini-SWE-agent without an outcome gain.

7 Challenges and Future Directions

The survey identifies domain concentration, fragmented process evaluation, weak governance integration, and difficult component attribution as connected challenges. It points toward fresh replayable traces, joint task-and-governance evaluation, and cross-domain comparisons of terminal competence.

  • Cross-domain competence: Evidence remains concentrated in software engineering, limiting conclusions about transferable terminal competence across operational and scientific domains.Repositories provide executable verification but cannot determine whether behavior reflects reusable interaction competence or task-specific heuristics.
  • Process-level evaluation: Final outcomes do not reveal diagnostic quality, state preservation, recovery, or unsafe intermediate behavior.Live or regenerated tasks combined with replayable traces could connect fresh distributions to process evidence and acquisition-relevant trajectories.
  • Safety and governance: Governance remains distributed across separate safety and policy protocols even though terminal agents can alter filesystems, processes, networks, and credentials.The survey calls for joint assessment of task success and authorization, containment, reversibility, destructive-action prevention, and external side effects.
  • Component attribution: Observed performance can reflect the model, interface, runtime, context policy, retry budget, or verifier access, complicating separation of model capability from system support.The matched diagnostic further shows that observed system differences can vary across benchmarks.

8 Conclusion

The survey organizes terminal-agent research around terminal-mediated execution and a seven-dimensional competence profile. It concludes that behavior depends jointly on system components and that evaluation should expose process, runtime, trace, and governance conditions.

  • The survey uses terminal-mediated execution and a seven-dimensional competence profile to connect architecture, acquisition, and evaluation.
  • Terminal-agent behavior is jointly shaped by the model, interface, harness, runtime, and environment.
  • Executable trajectories ground learning in feedback, verification, and recovery, but acquisition underrepresents environment management, persistent state, recovery, and governance.
  • Prevailing evaluations emphasize final outcomes while exposing process quality unevenly, and measured performance depends on benchmark conditions and system configuration.
  • The review uses workload-level scope rules, coded evidence, corpus reconciliation, and passage-level calibration to support a traceable synthesis.
  • A terminal agent is characterized by a dominant progress-bearing loop involving command execution, textual feedback, and stateful environment interaction.
  • Evaluation evidence spans outcome, process, environment, trace, governance, and freshness layers, each exposing different aspects of terminal competence.

C.1 Experimental Conditions and Evidence Chain

The diagnostics fix system, model, benchmark, and execution conditions while linking evaluator outcomes to complete traces and derived process indicators. Seven indicators combine deterministic rules with bounded semantic judgments, with missing evidence excluded from defined-denominator averages.

  • The benchmark-exposure diagnostic fixes mini-SWE-agent with DeepSeek-V4-Flash while varying benchmark families; the matched-system diagnostic compares three systems on identical task identifiers.
  • Each formal task links the official evaluator outcome to its complete execution trace, normalized events, task indicators, and benchmark summary.
  • Benchmark values are macro-averages over tasks with defined semantic denominators, and missing evidence is not assigned a zero.
  • The seven indicators measure invocation failure, feedback use, environment-command exits, state-tracking error, final-window verification, recovery, and governance-review triggers.
  • P5 counts tasks with a rule-matched verification action in the final five actions or final 20% of the trace, using the larger window.
  • P2, P4, and P6 use bounded, target-specific semantic judgments over visible episode windows, with the official outcome withheld unless visible in the episode.

C.4 Semantic-Label Coverage for the Benchmark-Exposure Diagnostic

The benchmark-exposure diagnostic reports task eligibility, episode coverage, and uncertainty for semantic indicators across four benchmark families.

  • Terminal-Bench 2.1 contains 241 tasks, with 167, 145, and 162 eligible for P2, P4, and P6; coverage is 88.0% and uncertainty 9.6%.

C.5 Matched-System Outcome and Efficiency Matrix

The matched-system matrix reports task-paired outcomes, efficiency, and token use under fixed benchmark and model conditions. Efficiency varies across systems, models, benchmarks, and harness-specific action schemas.

  • Claw Lite efficiency differs across systems and models, with mini-SWE-agent, SWE-agent, and OpenHands reporting distinct average action counts and times.
  • The outcome table defines Pro-only and Flash-only task transitions and applies exact McNemar tests, Holm adjustment, and paired bootstrap intervals.
  • SWE-bench Lite efficiency likewise varies across mini-SWE-agent, SWE-agent, and OpenHands under Flash and Pro configurations.
  • Mini-SWE-agent token use ranges from 1.285/16.427 to 2.223/18.116 input millions/output thousands across the reported benchmark-model cells.

C.6 Paired Model-Variant Analysis

Across the six paired model-variant contrasts, no statistically significant resolved-rate difference was detected under the reported conditions.

  • None of the six paired model-variant contrasts is statistically significant after Holm adjustment.All paired bootstrap intervals include zero.
  • The evaluated model variants show no directional resolved-rate difference across the system–benchmark pairs.Task pairing controls task identity, while each cell remains a single formal run per task.

C.7 Paired Cross-System Analysis

Matched-system comparisons reject equal resolved rates across all four benchmark–model blocks, but corrected pairwise differences are benchmark-specific rather than forming a benchmark-independent system ordering.

  • Four benchmark–model blocks reject equal resolved rates among the three matched systems.Cochran’s Q values range from 10.57 to 16.00, with p-values from 0.0051 to 0.0003.
  • After Holm correction, SWE-agent differs from mini-SWE-agent on Claw Lite for both model variants.
  • After Holm correction, SWE-agent differs from OpenHands on SWE-bench Lite for both model variants.
  • The remaining pairwise contrasts do not cross the corrected threshold, so the findings do not establish a benchmark-independent system ordering.Table 12 reports task-paired contrasts as A minus B in percentage points, using exact McNemar tests with Holm adjustment across all 12 system contrasts.

D Extended Empirical Case Notes

The extended cases connect task outcomes with process indicators and mechanisms, showing that terminal-agent behavior depends on environment management, state tracking, verification, recovery, governance context, and system-specific patch scope.

  • Case-reading method: The cases use task conditions and evaluator outcomes, process indicators, and trace evidence to connect observed episodes with mechanism-level interpretations.This sequence separates outcome evidence, trace-derived signals, and qualitative interpretation while preserving their connections.
  • Cross-case synthesis: Across cases, identical outcome scores can arise from different process paths, while similar trace signals depend on task authorization and runtime context.Reading outcomes, indicators, and mechanisms together clarifies where progress stalls, how recovery proceeds, and which system layer shapes observed behavior.
  • Environment construction: Failed environment-management cases show that setup actions require a stable runtime strategy and evidence that configuration changes alter build state.The magsac-install case had official score 0.0, with a 38.5% environment-command non-zero-exit rate and 0.0% final-window verification rate; the dependency-resolution case had corresponding rates of 41.9% and 0.0%.
  • State tracking and closure: Long-horizon terminal tasks require state tracking and final verification in addition to local feedback use.In the LongCLI-Bench case, the official score was 0.0, with an 18.2% consequential state-tracking error rate and 0.0% final-window verification rate.
  • Execution-substrate damage: A destructive state change can damage the execution substrate needed to perform its own repair.The BashArena case reported a 73.1% consequential state-tracking error rate, 0.0% final-window verification, and 5.8% successful task-relevant recovery.
  • Governance signals: Governance-review triggers are contextual rather than direct unauthorized-action rates when task authorization permits the triggering behavior.In sanitize-git-repo, broad secret search and replacement produced a 25.9% P7 trigger rate while the task passed with official score 1.0.
  • System-dependent patch scope: Matched task traces can produce divergent outcomes and patch scopes across systems under the same model condition.For rubocop__rubocop-13560 with DeepSeek-V4-Flash, mini-SWE-agent and OpenHands were unresolved while SWE-agent resolved the task; their trace behaviors also differed in test editing, argument exemptions, and production-logic changes.
  • Research priorities: The roadmap prioritizes cross-domain competence, fresh replayable evaluation, runtime governability, and controlled model–harness attribution.It calls for shared task and trace formats, versioned runtimes, recoverable sandboxes, and portable harness descriptions.
Loading 2608.20485v1…