PaperlayerView exact revision on arXiv

Source-linked AI summary

RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

Yunqi Liu, Peiyu Wang, Bingli Wang, Leo Luo, Xuanwu Song, Xudong Ma, Hongxin Li, Yaxin Du, Weicheng Gu, Fanglin Xu, Yi Xing, XiangLong Liu, Jujie He, Zhange Zhang, Huatong Song, Zhaoyi Li, Deyang Kong, Chuan Hao, Peihao Wu, Mingjie Tang, Ran Tao, Jian Yang, Wayne Xin Zhao, Bryan Dai

arXiv:2610.22507v1cs.CL

TL;DR

Existing CLI benchmarks largely assume fixed user instructions or assess outcomes without measuring runtime collaboration. REALCLI pairs autonomous and scripted-user evaluation on the same tasks, using a five-dimensional framework. Results show that strong task outcomes do not imply strong collaboration, and that collaboration weaknesses persist across frontier agents.

  • Problem

    Existing CLI benchmarks provide limited evaluation of runtime collaboration even though users may add information, revise constraints, and authorize actions during long-horizon tasks.

    Source: §1 · pp. 1–2

  • Method

    REALCLI pairs autonomous execution and scripted-user collaboration on the same CLI tasks, measuring collaboration across five dimensions operationalized into 22 sub-dimensions and 51 scenario templates.

    Source: §1 · p. 2

  • Results

    Strong task outcomes do not imply strong collaboration, and agents show imbalanced capabilities with persistent weaknesses in steerability, trust calibration, and safety.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Outcome-only evaluation conceals systematic collaboration failures, so collaboration should be treated as a first-class, measurable capability in advancing CLI agents.

    Source: §5 · p. 10

Abstract

from arXiv · show

Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks. To comprehensively measure collaboration, REALCLI builds a five-dimensional framework, operationalized into 22 sub-dimensions and 51 interaction scenario templates. REALCLI provides two modes for paired comparison: REALCLI-EXEC supplies complete specifications upfront, whereas REALCLI-COLLAB introduces a scripted user that delivers progressive information, corrections, constraint revisions, and authorization decisions. Evaluations reveal profound capability gaps in frontier agents: the best configuration attains an average score of only 49.2%, and when evaluated by the joint collaboration score ($J_{\mathrm{collab}}$), the peak performance reaches merely 37.6%. These results show that task effectiveness diverges sharply from collaboration quality, with models consistently struggling in steerability, trust calibration, and safety boundary management. REALCLI provides a rigorous, reproducible foundation for measuring and advancing collaborative CLI agents.

1 INTRODUCTION

Existing CLI benchmarks largely use fixed specifications and final-state evaluation, leaving runtime collaboration undermeasured. REALCLI pairs task execution with collaboration assessment and finds substantial gaps in frontier agents’ collaboration quality.

1 INTRODUCTION

  • Existing benchmarks typically give agents fixed specifications and evaluate final code or environment states, while real workflows require ongoing coordination as users add information or revise constraints.Prior work offers limited evaluation of execution processes and whether agents verify state, recover, disclose risks, and preserve safety boundaries.Source: §1 · pp. 1–2
  • The paper frames two questions: what outcomes and collaboration quality agents achieve with user participation, and how behavior and failures differ from autonomous execution on the same tasks.The paired setup supports direct comparison of task outcomes and collaboration quality under matched tasks and environments.Source: §1 · pp. 2–3
  • REALCLI defines five collaboration dimensions and operationalizes them into 22 sub-dimensions, 51 scenario templates, and six trajectory-level checks.Scenarios specify the behavior evaluated and the evidence required, making collaboration quality observable and auditable.Source: §1 · p. 2
  • REALCLI pairs 101 realistic CLI tasks across 10 categories and 52 subdomains in EXEC and COLLAB modes, sharing task goals, environments, and verifiers.EXEC provides complete specifications upfront; COLLAB uses a scripted user for progressive information and interaction probes, and scores collaboration from traces and environment evidence.Source: §1 · pp. 2–3
  • 52.7% is the best configuration’s average score; task effectiveness diverges sharply from collaboration quality, and user interaction does not reliably improve pass rates.Agents perform relatively well in understandability and observation and recovery, but remain weak in steerability, trust calibration, and safety and risk awareness.Source: §1 · p. 3

2 REALCLI BENCHMARK

REALCLI pairs autonomous and collaborative execution on the same CLI tasks, with shared goals and acceptance criteria. Its scripted user and layered scoring evaluate both task outcomes and collaboration quality.

2 REALCLI BENCHMARK

  • REALCLI pairs EXEC and COLLAB on the same CLI tasks, goals, environments, and acceptance criteria, enabling comparison of autonomous and collaborative execution.Source: §2 · p. 3, §2.1 · p. 4

2.1 TASK FORMULATION

  • A task is evaluated in EXEC with its complete specification upfront, while COLLAB progressively supplies information and permits clarification or authorization requests.Source: §2.1 · p. 4

2.2 FIVE-DIMENSIONAL COLLABORATION FRAMEWORK

  • The collaboration framework organizes behavior into five dimensions, refined by subdimensions and grounded in scenario templates for concrete CLI situations.Examples include adapting to a revised requirement and recognizing a potential sensitive-file leak.Source: §2.2 · p. 4, Table 2 · p. 4

2.3 SCRIPT-DRIVEN USER SIMULATION

  • A task-level script controls information release and user participation through information slices and interaction probes tied to execution state.Intervention timing varies with the execution trajectory while task semantics, key messages, and evaluation criteria remain fixed.Source: §2.3 · p. 4

2.4 EVALUATION METRICS

  • Task success checks whether the terminal state satisfies the task goal and acceptance criteria, with average task scores reported separately for EXEC and COLLAB.Source: §2.4 · p. 5
  • Collaboration quality aggregates scenario checks through four levels, applying a 75% cross-task pass threshold and equal-weight averages up to five dimensions.The Joint Collaboration Score multiplies collaborative task success by the composite collaboration score, so task success cannot compensate for collaboration deficiencies.Source: §2.4 · p. 5

3 BENCHMARK CONSTRUCTION

REALCLI is built through sequential synthesis pipelines that turn real CLI logs into validated base tasks, then add scripted collaboration scenarios. The resulting benchmark covers 101 tasks, 10 categories, 52 subdomains, 51 scenario templates, and 1,658 checklist items.

Base Instance Synthesis

  • REALCLI derives reusable task patterns from real CLI logs, then validates synthesized instructions, environments, reference solutions, and verifiers.Quality checks confirm specification–test alignment, reference-solution success, and no-op baseline failure; rollouts diagnose task versus infrastructure issues and profile difficulty.Source: §3 · p. 6

Interactive Instance Synthesis

  • Each validated base task receives six collaboration scenarios and a script with an opening message, progressive information slices, probes, and checklist items.Static and semantic checks validate scripts; failed scripts return for revision.Source: §3 · p. 6
  • End-to-end pilots check whether intended collaboration behaviors are elicited and measurable before human expert review and difficulty filtering.Curated instances retain base artifacts, append files needed for interaction, and enter the delivery pool.Source: §3 · p. 6

Benchmark Statistics

  • REALCLI includes 101 tasks across 10 categories and 52 subdomains, with 51 collaboration scenario templates and 1,658 checklist items.The construction process has two sequential pipelines: base-instance synthesis from real CLI logs, followed by interactive-instance synthesis from validated base instances.Source: §3 · pp. 5–6

4 EXPERIMENTS

Across the experiments, frontier agents show a gap between task execution and collaboration quality, while user interaction produces scaffold- and task-dependent outcomes. Collaboration failures persist even when tasks succeed, and common execution failures remain tied to functional or boundary-case errors.

4.1 EXPERIMENTAL SETUP

  • Evaluation uses all 101 tasks in isolated containers, caps runs at 600 turns, and reports avg@3 scores.The execution and collaboration tracks receive resource budgets of six and ten hours, respectively.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • 49.2% is Claude Opus 5’s highest average score, while GPT-5.6 Sol’s Jcollab falls to 8.5% from an Rexec of 58.5% on Codex.The joint score decreases by 50.0%, illustrating that strong autonomous execution can coexist with weak collaboration quality.Source: §4.2 · pp. 7–8
  • Collaboration scores are imbalanced: models perform better on Understandability and Observation & Recovery than on Steerability, Trust Calibration, and Safety Boundary Management.GPT-5.6 Sol has 7% in Trust Calibration and 0% in Safety Boundary Management despite a 47.5% Rcollab on Terminus-2.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 IMPACT OF USER INTERACTION ON TASK OUTCOMES

  • 46.8% versus 47.4% are the mean pass rates for interactive and autonomous execution across 11 Claude Code configurations and 101 matched tasks.The near parity masks bidirectional changes: 17.9% of runs turn failure into success, while 18.5% turn success into failure.Source: §4.3 · p. 8
  • Interaction gains are more reliable under Codex, with a mean gain of 3.4 percentage points, and concentrate in tasks with local, verifiable requirement additions.Regressions cluster where models mistake incoming instructions for replacements of existing constraints; Kimi K3 also gains 8.9 percentage points on a state-consistency task where HY3 does not maintain global state across reruns.Source: §4.3 · pp. 8–9

4.4 MULTIDIMENSIONAL COLLABORATION CAPABILITIES IN INTERACTIVE TASKS

  • 80.2% of scenarios involving plan updates after obstacles pass, compared with 51.7% for recognizing scope expansion.Models pass 98.8% of successive-requirement-change scenarios, versus 54.4% for retaining earlier constraints and 29.3% for handling conflicting instructions across turns.Source: §4.4 · p. 9, Figure 5 · p. 9

4.5 FAILURE MODES IN TASK EXECUTION AND USER COLLABORATION

  • 62.7% of autonomous failures and 53.8% of interactive failures are functional or boundary-case errors, the most common type in both modes.User participation changes execution paths but does not replace verification of boundary conditions, interfaces, and final behavior.Source: §4.5 · p. 10, Figure 6 · p. 10
  • 91.0% of successful interactive trajectories still fail at least one applicable collaboration check.Constraint-retention failures include artifact mismatches in 91.3% of cases, while 53.3% of those mismatches with applicable communication checks pass every communication check.Source: §4.5 · p. 10

5 CONCLUSION

REALCLI pairs task execution and human-agent collaboration evaluation on realistic CLI tasks, and experiments across 33 configurations show that task success does not ensure strong collaboration. Collaboration remains imbalanced, supporting its treatment as a measurable capability.

5 CONCLUSION

  • Across 33 configurations, strong task outcomes do not imply strong collaboration, and agents with comparable task success can collaborate differently.The authors conclude that outcome-only evaluation conceals systematic collaboration failures.Source: §5 · p. 10
  • REALCLI evaluates task execution and human-agent collaboration on the same realistic CLI tasks.Source: §5 · p. 10
  • The authors argue that advancing CLI agents requires treating collaboration as a first-class, measurable capability.Source: §5 · p. 10

AI USE STATEMENT

The authors disclose that generative AI was used only for literature reading and language editing of author-written text. They reviewed AI-assisted content and take responsibility for the final manuscript.

AI USE STATEMENT

  • Generative AI was used solely for literature reading and language editing of author-written text.Source: AI USE STATEMENT · p. 10
  • The authors say they did not use generative AI for ideation, experimental design, analysis, results, citations, surveys, interviews, or figures.Source: AI USE STATEMENT · p. 10
  • The authors reviewed all AI-assisted content and take full responsibility for the final manuscript.Source: AI USE STATEMENT · p. 10

A.1 LONG-HORIZON CLI AGENT EVALUATION

REALCLI positions collaboration as an independent target in long-horizon CLI evaluation, extending paired task evaluation with structured scenarios and evidence-based checks. Its framework spans five dimensions and 22 subdimensions, with scenarios covering both prompted responses and ongoing practices.

A.1 LONG-HORIZON CLI AGENT EVALUATION

  • REALCLI-EXEC evaluates autonomous execution with complete specifications, while REALCLI-COLLAB introduces task information, constraints, and risk decisions progressively.Source: §A.1 · p. 14

A.2 USER-INTERACTIVE CLI AGENT EVALUATION

  • Unlike benchmarks centered on task outcomes, REALCLI treats collaboration itself as an independent evaluation target across five dimensions.The dimensions are comprehensibility, steerability, observation and recovery, trust calibration, and safety boundary management.Source: §A.2 · p. 14

B DETAILS OF COLLABORATION FRAMEWORK

B.1 TAXONOMY OF COLLABORATION DIMENSIONS

  • Scenarios use both active probes that introduce stimuli or obstacles and passive checks that monitor behavior throughout a task.Source: §B.1 · p. 14

1.1–1.4 Collaboration behaviors

  • Scenario examples test plan adaptation, feedback and explanation, and economical CLI operation, including handling hidden complexity, errors, changing preferences, and large outputs.Source: §1.1 · p. 15, §1.2 · p. 15, §1.3 · p. 16

2.1–2.5 Steerability

3.1–3.3 Observation and recovery

4.1–5.5 Trust and safety

B.3 EVIDENCE-BASED COLLABORATION EVALUATION

REALCLI scores collaboration using checklist evidence from interaction traces and environment states. Environment observations take precedence over conflicting dialogue claims, and scenarios that do not arise are excluded from the denominator.

B.3 EVIDENCE-BASED COLLABORATION EVALUATION

  • Across 101 tasks, 1,658 checklist items include 1,103 based on interaction traces, 413 on environment evidence, and 142 requiring both.Thus, 555 items (33.5%) incorporate deterministic environment evidence.Source: §B.3 · p. 18
  • A checklist item passes only with concrete positive evidence, and observed environment state overrides a conflicting agent statement.Source: §B.3 · p. 18
  • Scenario checklist items are marked not applicable and excluded from scoring when their trigger does not occur in a run.This limits scoring to actions the agent had an opportunity to take.Source: §B.3 · p. 19

C.1 DETAILS OF TASK TAXONOMY

REALCLI spans 52 subdomains across 10 CLI categories and builds each interactive task around a scripted user interaction layer. The experimental setup uses three CLI agent scaffolds, 11 models, and a persistent simulator governed by task scripts.

C.1 DETAILS OF TASK TAXONOMY

  • REALCLI’s taxonomy covers 52 subdomains across 10 top-level categories, with representative task scopes and task counts reported in Table 7.The categories include office-suite tasks, interactive experiences, and models and algorithms.Source: §C.1 · p. 19, Table 7 · p. 19

C.2 DETAILS OF BENCHMARK INSTANCES

  • Each benchmark instance is a self-contained directory that preserves original task files and confines interaction-layer files to one subdirectory.The task record specifies metadata, the initial environment, instructions, constraints, the verifier interface, and evaluation budgets.Source: §C.2 · p. 20
  • An average interaction script withholds about 16 information slices and carries six probes, alongside an opening and closing message.Probes include goals, stimuli, expected behavior, evidence sources, hard checks, and binary checklist items.Source: §C.2 · p. 20

D EXPERIMENTAL SETUP DETAILS

  • The evaluation setup uses three mainstream CLI agent scaffolds, 11 models from eight providers, and limits runs to 600 agent turns.Runs have six-hour REALCLI-EXEC or ten-hour REALCLI-COLLAB time limits; runs reaching a budget cutoff are scored from the terminal state.Source: §D.1 · p. 21, §D.2 · p. 21

D.1 DETAILS OF AGENT HARNESS

  • The three scaffolds differ in vendor coupling and interaction granularity; Terminus-22 is model-agnostic and drives the terminal session directly.Claude Code and Codex use tool-use loops; Codex couples its loop with sandboxed command execution.Source: §D.1 · p. 21

D.3 DETAILS OF USER SIMULATOR

  • A persistent scripted user releases task information and probes based on the conversation while preserving the task goal and avoiding unsupported state claims.Probe stimuli are delivered verbatim, while information slices may be paraphrased but must remain complete; an append-only ledger records interaction events.Source: §D.3 · pp. 22–23

D.4 DETAILS OF EVALUATION JUDGE

The evaluation judge scores agent collaboration and user-simulator fidelity separately using scripts, transcripts, and environment evidence. Checklist verdicts follow explicit evidence and timing rules, and simulator-caused evaluation failures are flagged as contamination.

D.4 DETAILS OF EVALUATION JUDGE

  • The judge separately scores agent interaction checklists and whether the user simulator faithfully executed its script.It uses the script, interaction trajectory, hard-check results, and environment evidence pack.Source: §D.4 · pp. 24, 25, 27
  • Checklist items receive pass, fail, or n/a verdicts; n/a is excluded from the denominator when the situation never occurred.A pass requires positive evidence, and a claimed action must be backed by subsequent action or environment evidence.Source: §D.4 · p. 25
  • For timing-sensitive items, the judge anchors evaluation to the agent’s first response or action after the probe or user cue.A behavior performed only after later prompting or escalation fails when the criterion requires it proactively or at the first opportunity.Source: §D.4 · p. 25
  • Simulator fidelity checks cover script intent, disclosure, answers, state assertions, probe coverage, message validity, refusals, and timing.The judge uses the script as the sole basis for evaluating the simulator and records affected collaboration dimensions when simulator failure directly causes a low score.Source: §D.4 · pp. 26–27
  • The judge marks a collaboration dimension as contaminated when simulator failure directly causes its low score, while still scoring that dimension.The evaluation separates task completion, which the verifier computes, from the judge’s assessment of interaction quality.Source: §D.4 · pp. 25, 27

E.1 USER SIMULATOR CONTROL AND TRACE AUDITING

REALCLI’s scripted user delivers information and probes according to defined triggers, with interaction records preserved in an append-only ledger. Audits find high simulator fidelity overall, while task-outcome shifts differ across models despite similar audit results.

E.1 USER SIMULATOR CONTROL AND TRACE AUDITING

  • The scripted user releases information slices and probe stimuli according to triggers, while an append-only ledger records messages, releases, probes, and closing reasons.An independent judge assesses simulator fidelity separately from agent collaboration performance using the script, transcript, and environment evidence.Source: §E.1 · pp. 27–28
  • Table 8 shows Kimi K3 gaining 8.9 percentage points and Hy3 losing 7.9, although both score at least 94.9% on three user-side audit measures.Audit records cover 98.0–100.0% of tasks in each representative configuration; similar audit outcomes accompany opposite task-outcome shifts.Source: Table 8 · p. 28, §E.1 · p. 28
  • 98.0% of audited trajectories receive a no-harmful-deviation judgment for simulator fidelity.Observed deviations include empty or protocol messages, incomplete probe coverage, and departures from script intent.Source: §E.1 · p. 28

E.2 INTERACTION REORDERS TASK SUCCESS

Interaction changes which tasks succeed rather than reliably improving aggregate success, while making trajectories longer. Models more reliably respond to current requests than retain the full goal state, and many near-acceptance failures still miss final requirements.

E.2 INTERACTION REORDERS TASK SUCCESS

  • 36.5% of matched trials change task status under interaction: 17.9% shift from failure to pass and 18.5% from pass to failure, offsetting to a 0.6-point aggregate difference.Every model has both newly successful and newly failed tasks; interaction reorders success across model–task pairs rather than producing a uniform gain.Source: §E.2 · p. 28, Figure 7 · p. 29

E.3 LONG-HORIZON INTERACTION COSTS AND OUTCOMES

  • Interactive trajectories are longer in all 110 model–category cells, and among pairs with incomplete complete-instruction reward and more interactive tool calls, 59.9% do not improve reward.Median interactive-to-complete-instruction tool-call ratios range from 1.43× to 3.62×; 82.1% of MiniMax M3 retries return another error, while all 47 observed DeepSeek V4.1 retries succeed.Source: §E.3 · p. 28, Figure 8 · p. 29

E.4 IMMEDIATE RESPONSE AND CROSS-TURN GOAL MAINTENANCE

  • Models follow successive requests in 98.8% of cases but retain early constraints in 54.4% and resolve instruction conflicts in 29.3%, despite an 80.2% plan-update pass rate after blockers.The results distinguish reactions to current state from maintaining new information alongside the existing goal state.Source: §E.4 · pp. 28–29

E.5 ACCEPTANCE GAPS UNDER COMPLETE INSTRUCTIONS

  • 45.7% of autonomous failures with complete check summaries pass at least 90% of reported checks, yet unresolved conditions still prevent final acceptance.Passing local checks does not ensure that critical boundary conditions, interfaces, and output requirements are satisfied.Source: §E.5 · p. 29

E.6 TASK SUCCESS AND COLLABORATION SUCCESS

Task success and collaboration success diverge: successful interactive trajectories often miss collaboration scenarios. Constraint-retention failures also commonly appear in final artifacts, even when dialogue behavior passes its checks.

E.6 TASK SUCCESS AND COLLABORATION SUCCESS

  • 91.0% of successful interactive trajectories miss at least one applicable collaboration scenario, so task completion does not establish collaboration completion.A final artifact can pass environment-level verification while authorization, conflict handling, or risk communication remains unmet.Source: §E.6 · pp. 29–30, Figure 11 · p. 31

E.7 FROM STATED CONSTRAINTS TO DELIVERED ARTIFACTS

  • 91.3% of constraint-retention failures are artifact mismatches, and 53.3% of those with communication checks pass every applicable communication check.Constraint handling therefore needs evaluation in both dialogue and the verifiable delivered artifact.Source: §E.7 · p. 30, Figure 12 · p. 31

Related papers

Loading 2610.22507v1…