Matched Capabilities, Different Behaviors: A Controlled Study of MCP and CLI Interfaces for Language Agents

1 Introduction

• Different representations of the same executable capabilities can shape local action choices and, through closed-loop interaction, produce different trajectories.Interfaces present actions, arguments, response conventions, and execution feedback as part of the context from which the next action is selected.Source: §1 · pp. 1–2
• MCP presents named tools with schema-defined arguments, while CLI presents operations through commands, flags, and textual conventions.Both can encode the same semantic operations while offering different cues for selecting and expressing them.Source: §1 · p. 2
• Unmatched system comparisons confound interface representation with factors such as tool granularity, prompting, validation, and execution scaffolding.Endpoint metrics can also conceal distinct interaction pathways and the effects of early divergence on later decisions.Source: §1 · p. 2
• MICA compares MCP and CLI while holding executable semantics fixed, then uses matched-state probes and paired trajectories to analyze local sensitivity and downstream propagation.Its analyses distinguish semantic action selection from interface-specific realization.Source: §1 · p. 2
• MCP and CLI with matched executable capabilities still differ substantially in task performance and interaction cost, including differences beyond surface validity.The reported behavioral differences include semantic action selection, failure pathways, and recovery.Source: §1 · p. 2
• DART improves task performance across all six evaluated benchmark–interface settings and reduces interaction cost under MCP.The study presents DART as an intervention on interface-specific realization that preserves validated semantic decisions.Source: §1 · pp. 2–3

Tool Use and Interface Representation

• Prior work links action representations, interface design, and tool metadata to changes in agent behavior and action selection [5] [8] [7] [6] [18].Work on constrained generation and conversational recovery also treats valid realization and post-failure correction as integral to tool-mediated interaction [23] [24].Source: §2 · p. 3

MCP-based Tool Use

• MCP benchmarks cover multi-step planning, tool discovery, and cross-tool coordination, while MICA fixes the semantic contract and post-decoding runtime to study interface presentation, realization, and feedback [19] [25] [26] [27].System-level MCP–CLI comparisons report variation across deployed configurations [17].Source: §2 · p. 3

CLI and Terminal Agents

• Terminal benchmarks capture broad command grammars, compositional actions, and long-horizon workflows; prior enterprise work finds capability packaging explains a substantial part of observed performance differences [28] [29] [16].MICA aligns benchmark-specific operations and execution semantics before comparing MCP-style calls with CLI-style commands.Source: §2 · p. 3

3 Problem Formulation

• MCP and CLI expose matched executable capabilities through different representations built from the same benchmark-specific semantic contract.The normalized semantic action consists of an operation and its normalized typed arguments.Source: §3 · p. 4
• The interface is factored into capability presentation, response protocol and decoder, and feedback representation, while validation, compilation, and execution are shared downstream.This separates response decoding, semantic admission, and environment execution.Source: §3 · p. 4
• At a matched decision context, fixing state, public history, contract, model, and generation configuration enables local comparison of interface components.Along closed-loop interaction, an early difference changes the histories and states conditioning later decisions.Source: §3 · p. 4
• The interaction signature records the semantic proposal, failure stage, execution outcome, and recovery status; trajectory analysis distinguishes aligned interaction before divergence from downstream propagation afterward.Pairs without an observed consequential divergence are recorded as having no observed divergence.Source: §3 · p. 4

4 Matched Interface Comparison and Analysis

• MICA combines contract-matched MCP–CLI construction, matched-state and trajectory analysis, and commitment-preserving realization with DART.The comparison holds executable capabilities fixed to isolate behavioral effects of interface representation.Source: §4 · p. 5

4.1 Contract-Matched Interface Construction

• MICA builds both interfaces from a shared benchmark-specific contract, fixing the executable action space while varying its model-facing representation.The contract specifies typed arguments, state-local argument domains, and native execution bindings.Source: §4.1 · p. 5
• MCP renders named tools with structured arguments and CLI renders registered commands with textual bindings, then both enter the same validation, compilation, and execution path.Surface syntax varies while executable semantics remain fixed.Source: §4.1 · p. 5

4.2 Local Interface Sensitivity and Trajectory Propagation

• Matched-state probes vary presentation, realization, and feedback at fixed contexts, while paired natural trajectories trace interaction after the first consequential divergence.Trajectory suffixes are characterized through semantic-action agreement, failure and recovery, interaction cost, and terminal outcome.Source: §4.2 · pp. 5–6

4.3 DART: Commitment-Preserving Realization

• DART separates semantic action selection from interface-specific expression so a validated decision remains fixed while its realization is repaired.If realization fails while the commitment remains valid, one preserving correction may change the expression but retains the committed operation and normalized arguments.Source: §4.3 · p. 6

Benchmarks and models

• The study evaluates ALFWorld, WebShop, and ScienceWorld with Qwen3-4B/8B/14B/32B, DeepSeek-v4-Flash, and Gemini-2.5-Flash.Native benchmark interfaces serve as calibration references.Source: §5 · p. 7

Controlled evaluation

• MCP and CLI use the same semantic action contract while model, task state, public information, action space, validation, execution, generation settings, and interaction budget are held fixed.Primary metrics are benchmark score or success and total charged model tokens per task.Source: §5 · p. 7

Research questions

• Four research questions cover task performance and cost, semantic decisions and recovery, the origins and propagation of interface effects, and commitment-preserving realization.Source: §5 · p. 7

6.2 Failure and Recovery

• For WebShop/Qwen3-14B, MCP shifts decoded action frequency by +6 percentage points toward click and −6 points from search.ALFWorld and ScienceWorld show corresponding redistributions across their larger operation sets.Source: §6.2 · pp. 7–8

6.1 Task Performance and Interaction Cost

• For Qwen3-14B, MCP trails CLI by 36.6 score points on ALFWorld (20.1 vs. 56.7) but exceeds it by 4.9 points on WebShop (64.4 vs. 59.5).MCP uses 54.9k more tokens per ALFWorld task but 35.0k fewer on WebShop; direction and magnitude vary across model–environment pairs.Source: §6.1 · p. 7, Table 1 · p. 8

6.2 Failure and Recovery

• On WebShop, MCP lowers attempt-level failure rates for all six models by 1.1–8.9 percentage points, while remaining failures shift toward later availability and execution stages.ScienceWorld has a different failure composition, with availability and execution accounting for a larger share of unsuccessful attempts.Source: Figure 2 · p. 8, §6.2 · p. 9
• On WebShop/Qwen3-4B, retry acceptance is 50% under CLI and 77% under MCP, while later task success is 18% and 34%, respectively.Recovery paths update the visible interaction history, changing the context for later decisions.Source: §6.2 · p. 9

6.3 Local Effects and Trajectory Propagation

• Realization effects reach 84.4 percentage points for ALFWorld/Qwen3-32B and 72.9 points for ScienceWorld/Qwen3-4B; WebShop/Qwen3-14B's three-way interaction changes admission by 37.8 points.The figure separates realization failure and admission given decoding; component effects depend on surrounding interface configuration.Source: Figure 3 · p. 9, §6.3 · p. 9
• Across 2,004 paired MCP–CLI episodes, 1,889 (94.3%) exhibit structured divergence, with first divergence near the beginning of interaction.Semantic-action differences dominate divergent pairs, realization differences recur, and trajectories accumulate different token costs, attempt counts, and terminal outcomes.Source: §6.3 · p. 10

6.4 DART Evaluation

• DART gains +1.5/+9.0 points on ALFWorld, +0.8/+4.4 on WebShop, and +6.7/+5.3 on ScienceWorld for CLI/MCP.Under MCP, token reductions are 27.1k, 11.2k, and 4.3k per task across the three environments; under CLI, token cost changes by −0.8k, +4.8k, and +1.6k.Source: Figure 4 · p. 10, §6.4 · p. 10
• DART preserves validated semantic commitments and changes only interface-specific realization; 71–100% of triggered interventions are admitted on the first realization.Three of four preserving-correction opportunities are admitted after one correction.Source: §6.4 · p. 10

Interface representation

• With the same semantic action contract and downstream runtime, MCP and CLI induce different policies through differences in presentation, realization, and feedback.The discussion argues that interface design belongs in the agent’s decision environment, not merely its access layer.Source: §7 · p. 10

Trajectory-level behavior

• Local changes to responses alter the interaction history and state that condition subsequent decisions, making trajectories an appropriate unit of analysis.Later semantic realignment may occur after the trajectories have already followed different histories.Source: §7 · p. 11

Semantic commitment

• DART makes semantic commitment an explicit controller boundary by preserving a validated action and repairing only its interface-specific realization.The discussion frames this as a recovery design that restricts repair to the representation-specific component requiring correction.Source: §7 · p. 11

Conclusion

• MICA connects local representational sensitivity to downstream differences in semantic decisions, recovery, interaction cost, and task outcomes; DART improves benchmark score in all six evaluated benchmark–interface settings.DART preserves validated semantic commitments while repairing interface-specific realization.Source: §8 · p. 11

• MICA and DART preserve the existing executable capabilities, action contract, runtime, and evaluation safeguards rather than expanding the agent’s action authority.Source: Ethics Statement · p. 11

• Reproduction materials cover task sets, generation settings, budgets, protocols, metrics, statistical analysis, interface construction, conformance checks, and DART configuration.The paper also specifies response contracts and states that code is available in its repository.Source: Reproducibility Statement · p. 11

AI Use Statement

• The authors used generative AI for language polishing, code implementation assistance, and figure preparation, and reviewed the resulting material.They retain responsibility for the paper’s technical content and final presentation.Source: AI Use Statement · p. 12

A Supplementary Results by Research Question

• Supplementary results examine endpoint costs, action selection and recovery, matched-state sensitivity and trajectory divergence, and DART’s realization-boundary intervention.Source: A Supplementary Results by Research… · p. 16

A.1 RQ1: Endpoint Performance and Interaction Cost

• On ALFWorld, Qwen3-14B’s MCP cost increase is driven mainly by more calls (46.9 to 57.1); on WebShop, MCP reduces calls (14.2 →9.6) and tokens per call (4.26k →3.23k).Table S1 reports task success, calls per task, and tokens per call; Figure S1 shows paired task-level token effects and model-level means.Source: Figure S1 · p. 16, §A.1 · p. 17, Table S1 · p. 17

A.2 RQ2: Semantic Decisions, Failures, and Recovery

• Interfaces change decoded semantic-action frequencies and failure pathways; on WebShop, MCP lowers attempt-level failure rates for all six models by 1.1–8.9 percentage points.For Qwen3-14B on WebShop, MCP shifts action frequency by +6 percentage points toward click and −6 from search. On WebShop/Qwen3-4B, retry acceptance is 50% under CLI and 77% under MCP, while later task success is 18% and 34%, respectively; immediate acceptance and later success move on different scales.Source: §A.2 · pp. 17–19, Figure S2 · p. 18, Table S2 · p. 18

A.3 RQ3: Local Origins and Downstream Propagation

• 1,889 of 2,004 paired trajectories (94.3%) show structured divergence; differences often emerge early and lead to path-dependent interaction costs and outcomes.After the first consequential difference, paired runs can accumulate different token costs and attempt counts and reach different terminal outcomes; later semantic actions can realign to varying degrees.Source: §A.3 · pp. 19–20, Figure S7 · p. 21
• At matched decision states, interface components interact: for Qwen3-32B on ALFWorld, changing only realization changes failure from 1% to 87% under MCP presentation and from 0% to 89% under CLI presentation.Admission and semantic-action entropy also shift across component assignments, indicating that presentation, realization, and feedback jointly affect expression success and decoded choices.Source: §A.3 · p. 19, Figure S5 · p. 20

A.4 RQ4: Commitment-Preserving Realization with DART

• DART resolves 71–100% of triggered interventions on the first realization; among four that proceed to preserving correction, three are admitted after that correction.It fixes the validated operation and typed arguments while allowing the interface-specific expression to change; correction is used only when the initial realization fails and the commitment remains valid.Source: §A.4 · pp. 23–24, §D.1 · p. 31
• DART improves benchmark score in every evaluated benchmark–interface setting and uses fewer total tokens under MCP in all three environments.The passages report these overall results without giving the score gains or token reductions here.Source: §A.4 · p. 25

B.1 Evaluation Suite and Controlled Comparison

• The controlled evaluation pairs MCP and CLI on the same model checkpoint, task state, semantic contract, validation and execution path, generation configuration, and interaction budget.It evaluates six models across ALFWorld, WebShop, and ScienceWorld, yielding 2,004 paired episodes; the controlled variation is capability presentation, action realization, and feedback representation.Source: §B.1 · p. 25

B.4 Matched-State Probe Protocol

• Matched-state probes sample 338 source states and 21,632 responses across eight assignments of presentation, realization, and feedback, restarting each draw from the same registered context.The assignments vary those interface components while preserving the semantic interaction history; the response statistics distinguish realization failure, validator admission, and decoded-action entropy.Source: §B.4 · pp. 26–27, §B.5.3 · p. 28

C.1 Shared Semantic Boundary

• MCP and CLI render the same state-local operations and typed arguments differently, then decode proposals into a shared semantic action space and common validation, compilation, and execution path.The paper treats proposals as semantically equivalent when their registered operation and normalized typed arguments agree.Source: C Contract-Matched Interface… · p. 28, §C.1 · pp. 28–29

D.2 Error-Triggered Controller

• DART selects and validates a semantic action, realizes it through the assigned interface, and permits one correction only if the commitment remains valid.The controller releases control when selection or realization fails validation, or when the preserving correction fails.Source: §D.2 · p. 32

D.3 Trigger Scope and Benchmark Instantiation

• DART activates only for retryable, policy-owned failures under confirmed unchanged state, when a valid semantic commitment needs corrected realization.Eligible failures include interface decoding or validation errors and backend execution rejections.Source: §D.3 · p. 32

D.4 RQ4 Paired Evaluation

• Each of six Gemini-2.5-Flash benchmark–interface settings pairs DART with Base on the same tasks, sharing the pipeline and budgets except for DART’s controller and trigger configuration.ScienceWorld uses the same stable domainorder-v2 protocol for both conditions; endpoint token cost counts all dispatched requests.Source: §D.4 · pp. 32–33

E.1 Prompt Assembly and Ordinary Policy

• The ordinary policy and message order stay fixed across MCP and CLI, while the rendered contract, response surface, and visible interaction context provide interface-specific information.Outside DART triggers, this policy governs the trajectory; prompts also instruct the model not to repeat an accepted action when a controller notice reports no progress.Source: §E.1 · p. 33

E.2 Semantic Selection Contract

• Semantic selection requires one JSON object with schema_id and arguments, validated against the current contract before a commitment is created; both model paths use local validation without guided decoding.Interface-specific realization begins only after validation.Source: §E.2 · p. 34

E.3 Realization and Preserving Correction

• Realization and correction prompts require the model to preserve the committed operation and every typed argument, correcting only the interface expression after failure.The failed realization and public failure feedback are supplied for correction while the commitment remains unchanged.Source: §E.3 · p. 34