PaperlayerView exact revision on arXiv

Source-linked AI summary

Matching Matters: A Fair Quality-Efficiency Benchmark for Command-Line Agents

Han Chi, Jiaxin Qi, Yan Cui, Baisheng Lai, Jianqiang Huang

arXiv:2606.21140v2cs.SEcs.AI

TL;DR

Existing agent benchmarks often emphasize task completion and use fixed interfaces, leaving LM–CLI matching and deployment efficiency less directly assessed. The paper introduces AGENTMETER and AGENTMETER-OPT to compare complete configurations using quality–efficiency measures and task-preserving description variants. Experiments find that leading configurations vary by metric and model, while description changes can alter pair rankings.

  • Problem

    Existing benchmarks often use a fixed interface and emphasize task completion, leaving LM–CLI matching and deployment efficiency underexamined.

    Source: Introduction · p. 1

  • Method

    AGENTMETER evaluates complete LM–CLI configurations with AMS, while AGENTMETER-OPT uses pair trajectories to construct task-preserving descriptions and evaluates targets across shared conditions.

    Source: Introduction · p. 2

  • Results

    No CLI is consistently best across language models, pass rate, execution cost, and AMS select different leaders, and task descriptions can alter pair rankings.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    Source-balanced Pooled evaluation reduces dependence on any single formulation, while generic public-only rewriting does not explain the main gains.

    Source: Conclusion · p. 7

    Pooled evaluation is a controlled source-balanced description distribution, not an estimate of real-user instructions, which would require a separate collection study.

    Source: Further Discussion and Limitations · p. 11

Abstract

from arXiv · show

Rapid advances in large language models have improved the task-solving capabilities of command-line-interface (CLI)-based agents, whose CLIs determine how models invoke tools, maintain interaction history, and recover from failures. Consequently, effective matching between CLIs and LLMs has become essential. However, existing agent benchmarks largely emphasize success rate while overlooking practical objectives such as cost and efficiency, as well as the selection of LM-CLI combinations, all of which are critical in real-world deployment. We therefore introduce AgentMeter, a quality-efficiency benchmark with a new metric, the AgentMeter Score (AMS), that jointly characterizes task quality, budget sensitivity, and resource-intensive zero-reward execution, enabling a more complete assessment of deployed LM-CLI pairs. Furthermore, collected task descriptions may inadvertently favor LM-CLI pairs that are particularly compatible with their wording and structure, causing evaluation results to reflect description-specific advantages rather than general task-solving capability. We therefore propose AgentMeter-Opt, a trajectory-grounded optimization framework that constructs pair-adapted, task-preserving description variants to build a fairer evaluation set across LM-CLI pairs. Extensive experiments show that no CLI is universally optimal across language models and that task success, execution cost, and AMS identify different competitive configurations. Results on AgentMeter-Opt further reveal that task-preserving description changes affect LM-CLI pairs unevenly and can alter their relative ordering across valid description conditions. Together, AgentMeter and AgentMeter-Opt provide a practical foundation for fair and deployment-relevant evaluation of command-line agents.

Introduction

Existing evaluations often fix one CLI and emphasize task completion, leaving LM–CLI matching and execution efficiency underexamined. AGENTMETER measures complete configurations, while AGENTMETER-OPT tests them across pair-adapted, task-preserving descriptions.

Introduction

  • Fixed-interface benchmarks obscure LM–CLI matching, while completion-only metrics treat concise and costly successes—and early and prolonged failures—the same.CLIs differ in tool access, prompting, context management, and failure recovery, so one-interface conclusions may not generalize to other configurations.Source: Introduction · pp. 1–2
  • AGENTMETER evaluates complete LM–CLI configurations and defines AMS to combine graded task quality, quality under calibrated budgets, and costly zero-reward executions.Its benchmark includes 80 executable tasks and a 30-task Core Subset balanced across observed-effort strata.Source: Introduction · p. 2
  • AGENTMETER-OPT uses each source pair’s original trajectory to create task-preserving descriptions, then evaluates targets under Original, Self, Cross, and source-balanced Pooled conditions.The preservation gate retains task objectives, artifact roles, required outputs, and success criteria while excluding hidden evaluation information and prescriptive solution steps.Source: Introduction · p. 2, Figure 2 · p. 3
  • No CLI is uniformly best across language models, and pass rate, execution cost, and AMS select different leading configurations; Core and Full rankings align at ρ = 0.835.The authors describe the Core Subset as a practical, efficient, and scalable setting for broad LM–CLI comparison.Source: Introduction · p. 2
  • Pair-optimized descriptions improve Self performance, transfer unevenly across pairs, and change Pooled rankings, whereas public-only rewriting yields substantially smaller aggregate gains.The reported Core–Full ranking alignment supports Core as an efficient alternative for broad configuration comparisons.Source: Introduction · p. 2

Related Work

Prior work studies executable-agent benchmarks, interface effects, efficiency metrics, and prompt optimization, but commonly evaluates models through fixed interfaces or optimizes for a target system. This paper compares complete LM–CLI pairs and uses task-preserving, source-conditioned descriptions to study evaluation sensitivity.

Agent benchmarks

  • Existing benchmarks span software engineering, terminal tasks, structured skills, data analysis, and broader tool use, but model comparisons usually use a fixed agent interface.AGENTMETER instead compares multiple models and CLIs under the same tasks, environments, and evaluators.Source: Related Work · p. 3

Agent interfaces and evaluation metrics

  • Prior work measures trajectory progress and accuracy–cost trade-offs, while agent systems show that observations, action spaces, tool protocols, and execution harnesses affect behavior.AMS combines graded task quality, quality within calibrated budgets, and costly zero-reward execution when ranking LM–CLI pairs.Source: Related Work · p. 3

Prompt sensitivity and optimization

  • Unlike prompt-optimization methods that improve a target system, AGENTMETER-OPT studies evaluation sensitivity using source-conditioned descriptions and shared Original, Self, Cross, and equal-weight Pooled conditions.Each description is fixed before transfer, and every target is evaluated over the same source-balanced set.Source: Related Work · p. 3

Methodology

The methodology evaluates language-model–CLI configurations on defined tasks, measures quality and execution effort through AMS, and constructs task-preserving descriptions adapted to each configuration.

Problem Formulation

  • Each LM–CLI configuration is evaluated on tasks defined by their objective, artifacts, environment, evaluator, and success criterion, producing a trajectory, reward, and billable cost.Source: Problem Formulation · p. 3

AgentMeter Score

  • AMS calibrates task effort from reference configurations’ token use, execution steps, and zero-reward frequency, then uses effort to set cost scales and form three task strata.Source: AgentMeter Score · pp. 3–4
  • AMS combines graded task rewards, reward achieved within calibrated cost budgets, and the rate of costly zero-reward executions.The budget-quality component counts reward only when execution cost is within budget; expensive failures are treated separately because zero-reward runs contribute nothing to that component.Source: AgentMeter Score · p. 4
  • Within each effort stratum, AMS weights task quality against budget-conditioned quality, discounts frequent expensive failures, and macro-averages strata equally.Calibration settings are fixed before scoring and applied consistently across configurations.Source: AgentMeter Score · p. 4

AGENTMETER-OPT

  • AgentMeter-Opt uses each source configuration’s trajectory to adapt task descriptions while preserving task semantics and excluding hidden evaluation information and prescriptive solution steps.It selects descriptions by source-configuration reward, preferring lower billable cost when rewards tie, then gives every configuration equal representation in the shared expanded evaluation set.Source: AGENTMETER-OPT · pp. 4–5
  • Table 1 covers 28 LM–CLI configurations and reports different leaders for pass count, cost per success, and AMS.Tier-P reports Easy/Medium/Hard passes, and Gap is measured from the best AMS.Source: Table 1 · p. 5

Experiments

Experiments evaluate a 7 × 4 LM–CLI grid and test how benchmark rankings respond to metric choice and task descriptions. The results show that completion, cost, and AMS identify different leaders, while pooled descriptions change the leading configuration and pair-conditioned rewriting outperforms generic rewriting.

Datasets

  • On the 30-task Core Subset, the full 7 × 4 grid comprises 840 pair–task runs, while the 80-task Full Benchmark provides calibration reference coverage.The 16-task AGENTMETER-OPT subset spans 5 Easy, 7 Medium, and 4 Hard tasks.Source: Datasets · p. 5

Implementation Details

  • The four selected description-transfer pairs enable same-model/different-CLI and different-model/same-CLI comparisons, with Original, Self, and Cross conditions totaling 320 executions.Pooled balances four source-conditioned outcomes per target; Generic adds 64 executions and uses a shared public-only rewrite.Source: Implementation Details · pp. 5–6

Result Analysis

  • 18 passes each lead GLM-5.1/Qwen Code and Gemini 3.5 Flash/Codex CLI, while Qwen3.6-Plus/Kimi Code CLI leads AMS at 0.525 and Qwen3.6-Plus/Codex CLI has the lowest successful-execution cost.The cost leader uses 0.48M tokens and 0.074 USD per pass; the same 7 × 4 grid yields different leaders under completion, cost, and quality–efficiency criteria.Source: Result Analysis · p. 6
  • Across Qwen3.6-Plus’s four interfaces, AMS ranges from 0.325 to 0.525, and the best CLI varies across language models.The authors therefore caution that one fixed interface would conflate model capability with LM–CLI compatibility.Source: Result Analysis · p. 6
  • Across 16 configurations evaluated on both sets, Core and Full AMS rankings correlate at ρ = 0.835, supporting Core as a more efficient setting for complete-grid evaluation.The Core Subset contains 30 tasks; the Full Benchmark contains 80.Source: Datasets · p. 5, Result Analysis · p. 7
  • Pooled evaluation raises macro AMS from 0.333 to 0.458 and changes the leader from GLM-5.1/Qwen Code to Qwen3.6-Plus/Kimi Code CLI.Pooled evaluates each target over the same fixed, source-balanced description distribution; across 10,000 Adapted-1 assignments, Qwen3.6-Plus/Kimi Code CLI ranks first in 40.2% of cases versus 9.6% for the Original leader.Source: Result Analysis · p. 7
  • Cross-only retains macro AMS of 0.434 versus Original’s 0.333, and 114 of 192 Cross evaluations improve reward or preserve reward at lower cost.All four pairs retain an improvement over Original; Cross-only excludes each target pair’s own source description.Source: Result Analysis · p. 7
  • Self descriptions improve mean reward and AMS over Original for all four pairs, reaching macro values of 0.647 and 0.531 while reducing aggregate cost by 37.9%.Generic uses one shared rewrite per task based only on the Original public instruction and yields inconsistent pair effects with limited aggregate changes.Source: Result Analysis · p. 7

Conclusion

AgentMeter evaluates complete LM–CLI configurations with AMS, while AgentMeter-Opt studies sensitivity to task descriptions. The experiments find different leaders across evaluation objectives and describe source-balanced pooling as reducing formulation dependence, while broader pair coverage and repeated trials remain needed to assess generality.

Conclusion

  • AgentMeter evaluates complete LM–CLI configurations using AMS, which combines graded task quality, quality under calibrated cost budgets, and costly zero-reward failures.Experiments find that pass rate, execution cost, and AMS identify different leading configurations, and no CLI is consistently best across language models.Source: Conclusion · p. 7
  • AgentMeter-Opt shows that measured performance and ranking can depend on collected task formulation, while source-balanced Pooled evaluation reduces dependence on any one formulation.Generic public-only rewriting does not explain the main gains; the authors suggest effective adaptation depends on diagnosing ambiguities exposed by each pair’s trajectory.Source: Conclusion · p. 7
  • The description analysis uses a controlled source–target setting, and broader pair coverage and repeated trials are needed to assess how general the findings are.Source: Conclusion · p. 7

Benchmark Construction and Calibration

The benchmark calibrates effort and cost using Full Benchmark reference runs, then applies the resulting strata and budgets across evaluation sets. Core and AgentMeter-Opt use smaller task subsets for complete-grid and controlled description-variation evaluation, respectively.

Observed-Effort and Cost Calibration

  • The effort index summarizes three task-level effort statistics, with failure-related effort computed as non-pass frequency from the binary full-pass indicator.Calibration uses runs under Original descriptions on the Full Benchmark.Source: Observed-Effort and Cost Calibration · p. 8
  • For each effort stratum, AUBQC budgets use the 20th, 40th, 60th, 80th, and 90th percentiles of billable cost among positive-reward reference runs.The expensive-failure threshold uses the 90th percentile of zero-reward reference cost within the same stratum.Source: Observed-Effort and Cost Calibration · p. 8
  • The non-pass statistic orders tasks into effort strata, but EFR counts zero-reward executions; partial-reward non-pass runs contribute to TQ and AUBQC, not expensive failures.Source: Observed-Effort and Cost Calibration · p. 8

Evaluation Sets

  • The Full Benchmark’s 80 tasks supply the reference distribution for effort and cost calibration, while Core has 30 tasks and AgentMeter-Opt has 16.The sets contain 80, 30, and 16 executable tasks, respectively.Source: Evaluation Sets · p. 8
  • Core selects tasks using the frozen Full-calibrated effort index and forms 10/10/10 Easy/Medium/Hard strata; AgentMeter-Opt inherits Full effort labels in a 5/7/4 split.Source: Evaluation Sets · p. 8

Reproducibility Details

Experiments used each CLI’s default inference configuration and recorded release, with fixed timeout and execution-level cost and usage tracking. AGENTMETER-OPT builds task-preserving descriptions from public task anchors and a source pair’s trajectory, then screens and selects candidates using source-side outcomes before freezing them for transfer evaluation.

Runtime and Pricing

  • CLIs ran with default inference settings and recorded releases: Codex CLI 0.128.0, Qwen Code 0.15.6, Claude Code 2.1.126, and Kimi CLI 1.41.0.Decoding parameters and inference strength were not tuned for individual LM–CLI pairs.Source: Runtime and Pricing · p. 8
  • Chinese-provider prices were converted at 1 CNY = 0.1471 USD, with cache-read and cache-creation prices included when available.Table A3 reports model prices in USD per million tokens.Source: Runtime and Pricing · p. 8, Table A3 · p. 8
  • Agent execution had a 3,600-second timeout, and setup was limited to 900 seconds when applicable.Rewards, token usage, and billable cost were recorded per execution before task- and stratum-level aggregation.Source: Runtime and Pricing · p. 8

AGENTMETER-OPT Construction Details

  • For each task–source pair, AGENTMETER-OPT extracts public task anchors and compresses the source trajectory into evidence records for candidate generation.Candidates were generated with Qwen3.7-Max under a shared generation configuration.Source: AGENTMETER-OPT Construction Details · pp. 8–9
  • Only task-preserving candidates pass to source-side reruns, and the retained description is selected by reward then cost and frozen before transfer evaluation.Candidates changing the task specification or adding unsupported information are rejected; if none improves reward or preserves it at lower cost, the original is retained.Source: AGENTMETER-OPT Construction Details · pp. 8–9

Additional AMS Results

The additional results support Core as a close approximation to Full in the studied benchmark, while showing why expensive zero-reward executions and parameter sensitivity are assessed separately.

Full-Benchmark Results and Core Validation

  • Core’s size-matched subset audit places it in the 91.4–95.0th percentile for Spearman agreement, 93.8–96.2nd for Kendall agreement, and 90.9–94.6th under maximum rank shift.The common-16 audit also reports strong Core–Full agreement.Source: Full-Benchmark Results and Core… · p. 9

Expensive Zero-Reward Executions

  • Expensive zero-reward executions are a minority of failures but account for a disproportionate share of zero-reward cost, motivating EFR as a separate component.The pair-level analysis identifies configurations most affected when EFR is removed.Source: Expensive Zero-Reward Executions · p. 9

Parameter Sensitivity

  • Across tested sensitivity settings, Qwen3.6-Plus/Kimi remains the leading configuration, while alternative effort weights move 2–10 tasks across strata and α changes 12–20 ranks.The alternative effort-weight settings retain a 5/5 top-five overlap and Spearman correlation of at least 0.980; α ∈ {0.5, 0.7} keeps correlation above 0.96.Source: Parameter Sensitivity · p. 9

Additional Description Diagnostics

These diagnostics examine task-level effects, transfer directionality, and examples of accepted, harmful, or rejected description edits.

Task-Level and Adapted-1 Controls

  • Self has no task-level reward decreases by construction: candidates improve source reward or preserve it at lower cost, otherwise Original is used; Adapted-1 varies broadly by assignment and does not reliably reproduce the source-balanced result.Source: Task-Level and Adapted-1 Controls · p. 9

Cross-Transfer Directionality

  • 114 of 192 off-diagonal Cross evaluations (59.4%) improve reward or preserve reward at lower cost.Table A7 reports effectiveness by source–target direction.Source: Cross-Transfer Directionality · p. 9
  • Cross effectiveness ranges from 37.5% to 75.0%; transfers between pairs sharing Kimi CLI are effective in 65.6% of cases, versus 53.1% for the same model across Kimi and Codex.The results indicate that transfer effectiveness depends on direction.Source: Cross-Transfer Directionality · p. 9

Qualitative Audit

  • The qualitative audit distinguishes negative transfer, which passes the task-preservation gate but harms a target, from edits rejected before rerun.Table A8 provides instruction-level examples of accepted transfer, negative transfer, and gate rejection.Source: Qualitative Audit · p. 9

Further Discussion and Limitations

The discussion treats the complete LM–CLI configuration as the comparison unit and frames AMS and description adaptation as controlled, deployment-oriented evaluation tools. The reported conclusions remain bounded by the studied tasks, configurations, and construction choices.

Further Discussion and Limitations

  • CLI comparisons evaluate bundled prompting, tool schema, observation formatting, context management, and failure recovery rather than estimating the causal effect of any one mechanism.Isolating an individual component would require controlled reimplementations that vary one mechanism at a time.Source: Further Discussion and Limitations · pp. 9–10
  • AMS is a deployment-oriented comparison criterion, not a universal utility function, so the authors report pass and cost statistics alongside it.Its components distinguish absolute task quality, quality under calibrated cost budgets, and unusually costly zero-reward executions.Source: Further Discussion and Limitations · p. 10
  • Pooled descriptions represent a controlled, source-balanced distribution rather than the natural distribution of real-user instructions.Estimating the latter would require a separate collection study with multiple independent human formulations.Source: Further Discussion and Limitations · p. 11
  • The description effects characterize one rewrite generator and configuration; another generator, candidate-search procedure, or human reformulations could produce a different description set.The Generic control yields smaller and less consistent changes than trajectory-conditioned adaptation.Source: Further Discussion and Limitations · p. 11

Limitations

  • Reported rankings are conditional on the selected tasks, calibration reference, pricing configuration, and execution harness, and the AGENTMETER-OPT experiment is a controlled diagnostic rather than an exhaustive estimate.The preservation gate cannot formally prove natural-language equivalence; task-level bootstrap intervals do not quantify repeated-run stochasticity, and reported USD excludes local infrastructure costs.Source: Limitations · p. 11
Loading 2606.21140v2…