PaperlayerView exact revision on arXiv

Source-linked AI summary

QuoteBench: How Matched Scores Can Hide Command-Path Failures

Shangao Li, Yao Zhang, Volker Tresp, Yuanyuan Yang

arXiv:2608.13547v1cs.AIcs.SE

TL;DR

LLM command-agent scores can conflate generation errors with failures introduced when model output is wrapped or reparsed. QuoteBench crosses generation contracts with execution transports on 56 exact-state Bash tasks, showing that an added parser causes large losses and that disclosed-boundary generation can recover much of the damage. The results support reporting command paths and operating points explicitly rather than treating matched scores as intrinsic model properties.

  • Problem

    Current benchmarks combine command construction with other capabilities or fixed transport, leaving unclear whether agent success survives deployment through downstream command paths.

  • Method

    QuoteBench crosses generation contracts with execution transports on 56 one-shot Bash tasks from 14 incident-derived families and scores exact final states.

  • Results

    55.4–73.2 points of success are lost when fixed replies pass through one added parser, while disclosed-boundary generation recovers 30.4–60.7 points in six of eight configurations.

  • Takeaways & Limitations

    Command-agent evaluations should report model configuration, generation contract, execution path, operating point, and final-state validator rather than matched score alone.

  • Takeaways & Limitations

    The study isolates one-shot Bash generation with one added boundary; its results characterize the constructed benchmark and stored-reply portability, not deployment prevalence.

Abstract

from arXiv · show

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

1 Introduction

QuoteBench targets a gap in command-agent evaluation: matched execution scores do not reveal whether failures arise during generation or after a command crosses a downstream parser. It introduces a final-state benchmark and crossed contract–transport design to measure that distinction and its implications for reporting.

  • Benchmark and motivation: Current benchmarks entangle command construction with planning, navigation, recovery, or fixed transport, so success does not identify where command reliability fails.Bash provides a focused setting because parser rules are public and final state is exactly checkable.
  • Results and reporting: Matched nested success spans 14.3–91.1% at fixed configurations, while best observed operating points range from 14.3–98.2% and include three models at 100.0%.Effort improves matched success for some models but not others, and effort labels correspond to different token budgets across providers.
  • Mechanism identification: 55.4–73.2 points of success disappear when each fixed raw-conditioned reply passes through one added parser.Under the same transport, disclosed-boundary generations recover 30.4–60.7 points in six of eight same-window configurations.
  • Benchmark and motivation: 56 exact-state tasks across 14 operation families turn recurring Bash quoting and escaping incidents into a focused command-path benchmark.Tasks include benign controls and hazardous payloads involving multiline text, hostile filenames, metacharacters, heredocs, literal arguments, JSON, Git, and local SSH-like simulations.
  • Mechanism identification: The crossed design independently varies generation contract and execution transport, decomposing matched scores into transport damage and contract-conditioned compensation.Fixed-reply replay makes the transport intervention attributable to the command path rather than to a new model generation.
  • Results and reporting: Evaluations should report the generation contract, execution transport, operating point, and final-state validator rather than treating matched success as intrinsic to a model.The paper also motivates path-matched model and effort selection.

2 Related Work

Prior work evaluates agents, commands, action representations, and safeguards, but often leaves the command channel implicit. QuoteBench makes the model-facing contract and downstream transport explicit and measures their interaction with final-state validation.

  • Agent and terminal benchmarks: General agent and terminal benchmarks provide realistic end-to-end evaluation but do not isolate the contribution of the command interface.They cover web navigation, desktop control, coding, interactive execution, command-line workflows, and environment setup.
  • Shell-command generation: Shell-command generation benchmarks test translation, functional equivalence, or robustness, while QuoteBench focuses on post-generation command-path transport.The related benchmark landscape includes NL2Bash, NLC2CMD, NL2SH-ALFA, and BashBench.
  • Action representations and boundaries: Changing the agent–computer interface or action language can alter performance, but structured-call validity does not guarantee shell correctness after delivery.QuoteBench crosses the model-facing contract with downstream transport and checks the bytes delivered to the executor.
  • Evaluation validity and safeguards: QuoteBench complements repeated-sampling studies, validator audits, and pre-execution verification by auditing final-state validators and measuring the dispatch boundary.Its validators use initial states, oracles, naive probes, and targeted mutations.

3 The QuoteBench Benchmark

QuoteBench is a controlled one-shot Bash benchmark that combines exact final-state validation with independent generation contracts and execution transports. Its crossed cells make boundary damage and contract adaptation measurable while preserving fixed tasks, replies, and validators.

  • Benchmark construction: 56 tasks span 14 operation families, each with one benign control and three hazardous payload variants.The same frozen core is reused across campaigns, covering literal content, hostile filenames, metacharacters, heredocs, argument and environment passing, JSON, Git, and second-parser simulations.
  • Benchmark construction: Validators score exact final state—including bytes, argv, JSON, directory state, and Git history—rather than exit codes.A command receives credit whenever it reaches the target state; many failing executions exit zero while leaving the wrong state.
  • Scope: The benchmark covers repeatedly observed command-construction mechanisms, not prevalence or complete shell behavior.It focuses on POSIX/Bash and excludes PowerShell, Windows CMD, authentication, network failures, interactive state, and multi-turn recovery.
  • Contracts and transports: Generation contract selects the stored reply, transport selects how it reaches Bash, and final-state validation scores the resulting state.The design uses raw, native, and disclosed-boundary contracts crossed with raw or nested execution.
  • Contracts and transports: Nested transport adds a double-quoted parser boundary representative of remote, container, or CI command paths.A fixed-commit survey found both raw and native contracts, and real ssh localhost replay reproduced the nested damage for seven of eight configurations.
  • Operating points: Best-observed scorecard rows select each model’s highest matched-nested All-56 score from measured settings, making the table descriptive rather than a controlled compute ranking.Fixed same-window configurations instead support mechanism analysis, while effort ladders and settings vary by provider.

4 Results

QuoteBench shows that matched scores conceal substantial command-path effects: raw execution is nearly saturated, while nested transport damage and contract-conditioned adaptation determine observed robustness and can reorder models.

  • Three models pass all 56 tasks at their best observed settings, while the remaining scores range from 14.3% to 98.2%.
  • Raw-path scores reach 91.1–100%, but fixed matched-nested success spans 14.3–91.1, leaving command-path handling as the main discriminator.
  • 55.4–73.2 points are lost when fixed raw replies move from raw to nested transport, including 28.6–57.1 points on benign controls.
  • The boundary-aware contract changes command behavior in a transport-specific way, with interaction effects ranging from −7.1 to +119.6 points across configurations.
  • GPT-5.6-sol’s −3.6-point matched gap combines −64.3 points of transport damage with +60.7 points of compensation.
  • The deployment path partially reorders models: GPT-5.6-sol reverses position with Gemini-3.5-Flash, while 22 of 28 pairwise orderings remain stable in resampling.
  • 30.4–60.7 points of compensation appear in six of eight configurations under disclosed-boundary generation, while Qwen3.5-27B shows 0.0 and Gemini-3.1-Flash-Lite −5.4.
  • Disclosure elicits grammar-conditioned adaptation: capable models perform far better on the grammar they were told than on the alternate grammar.

5 Discussion

QuoteBench treats the command interface as part of the evaluated system: deployment paths, contracts, operating points, and userlands can alter observed reliability and model selection. Cross-path replay also separates transport damage from model compensation and motivates reporting practices beyond matched scores.

  • Reporting practices: Structured actions remove one quoting boundary, but payload-level representation errors remain possible.In the typed pilot, 11 of 36 programs failed and 10 left the wrong final state.
  • Model and operating-point selection: Provider ladders are non-monotonic, so users should compare models and effort settings on the deployed execution path.Defaults can map to different operating points across providers.
  • Model and operating-point selection: Path-aware selection reaches 51/56 nested successes, versus 50/56 when selecting GPT-5.5 by 56/56 raw success.The two scorecards answer different questions, and neither is a controlled compute ranking.
  • Userland sensitivity: Userland crossover cells differ by at most 3.6 points for six configurations and by 7.1–12.5 points for two others.Added-parser damage remains negative in both BSD/macOS and GNU/Linux.
  • Beyond shell: 51.8–66.1 points of success are lost when stored replies pass through naive JSON embedding, while correct round-trip serialization removes the effect.This extends the added-parser mechanism beyond shell transport.
  • Practical implications: Escaping at the interpolation point restores every raw-path success, while temporary scripts preserve the program boundary when callers cannot control transport.The script workaround incurs a file-lifecycle cost, and boundary disclosure permits model compensation.

6 Conclusion

QuoteBench demonstrates that matched execution scores can conceal substantial post-generation transport failures. Fixed-reply replay exposes the damage, while disclosed-boundary generation compensates for it in most configurations.

  • Conclusion: 55.4–73.2 points of success disappear under one added parser, while contract-conditioned generation recovers 30.4–60.7 points in six configurations.These results come from eight same-window configurations with fixed-reply replay.

Limitations

QuoteBench isolates one-shot Bash generation under quotation and interpolation hazards, using constructed families and a nested transport to study mechanism attribution. Its conclusions are bounded by the benchmark design, stored-generation analyses, and limited coverage of tools, shells, and recovery modes.

  • Scope: The benchmark characterizes one-shot Bash generation and stored-reply portability, not deployment prevalence.The nested transport reproduces a real remote-execution boundary, but the incident-derived families are not prevalence estimates.
  • Study design: Causal claims rely on fixed replies from eight same-window configurations, while effort ladders use one stored generation per task.Effort labels are not comparable compute budgets.
  • Study design: Native-tool results are observational, and the typed-operation study covers only six naturally typeable families.Held-out payloads test transfer to unseen literals without difficulty matching.
  • Open scope: Other shells and multi-turn recovery remain open beyond the tested Bash, one-shot setting.The paper tests one boundary rather than the full space of transformed-output pipelines.

Broader Impact Statement

QuoteBench executes untrusted model output in isolated, timeout-bounded, network-disabled fixtures and limits release of incident evidence. Publishing the frozen benchmark creates contamination risk, addressed through versioning and private regenerated variants.

  • Safety and release: Each attempt runs in a fresh fixture inside a timeout-bounded, network-disabled container.Incident evidence is released only as de-identified mechanism classifications.
  • Safety and release: Publishing the frozen core creates contamination risk, so the benchmark is treated as a versioned audit set with private regenerated variants held out.Released task files also embed fixed canary GUIDs for downstream contamination checks.

A Benchmark Construction, Coverage, and Contracts

QuoteBench builds an auditable Bash command-path benchmark from surveyed failure mechanisms, varying generation contracts and downstream parser boundaries independently. Its 56-task core uses exact final-state validation and controlled replay to distinguish model construction from transport effects.

  • Survey and coverage: 86 internal incidents and 412 screened public reports guided coverage of 14 command-construction mechanism families.The public survey retained 17 model-level incidents, each mapping to a mechanism represented in the core.
  • Experimental design: Generation contract and observed boundary are varied independently, while fixed-reply replay separates transport damage from contract-conditioned compensation.The contract specifies what the model produces; the observed boundary specifies what the harness does with the reply.
  • Execution boundaries: The nested condition adds one downstream double-quoted parser, representing boundaries found in remote, container, and CI command paths.Replay through real ssh localhost reproduces the synthetic nested damage to the decimal for seven of eight configurations.
  • Scope: The 56-task core excludes PowerShell, Windows CMD, authentication, real SSH networking, interactive terminal state, and complete multi-turn recovery.The incident evidence supports mechanism coverage, not prevalence estimation.
  • Generation contracts: The raw contract executes the reply directly with bash -c, whereas the disclosed contract interpolates reply R inside double quotes before execution.The two full prompts differ only in the execution-contract sentence; no escaping instruction is added.

B.1 Effort ladders and model configurations

The study compares provider-exposed effort settings at matched operating points, with fixed-reply replays enabling an eight-configuration same-window crossover. Effort labels are provider-specific rather than common compute units, and the analysis is restricted to supported same-window configurations.

  • Configuration comparability: Provider labels are within-model controls, not common compute units, and Qwen exposes a think toggle rather than a multi-rung effort parameter.Haiku-4.5 output-token means are right-skewed, with medians of 1,206, 1,560, 936, and 1,283 tokens.
  • Replay design: Six configurations provide stored raw and disclosed-boundary replies at every acted-on rung, while Gemini-3.1-Flash-Lite is excluded from the crossover count because its four rungs are byte-identical.Both transports replay stored replies without additional model calls.
  • Analysis scope: The eight same-window configurations are the scope of mechanism analysis; broader matched ladders support descriptive operating-point comparisons only.Off-diagonal cells across serving windows are not reconstructed because omitted effort fields cannot be treated as common neutral rungs.
  • Compensation analysis: Six configurations qualify for the supported positive-compensation set after effect, Holm-adjusted significance, and leave-one-family-out criteria are applied.The qualifying configurations are GPT-5.6-sol, GPT-5.5, Opus-5, Gemini-3.1-Pro, Opus-4.8, and Gemini-3.5-Flash.

C.1 Userland robustness

The fixed-reply transport effect persists across GNU and BSD/macOS userlands, although utility dialects can independently change rankings and transfer outcomes. Masked matched scores arise when large transport damage and compensation cancel at an operating point.

  • Userland robustness: Fixed-reply damage remains negative for all eight configurations in both userlands, while crossover cells differ by at most 3.6 points for six provider-hosted configurations.The two remaining configurations differ by 7.1–12.5 points, and four of 48 ladder comparisons change internal rung order.
  • Cross-userland transfer: 8.9 points is the largest BSD-to-GNU transfer shift, and both raw and nested rankings change across userlands.The transfer replays identical BSD-elicited commands in GNU, rather than estimating GNU-targeted generation.
  • Effect-sign transfer: Gemini-3.1-Pro and Fable-5 change effect sign when BSD-elicited commands move from BSD to GNU execution.Both shift from small positive native-minus-raw effects on BSD to small negative effects under GNU.
  • Masked fragility: GPT-5.6-sol meets the descriptive masked-fragility cut with −3.6 = −64.3 + 60.7, and ten of 30 rung-level crossovers meet the same cut.Masking depends on the operating point and selected threshold, not a model-level invariant.
  • Study-B replay scale: 8,736 generated arm records produce 17,472 replay outcomes across BSD and GNU userlands, with aggregate GNU pass rates reported in the main text.The paired analysis adds robustness comparisons and failure diagnostics without additional generation.
  • Final-state validation: 41–62 executions exit zero while leaving the wrong state, representing 23.4–47.0% of failures across the four contract–userland conditions.Native one-call schema adherence is 98.2–100%, so exit status alone misses many final-state failures.

D.1 Study-B robustness and failure analysis

Study B provides robustness and wire-integrity diagnostics for native-versus-raw behavior, but its family-sign sensitivity analysis supports a descriptive and exploratory interpretation rather than adjusted significance.

  • Statistical scope: All twelve Holm-adjusted family-sign values exceed .05, with a minimum of .18750 for Gemini-3.5-Flash in both userlands.The analysis treats 14 operation families as inferential units and applies adjustment separately across six models in each userland.
  • Effect decomposition: Figure 5 compares raw and native arms within each provider-specific effort rung to decompose aggregate effects.The figure organizes effects by within-rung contract comparison rather than by a common cross-provider effort scale.
  • Wire integrity: 660 Gemini-3.5-Flash, 497 Gemini-3.1-Pro, and 672 GPT-5.6-sol records permit byte-level verification between decoded command fields and executor input.Nineteen adherence failures contain no usable argument; other configurations begin at the decoded command-string boundary.

E Replication and Transport Robustness

The replication tests show that fixed-reply transport changes consistently damage command success, while disclosed-boundary gains reflect model adaptation and persist across transport variants. Validation and robustness checks support the measured corruption classes, but the private set’s hostile-only design limits absolute-rate comparisons.

  • Replication design: 168 accepted generations were replayed through both raw and nested transports, isolating transport effects without additional model calls.The private-v2 design interleaved raw- and disclosed-boundary calls within one serving window and retained the first usable response per scheduled cell.
  • Scope: The private set contains hostile payloads only and was not difficulty-matched to the 56-task public core, so absolute rates should be interpreted within the private set.The private result establishes transfer of fixed-reply transport damage across payload samples, not prevalence estimates.
  • Transport replication: The temporary-script transport reproduced raw-path outcomes for all 448 configuration–task pairs, recovering all 292 nested-only failures and rescuing no raw-path failure.Escaped interpolation likewise reproduces the raw-path outcome, separating post-generation parser damage from model-generated command changes.
  • Disclosure and grammar crossover: Matched disclosure gains are largest in the middle of the model ladder, while advice barely moves the top and does not reach the bottom.The same-window matched-nested comparisons use paired differences, and the grammar crossover tests matched versus mismatched disclosed and executed wrappers.
  • Real-ssh validation: Compensation on the real-ssh path matches synthetic nested compensation for five of six replayed configurations, while two non-adapting configurations show no positive compensation.Gemini-3.1-Flash-Lite differs by one task, with −1.8 versus −5.4 points.
  • Cross-parser robustness: A correct JSON serializer causes approximately zero damage, whereas naive embedding produces damage comparable to the shell nested transport because quotes and backslashes are reparsed.Many naively embedded replies fail to parse, and the comparison requires zero model calls.
  • Validator checks: Validator mutation tests rejected all 197 mutated states, although unenumerated false positives remain possible.The checks also accepted every oracle and benign tier-0 probe and rejected every untouched fixture and hostile probe.
Loading 2608.13547v1…