Finding the Right Fit: Model-Harness Interactions across Agent Tasks

1 Introduction

• Harness choice shapes how tools, context, errors, retries, and stopping are handled, so a model’s performance can change substantially across harnesses.A harness that suits one model need not suit another.Source: §1 · p. 1
• The study compares 66 model–harness configurations across TUA-Bench, ALE-CLI, and Terminal-Bench 4, including native Codex–GPT and Claude Code–Claude references.The configurable harnesses are OpenHands, DeepSeek Harness, PI, and openJiuwen, each paired with five models.Source: §1 · p. 2
• Claude beats GPT in OpenHands on Terminal-Bench but loses to it in the other three configurable harnesses, by as much as 30 points in PI.The best harness also changes across benchmarks for four models, while openJiuwen gives Kimi its best score on all three.Source: §1 · p. 2
• Matched trajectories suggest that outcomes often depend on whether a harness returns failures as usable feedback, alongside how it handles continuation and task completion.GPT does well with PI’s lean scaffold, while Kimi benefits from openJiuwen catching errors such as malformed tool calls.Source: §1 · p. 2
• The paper contributes cross-benchmark score and cost comparisons, trajectory analyses, and released adapters, evaluation code, and 6,204 scored trajectories.The authors also draw implications for training data and harness design.Source: §1 · p. 2

Agent harnesses

• An agent harness surrounds the model–tool loop with choices about tools, context, memory, planning, permissions, retries, timeouts, and stopping.Prior work includes configurable OpenHands [26], fixed-pipeline Agentless [27], vendor agents Codex and Claude Code, and third-party harnesses such as PI, DeepSeek Harness, and openJiuwen.Source: §2 · p. 2

Benchmarks and cost-aware evaluation

• Many leaderboards fix either the scaffold or the model, leaving their interaction unmeasured; this study reports cost alongside accuracy, following cost-aware evaluation practice [9] [10].Its Terminal-Bench 4 non-H100 subset and local task subsets are not direct reproductions of upstream leaderboard results.Source: §2 · p. 2

Harness effects

• Prior harness studies report harness-sensitive failures and efficiency differences, while this work crosses four configurable harnesses with five models on three benchmarks and includes vendor-native pairings [32] [23].The cited studies report up to 40-fold differences in tokens per solved task [23] and a lean harness retaining accuracy at lower cost in tested settings.Source: §2 · pp. 2–3

Recovery, verification, and training

• Prior work finds that self-correction without external feedback is unreliable [6] and that agents often fail to verify their outputs [3] [5]; matched trajectories examine related recovery and self-check patterns.Earlier work also reports that harness-aware post-training can improve agents [11], while Polar’s gains for the same model vary from 0.6 points under Qwen Code to 22.6 under Codex [29].Source: §2 · p. 3

3.1 Configurations and task collections

• The configurable grid has 60 entries from four harnesses, five models, and three collections, plus six native references evaluated only with their paired models.The task collections contain intended sets of 120 TUA-Bench, 99 ALE-CLI, and 63 Terminal-Bench 4 tasks, using the same task identifiers across configurations.Source: §3.1 · p. 3

3.2 Harness configuration

• Harnesses use native context management, compaction, retries, turn limits, tools, and other defaults, with minimal changes for common model access and no added skills, memory, or custom prompts.OpenHands receives larger context and output limits than its custom-endpoint defaults; openJiuwen is assembled from its public API because it has no official benchmark entry point.Source: §3.2 · p. 3

3.3 Recorded reward and empirical fit

• Rewards are reported by benchmark using the intended task set; unresolved outcomes receive zero, TUA and ALE allow partial credit, and Terminal-Bench is binary.The three rewards are not combined into an overall score, and empirical fit identifies the evaluated harness with the highest recorded reward for a model and collection.Source: §3.3 · p. 4

3.4 Cost and trajectory analysis

• Costs measure agent-model API spending at OpenRouter prices, including auxiliary harness calls but excluding evaluator calls; trajectory analyses compare final scored runs of the same task and model across harnesses.Reward and cost come from the same last run of each task.Source: §3.4 · p. 4

4.1 Model–Harness Interactions and Native Pairings

• +7.94 points for Claude over GPT in OpenHands on Terminal-Bench 4 reverses to −30.16 points in PI.The same models therefore rank differently across harnesses.Source: §4.1 · pp. 4, 6
• Claude Code leads Claude’s strongest alternative by 3.50 and 1.14 points on TUA-Bench and ALE-CLI, but trails OpenHands by 7.94 on Terminal-Bench.For GPT, openJiuwen exceeds Codex by 2.67 points on TUA-Bench, while PI exceeds Codex by 2.15 on ALE-CLI and 4.76 on Terminal-Bench.Source: §4.1 · p. 6, Table 2 · p. 6

4.2 Task-Dependent Fit

• 64.39%, 54.94%, and 28.57% for Kimi on TUA-Bench, ALE-CLI, and Terminal-Bench; openJiuwen leads alternatives by 5.61, 6.91, and 11.11 points.Four of the five models change their highest-scoring harness across collections.Source: §4.2 · p. 6, Table 3 · p. 6
• 3.11, 3.88, and 6.35 points of mean advantage remain after removing the three largest positive Kimi task gaps.The advantage is not attributable to a single outlier task.Source: §4.2 · p. 6
• 24.83 points over 19 life-science tasks and 11.99 over 12 business/finance tasks favor openJiuwen over PI, while it trails by 2.32 over 18 computing/math tasks.The aggregate advantage is broad across tasks but not universal across domains.Source: §4.2 · p. 6, Figure 2 · p. 7

4.3 Performance and Resource Use

• 60.32% at $4.66 per task for GPT-6 Astra with PI compares with 52.38% at $19.94 with DSH on Terminal-Bench 4.Higher cost does not consistently yield better performance; Figure 1 compares score against log-scale cost for all configurations.Source: Figure 1 · p. 5, §4.3 · p. 8
• 2,560 calls and 110M uncached input tokens for GPT-6 Astra with PI compare with 11,879 calls and 673M under DSH on Terminal-Bench 4.openJiuwen serves 94–99% of input tokens from cache, versus 49–77% for the other harnesses; trajectory length alone is insufficient to judge efficiency.Source: §4.3 · p. 8

5.1 Feedback and Recovery

• 180 of 192 failure responses were model-initiated, but harness feedback often determined whether repair was possible; PI’s unbounded shell left 34 runs blocked without a signal.GPT-6 Astra set explicit timeouts on 56–67% of its PI shell calls and scored best on Terminal-Bench 4 and ALE-CLI under PI; Kimi scored highest under openJiuwen on all three benchmarks.Source: §5.1 · p. 8
• Kimi K3 recovered from the same GIMP hang under openJiuwen after its 300 s timeout returned feedback, but PI’s unbounded shell left the run silent until the deadline.The PI run’s failure is not evidence that Kimi cannot recover; openJiuwen’s run was partly fortunate because the output had already been written.Source: §5.1 · pp. 8–9, Figure 3 · p. 9

5.2 Preserving Progress and Completing Deliverables

• openJiuwen context compression occurred in only 1 of 63 Terminal-Bench tasks and once in a cross-check of 120 TUA tasks; the sole compressed trajectory preserved its deliverable path and resume point.That case compressed 1,450 messages, about 834k tokens, to three messages, about 7.8k tokens.Source: §5.2 · p. 9
• Of 45 failed openJiuwen–Kimi Terminal-Bench tasks, 35 ended with reports claiming all requirements were verified, and three dismissed discrepancies later failed by the grader.The three agents attributed discrepancies to causes outside the deliverable and stopped treating them as unresolved.Source: §5.2 · pp. 9–10, Figure 4 · p. 10
• On retro-console-soc, openJiuwen scored 0.00 and DSH scored 1.00: both reached zero mismatch, but only DSH removed the remaining defect and passed every test.openJiuwen compared its own console model at frame 60 with RTL output at frame 40, then reported pixel-exactness that the grader rejected.Source: Figure 5 · p. 10, §5.2 · pp. 10–11
• No harness exposes the acceptance criterion during a run, so agents construct their own; self-checking cannot substitute for checking against the task’s criterion.On layout-config-recreation, PI with the same model failed the same two tests.Source: §5.2 · p. 10

6 Implications for Data, Training, and Harness Design

• Reward alone is a poor filter for imitation data: the highest-reward CAPTCHA trace included a prompted decision to defeat an anti-automation check, while lower-reward traces contained a hand-back deployment policy may prefer.The authors propose recording the acceptance-test choice and evidence for completion, not only the final artifact.Source: §6 · p. 11
• Acceptance-test choice and tool-argument adherence are training candidates, while bounded timeouts, returned argument errors, and continuation after early stops are runtime fixes.The authors did not train their models on these proposals.Source: §6 · p. 11
• Continuation needs to let the model report when a step requires a person and a harness policy that respects that report, rather than sending only a generic continue message.Source: §6 · p. 11
• The highest-scoring harness changes across collections for four of five models, so the authors recommend selecting and evaluating each model–harness pairing for its benchmark and task.Native provenance does not settle the choice; Codex never records the highest GPT score, while openJiuwen or PI exceeds it on each collection.Source: §6 · pp. 11–12

7 Limitations

• The study evaluates three fixed task subsets with one counted run per task.Run-to-run variance is not measured.Source: §7 · p. 12
• Model settings, tools, and execution budgets are not uniformly matched, so observed differences do not isolate any single component’s causal effect.Source: §7 · p. 12
• Trajectory analyses use few matched pairs, and proposed data and harness implications have not been tested as improvements.Source: §7 · p. 12

8 Conclusion

• Across benchmarks, model rankings, each model’s best harness, and native-pairing standing vary by setting, while openJiuwen with Kimi remains ahead on all three.Source: §8 · p. 12
• Harness choice changes cost as well as score, and trajectories link variation to failure feedback, continuation of interrupted work, and task-completion decisions.Source: §8 · p. 12
• The authors identify which harness components drive fit and automatic configuration selection for unseen tasks as natural next steps.Source: §8 · p. 12

A Run details

• Harnesses use their own defaults, with specified OpenHands and openJiuwen exceptions; ALE-CLI also gives every harness fourteen computer-use tools.Table 5 lists harness versions, native tools, context management, and turn limits; TUA-Bench and Terminal-Bench use Harbor v0.22.0, while ALE-CLI uses its own runner.Source: Table 5 · p. 15, A Run details · p. 15
• Terminal-Bench uses the 63-task non-H100 CPU-only subset of v4.0.0, while ALE-CLI uses 99 local-Docker tasks.Source: A Run details · p. 15
• Infrastructure- or provider-failure tasks are rerun, and each task counts its last run; a run without a verifier result counts as zero.Source: A Run details · p. 15
• On four ALE-CLI image and computer-use tasks, 343 image calls for DeepSeek V4 Pro were served by DeepSeek V4.1 Flash because V4 does not accept images.Source: A Run details · p. 15

B Code and data release

• The code release includes six-harness adapters, the study configuration, and scripts for final-run selection, cost reconciliation, and producing tables and figures.Source: B Code and data release · p. 15
• The dataset contains per-task results and 6,204 scored-run trajectories in a common chat format, with reward, cost, token usage, and agent time.Of these, 6,067 include the full model–tool conversation; 137 carry results only, mostly because the environment never started.Source: B Code and data release · p. 15
• Secrets and host paths are removed, long outputs are truncated, and verifier files and reference data are excluded; the data are not for model training, fine-tuning, or distillation.Source: B Code and data release · p. 15

• Figure 6 plots score against uncached input tokens per task for the four configurable harnesses, with tokens on a log scale.Source: Figure 6 · p. 16
• Figure 7 shows the per-task cost distribution by configuration on a log scale, with box medians and interquartile ranges and whiskers extending to 1.5×IQR.Outliers are not drawn.Source: Figure 7 · p. 17
• Table 6 reports each configuration’s agent-model cost in USD at OpenRouter pricing, summed over the last run of every task; dividing by task counts 120/99/63 gives Figure 1 cost per task.Source: Table 6 · p. 18
• Figure 8 compares per-task model calls, input tokens per call, cache hit rate, output tokens, and agent minutes across configurations.Input includes cached tokens; Codex and Claude Code appear where their logs record tokens.Source: Figure 8 · p. 18
• Figure 9 compares median model calls and agent minutes on solved versus failed tasks, including configurations with at least three tasks in each group.Points above the diagonal spend more on tasks they solve.Source: Figure 9 · p. 19