PaperlayerView exact revision on arXiv

Source-linked AI summary

HarnessMM: Recursive Self-Improvement for Harness Token Efficiency

Yuhua Jiang, Chang Xu, Feifei Gao, Li Zhao

arXiv:2610.32813v1cs.AI

TL;DR

Agent harnesses can repeat reasoning after tool feedback even when an earlier call already resolved the next action. HarnessMM reuses guarded decisions to replace later reasoning callbacks while actor weights remain fixed. On HumanEval and DeepSeek Harness, it reduces token use while preserving or improving success.

  • Problem

    Harnesses repeatedly reason after tool feedback even when prior reasoning has already fixed an action, while feedback-driven harness optimization alone does not determine whether deployment callbacks recur.

    Source: §1 · p. 1

  • Method

    HarnessMM stores resolved actions and literal arguments in guarded decision graphs, reuses them from observed receipts, and substitutes native execution for registered reasoning callbacks, with RSI selecting companion configurations.

    Source: §1 · pp. 1–2

  • Results

    Across HumanEval comparisons, success was preserved or improved as token use fell; for DeepSeek Harness, success rose from 71.43% to 85.71% while Tokens/task (k) fell from 96.28 to 11.62.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The results support guarded decision reuse as a practical way to reduce repeated reasoning in existing harnesses while measuring complete-task quality and cost.

    Source: §1 · p. 2, §7 · p. 8

    Savings depend on resolved actions being available, and small development sets leave policy-level quality differences uncertain.

    Source: §6 · p. 8

Abstract

from arXiv · show

Agent harnesses repeatedly invoke reasoning after tool feedback, even when an earlier call has already fixed the next action. We introduce HarnessMM (Harness Minus Minus), a method for improving harness token efficiency through guarded decision reuse and recursive self-improvement (RSI). RSI iteratively refines companion harness configurations from training feedback, with validation selecting the configuration for deployment. A controller at the model-tool boundary reuses resolved decisions to replace later reasoning callbacks while actor weights remain fixed. Executable Decision Graphs store exact actions, literal arguments, causal dependencies, and public guards. Guarded Decision Reuse activates these persistent decisions from observed receipts; Certified Callback Substitution executes them through the native interface and records removal of the registered slow callback. New semantic decisions, unsupported evidence, and selector abstention invoke the original actor again. The representation restricts what feedback may change, while complete-task measurements establish quality and cost. On HumanEval, HarnessMM preserves Codex success at 90.82\% while reducing Tokens/task (k) from 66.72 to 47.02 (29.54\% saving). For DeepSeek Harness, success rises from 71.43\% to 85.71\% while Tokens/task (k) fall from 96.28 to 11.62 (87.93\% saving). A separate MBPP ablation measures 26.91\% lower cost than single-use graphs and 19.52\% lower cost than retained callbacks. These results establish guarded decision reuse as a practical way to reduce repeated reasoning in existing harnesses.

1 Introduction

HarnessMM addresses repeated reasoning after tool feedback by reusing previously resolved actions when observed receipts make them executable. Its controller combines persistent guarded decisions with callback substitution, while RSI refines companion harness configurations.

1 Introduction

  • HarnessMM studies a narrower reuse contract than reusable programs: episode-specific actions remain literal, while later feedback changes their eligibility for execution.Prior methods establish programmatic execution; HarnessMM focuses on whether a resolved action can replace a later reasoning callback.Source: §1 · p. 1
  • The Substitution Paradox names the accounting mismatch in which computation is saved only if eliminated reasoning exceeds added authorship, selection, and recovery.Repeated deliberation can regenerate decisions whose semantics have not changed, such as a test command already fixed before a successful compilation receipt arrives.Source: §1 · p. 1
  • HarnessMM stores resolved actions and literal arguments in Executable Decision Graphs, then uses guarded reuse and certified native execution to replace redundant reasoning callbacks.New semantic decisions, unsupported evidence, and selector abstention return control to the original actor; complete-task accounting includes authorship, selection, fallback, and unsuccessful executions.Source: §1 · pp. 1–2
  • Recursive self-improvement iteratively refines companion harness configurations from training feedback, with validation selecting the configuration frozen for test evaluation.The overview depicts RSI updates above the guarded-reuse and callback-substitution edits; its graphs are schematic.Source: §1 · p. 2, Figure 1 · p. 2
  • Codex success remains 90.82% on 98 HumanEval tasks as Tokens/task (k) falls from 66.72 to 47.02; DeepSeek Harness success rises from 71.43% to 85.71% as Tokens/task (k) falls from 96.28 to 11.62.The MBPP ablation also measures 26.91% lower cost than single-use graphs and 19.52% lower cost than retained callbacks.Source: §1 · p. 2

2 Related Work

HarnessMM is distinguished from related efficiency approaches by changing whether a reasoning callback recurs, rather than only reducing call size, depth, or providing reusable guidance.

2 Related Work

  • Prompt compression, context management, and reasoning budgets reduce input replay or generated reasoning, whereas HarnessMM targets whether a call needs to recur.Examples include LLMLingua, AgentDiet, observation masking, and thinking budgets.Source: §2 · p. 2
  • Fast–slow architectures and routing change who reasons or how deeply, while HarnessMM asks whether compact selection actually removes a future reasoning callback.Its Jev-motivated typed selector returns a candidate ID or abstains, with single-token decoding when a model call is needed.Source: §2 · p. 3
  • Skills and memory can inform later reasoning or encapsulate routines, but EDGs store episode-specific resolved commitments activated by public execution conditions.This distinction allows execution to replace a reasoning call rather than merely guide it.Source: §2 · p. 3
  • Prompt, program, and agent-configuration optimization constructs companion harness configurations, while HarnessMM changes when the harness calls its reasoning model and directly executes resolved actions.Instruction optimization, context compaction, and RSI remain outside the core execution method and retain separate experimental attribution.Source: §2 · p. 3

3 Problem Formulation

The formulation treats HarnessMM as a policy that substitutes stored actions for reasoning nodes in a fixed harness, subject to both computation-cost and capability constraints.

3.1 Reasoning calls and tool actions in a harness

  • For each task, the harness execution is unrolled into a computation graph whose reasoning, retrieval, routing, tool, verification, and recovery nodes have token costs.Edges encode execution dependencies, and expected policy cost sums costs for invoked nodes.Source: §3.1 · p. 3
  • The fast–slow policy changes which reasoning nodes recur after public feedback, while deterministic tools affect later calls and quality without using actor tokens.Public state includes the request, observed tool feedback, and completed action history.Source: §3.1 · p. 3

3.2 Capability-constrained computation substitution

  • Capability regret compares a policy’s benchmark score with the original harness score, with a declared tolerance defining the permitted quality difference.The formulation defines regret as Q_b(h0) − Q_b(hπ).Source: §3.2 · p. 3
  • The capability constraint requires each declared quality endpoint to remain at least the original score minus its fixed tolerance, assessed on the same full task set.Zero-regret protocols set the tolerance to zero and include unsuccessful tasks in evaluation.Source: §3.2 · p. 3

3.3 The Substitution Paradox

  • A fast stage that adds computation without removing downstream deliberation increases path cost rather than substituting for the original continuation.Short output or frequent fast-path activation alone does not establish callback elimination.Source: §3.3 · p. 3
  • Beneficial substitution requires removed computation to exceed introduced computation, including authorship, routing, added context, fallback, and recovery.The cost condition is separate from the capability constraint.Source: §3.3 · p. 3

4 HarnessMM: Reusing Decisions

HarnessMM inserts a controller at an existing harness’s reasoning boundary to reuse resolved actions when public receipts satisfy their guards, and to substitute native execution for redundant callbacks.

4 HarnessMM: Reusing Decisions

  • The controller ingests public receipts, updates action eligibility, and either executes a stored action or invokes the slow actor, which can author another graph.Prompt, context, and finalization settings configure the surrounding harness and are held fixed during each run.Source: §4 · pp. 3–4

4.1 Executable Decision Graphs (EDG)

  • EDGs store resolved native actions, complete literal arguments, predecessors, and public execution guards in a bounded continuation.Guards are fixed with each node and evaluated against later receipts; dependencies form an acyclic graph, and missing or rejecting receipts prevent admission.Source: §4.1 · p. 4

4.2 Guarded Decision Reuse (GDR)

  • GDR retains the graph across tool transitions and admits a decision only when dependencies and guards are satisfied without new semantic inference.Unsupported observations, unmet dependencies, or arguments requiring new inference defer to the original actor; a passing guard establishes its declared public condition, not general task correctness.Source: §4.2 · p. 4

4.3 Certified Callback Substitution (CCS)

  • CCS dispatches an admitted action through the original tool interface and binds its receipt to the omitted registered callback.The certificate identifies the graph and decision, predecessor completion, exact arguments, tool result, and bypassed callback; the audit verifies native execution and absence of that request.Source: §4.3 · p. 4

4.4 Complete-Task Accounting

  • Complete-task accounting charges authorship, selection, added context, remaining slow calls, fallback, recovery, finalization, and failed paid executions.Shared authoring is charged once to its declared cohort; token savings alone do not establish latency or energy savings.Source: §4.4 · pp. 4–5

5 Experiments

The experiments assess HarnessMM’s integrated harness effects, companion configurations, execution mechanisms, and training-guided RSI. Across the coding-harness comparison, token costs fall while task success is preserved or improves; ablations and RSI results further test persistent reuse and callback substitution.

5.1 Evaluation protocol

  • The evaluation compares methods on the same tasks, backbone, tools, limits, and scorer, with complete-task input-plus-output token costs and separate development optimization costs.For coding comparisons, TRAIN supplies revisions, validation selects frozen candidates, and TEST measures the selected configuration.Source: §5.1 · p. 5

5.2 Does fast execution substitute slow reasoning?

  • On MBPP-sanitized, DeepSeek Harness reduced paid calls from 1,633 to 1,088 with 472 certified bypasses, while success rose from 90.27% to 91.83%.The original harnesses have zero certified bypasses; the reported rates use each method’s full task set.Source: Table 1 · p. 5, §5.2 · p. 5

5.4 Companion configurations: quality and cost

  • Figure 2 reports benchmark-specific companion configurations across ALFWorld, SearchQA, LiveMath, AIME, Terminal-Bench 2, and GAIA2; the MBPP ablation directly tests EDG persistence and callback removal.The companion studies measure their stated configurations rather than directly isolating EDG persistence and callback removal.Source: Figure 2 · p. 6, §5.4 · p. 6
  • On SearchQA validation, HarnessMM reaches 76.76% EM at 0.99 Tokens/task (k), saving 19.08% against No Skill; on AIME, it reaches 87.14% accuracy at 5.90 Tokens/task (k).On GAIA2, HarnessMM reaches 43.96% success with 639.64 Tokens/task (k), saving 12.41%, 17.00%, and 17.17% versus No Skill, SkillOpt, and AutoSaddler, respectively.Source: §5.4 · p. 6

5.3 Industrial harness results

  • On HumanEval, HarnessMM preserves Codex success at 90.82% while reducing Tokens/task (k) from 66.72 to 47.02 (29.54% saving); DeepSeek Harness success rises from 71.43% to 85.71% as cost falls from 96.28 to 11.62 (87.93% saving).These are complete 98-task comparisons; integrated effects combine literal-bank reuse, context management, and finalization.Source: §5.3 · pp. 5–6, Figure 3 · p. 7

5.4 Companion configurations: quality and cost

  • On ALFWorld, frozen HarnessMM instructions reach 66.20% success at 31.12 Tokens/task (k), versus SkillOpt’s 63.29% at 101.33.The paired success difference is 2.91 points, with 69.29% fewer model tokens; the family-only instruction condition is cheaper but loses 3.69 success points.Source: §5.4 · pp. 7–8

5.6 HarnessMM harness RSI

  • By the eighth MiniWoB++ RSI iteration, both arms solve 29 of 32 tasks, while persistence reduces Tokens/task (k) from 1.51 to 1.06 (29.83%).Across the first eight iterations, HarnessMM success rises from 34.38% to 90.63%, while cost falls from 3.26 to 1.06 Tokens/task (k).Source: Figure 5 · p. 8, §5.6 · p. 8

5.5 GDR and CCS ablation on MBPP

  • On MBPP-sanitized, full HarnessMM reaches 93.77% success at 13.17 Tokens/task (k), versus 93.00% and 18.03 for single-use EDGs, reducing complete cost by 26.91%.Against retained callbacks, full HarnessMM measures a 19.52% cost reduction; disabling EDG execution uses 18.62% fewer tokens but solves four fewer tasks.Source: §5.5 · p. 8

6 Limitations

HarnessMM’s savings and quality comparisons have boundaries tied to accounting assumptions, resolved-action availability, and the size of development data.

6 Limitations

  • Savings depend on provider input, cache, and context accounting, as well as the availability of resolved actions.New semantic decisions still require slow reasoning.Source: §6 · p. 8
  • Authorship, selection, and recovery can outweigh the reasoning calls avoided by reuse.The limitation passage does not assert that these costs always outweigh avoided calls.Source: §6 · p. 8
  • Public-state adapters are task specific, and small development sets leave policy-level quality differences uncertain.Source: §6 · p. 8

7 Conclusion

HarnessMM stores resolved decisions, checks when they can execute, and uses them to replace repeated reasoning calls. Its conclusion reports lower actor-token costs with preserved or improved coding success, while the RSI appendix describes training-task measurements across iterations.

7 Conclusion

  • On all 98 HumanEval test tasks, HarnessMM preserves Codex success at 90.82% with 29.54% fewer actor tokens and improves DeepSeek Harness success from 71.43% to 85.71% with 87.93% fewer tokens.Source: §7 · p. 8

A HarnessMM visual harness RSI study

B Implementation and selection

This section describes HarnessMM’s selection and execution of reusable decisions while preserving the existing actor and native interface, alongside its rule-development protocol, QA and coding adapters, and limits on execution evidence. It also states the shared comparison objective: reduce complete-task input-plus-output tokens while preserving task performance.

B.1 QA profiles and execution

  • QA routing selects frozen DELETE or MODIFY profiles using input-only features, while the full-source adapter accepts checked drafts unchanged or falls back to the selected profile.Explicit negative/comparison questions and multiple blanks bypass drafting; draft and fallback tokens are charged. The adapter checks format, source location, and answer type, and may change request and response lengths.Source: §B.1 · p. 12, §C.5 · p. 15

B.2 Recorded rule-development protocol

  • RSI retains a child only after it changes the harness, preserves validation EM and F1, uses at most 90% of Original’s inference tokens, and improves a parent endpoint.Training feedback guides edits, while disjoint validation selects candidates; proposals cannot use protected validation examples or answers. Restricted MetaRSI adds strategy and child-comparison requirements, and frozen releases face separate final-screen thresholds.Source: §B.2 · pp. 12–13
  • Original, fixed-policy RSI, restricted MetaRSI and Original A/A retained identical final request bytes, each solving 78.91% of tasks (78.91 EM, 86.59 F1) at 3.02 Tokens/task (k).The two valid three-rule edits added 0.06 input Tokens/task (k) without improving quality or output usage; six other proposals failed the literal-overlap guard.Source: §B.2 · p. 13

C.3 Shared commit-or-defer control

  • The adapter exposes supported candidates to a shared controller, which commits an accepted decision once or invokes the original actor when support is absent or rejected.The task adapter builds public state and task-specific support predicates; actor parameters, tools, and evaluator remain fixed. Offline RSI uses training feedback, disjoint validation, and a frozen deployment configuration, while paired executions account for run-to-run variability under the same prescribed schedule.Source: §C.1 · p. 13, §C.2 · p. 14, §C.3 · p. 14

C.4 Action adapter: CoT-authored conditional execution

  • The coding adapter stores an acyclic graph of literal tool arguments, dependencies, and public-result guards; eligible actions execute through the native interface while unsupported states defer to slow reasoning.The selector chooses a declared candidate or abstains, cannot revise arguments, and all authorship, selection, failures, and fallbacks count toward complete-task cost. The historical MBPP training comparison used v23, while the selected test comparison used separately frozen v66; the Codex v10 candidate did not pass the broader adoption gate.Source: C Executable continuation… · p. 13, §C.4 · p. 14

D.2 Closure, guards and structural invariants

  • The execution contract reuses episode-specific authored actions and records omission of a registered callback, but literal closure alone does not establish semantic correctness or task-quality improvement.Eligibility follows completed dependencies and passing public guards; certificates bind the graph, arguments, receipts, and callback, while the request ledger verifies that the slow request was absent. Full-task comparisons, not certificates, measure quality and cost effects.Source: D Execution contract, algorithm and… · p. 15, §D.1 · p. 15, §D.2 · p. 16

D.3 Break-even analysis without counterfactual token attribution

  • Token savings are measured from total episode tokens, including paid author, selector, fallback, and unsuccessful calls, rather than assigning counterfactual costs to omitted callbacks.A same-trajectory break-even relation is presented only as a condition, not a prediction, because omitted calls are not executed and substituted trajectories can differ.Source: §D.3 · p. 16

D.4 Novelty and closest mechanisms

  • HarnessMM’s episode-specific literal-action reuse differs from related executable planners, and the paper does not experimentally compare those systems or claim superiority over them.The surrounding configuration fixes prompts, context serialization, adapters, and finalization before a run; MBPP ablations vary graph reuse and callback execution while holding those settings fixed.Source: §D.4 · p. 16, §D.5 · p. 17

D.5 Core method and companion configuration map

  • All comparison arms received the token-efficiency instruction, while No Skill omitted the added skill policy.Method-specific mechanisms and native execution limits followed the respective configurations.Source: §D.5 · p. 17

D.6 Which experiment supports which claim?

This section distinguishes experiments that measure complete harness outcomes from studies of specific instruction or execution interventions. Historical development results, complete quality–cost comparisons, and reproducibility boundaries are reported separately.

D.6 Which experiment supports which claim?

  • The experiment map separates complete harness comparisons, callback-omission evidence, execution ablations, and instruction-content studies; the MBPP ablation does not isolate certificate logging from exclusive routing.Certificate logging overhead is included where it makes model calls.Source: §D.6 · p. 17

D.7 Historical development evidence

  • Historical development results report benchmark-specific quality and token-cost changes, while instruction ablations compare content under a common actor and fixed 50-step cap.The historical MBPP and GAIA2 comparisons use development populations; the instruction ablation reports gains against both w/o-skills and neutral-text conditions.Source: §D.7 · p. 18

D.8 Complete quality–cost trade-offs

  • On 18 Terminal-Bench development tasks, success fell for both harnesses as token cost fell; the CoT/Jev controls also cost more than their stated baselines.The CoT/Jev TEST comparison cost 127.07% more than the original harness and lost 7.29 success points; input tokens were 98.53% of its actor ledger.Source: §D.8 · p. 18

D.9 Reproducibility and use scope

  • Reproduction requires a declared native boundary and complete request logging; the controller preserves existing permissions but does not provide a general safety guarantee.Incorrect authorship, incomplete guards, and changed tool semantics can affect task quality despite a valid structural certificate.Source: §D.9 · p. 18

E Second ablation: GDR and CCS

The MBPP-sanitized ablation compares graph reuse and callback substitution with complete task costs. Full HarnessMM is cheaper than both single-use graphs and retained callbacks, while paired uncertainty qualifies the observed correctness differences.

E.1.1 Benchmark and evaluation settings

  • The study uses a custom 85/85/257 training/validation/test partition, with disjoint tasks, shared execution settings, and no test-driven revisions or selection.All arms use the same actor backbone, tools, decoding settings, one attempt per task, and fixed deadlines.Source: E Second ablation: GDR and CCS · p. 18, §E.1.1 · p. 19

E.1.2 Execution interventions

  • Full HarnessMM combines graph retention across tool transitions with certified callback substitution; the controls either discard a graph after one use or retain paid slow callbacks.The intervention tests callback substitution, not the independent benefit of certification.Source: E Second ablation: GDR and CCS · p. 18

E Second ablation: GDR and CCS

  • Full HarnessMM records 407 certified bypasses, including 201 graphs reused twice; the retained-callback condition records 257 paid regenerations and zero certified bypasses.The single-use condition records 212 bypasses and discards 212 graphs.Source: E Second ablation: GDR and CCS · p. 18
  • 26.91% lower measured complete cost than single-use graphs and 19.52% lower cost than retained callbacks accompany two and one additional solved tasks, respectively.Reuse cost intervals exclude zero, while all three correctness intervals include zero.Source: Table 3 · p. 19, E Second ablation: GDR and CCS · p. 19, Table 4 · p. 19

E.1.3 Complete results and paired uncertainty

  • On the earlier 257-task execution ablation, HarnessMM reduces cost by 76.19% versus the original harness, but its 2.72-point correctness increase is not significant by the paired interval.Recorded slow calls per task were 4.49 for full HarnessMM, 5.42 for single-use GDR, and 6.41 with callbacks retained; the single-use condition had slightly higher correctness and lower aggregate cost in this comparison.Source: §E.1.3 · pp. 19–20, Table 5 · p. 20, Table 6 · p. 20

F.1 HumanEval results

The section reports benchmark and ablation outcomes alongside limits on study scope, substitution attribution, and related adapters. Its results distinguish complete-task performance from locally verified callback removal.

F.1 HumanEval results

  • 18 and 147 valid substitution certificates were recorded for Codex+HarnessMM and DeepSeek Harness+HarnessMM, versus zero for the original harnesses; complete cost was 11.62 Tokens/task (k).Certificates bind literal arguments, a tool, receipt hash, predecessor, and bypassed callback; complete cost covers the whole cohort, not just certified substitutions.Source: §F.1 · p. 20
  • On MBPP-sanitized, DeepSeek Harness and HarnessMM achieved 232/257 and 236/257 successes, respectively, with complete costs of 53.98 and 10.61 Tokens/task (k).The benchmark populations are reported separately.Source: §F.1 · p. 20

F.2 Additional study scope

  • Local decisions require a task-specific public-state adapter and support rules; textual ALFWorld observations do not establish visual-agent improvement.The custom test uses one seed and shares some floorplans across partitions.Source: §F.2 · p. 20

F.3 TRAIN substitution attribution

  • 79.68% more tokens and 6.25 fewer success points accompanied the TRAIN v20 arm despite fast actions in 93.75% of episodes.The arm made 618 slow calls against 502 baseline attempts; fast requests alone would save only 1.18% of complete cost.Source: §F.3 · p. 21
  • The v22 revision reduced slow calls from 15.69 to 12.63 per episode and saved 15.55% at 43.75% versus 40.62% baseline success.Renewing banks at every slow step in v23 raised cost by 20.20% and lost one success.Source: §F.3 · p. 21

F.4 Additional related-work and adapter context

  • The text-QA adapter checks an actor-generated draft’s source, type, and format, returning it or calling the original answering procedure; both paths are charged.Unlike the action adapter, it changes request and response lengths.Source: §F.4 · p. 21

G.1 Fresh ALFWorld instruction comparison and skill removal

  • Adding frozen task-family skills yielded 603 paired gains and 93 losses, a net 23.91-point difference, while reducing complete-task model tokens by 8.60%.The comparison is secondary and descriptive, not the archived campaign’s predeclared primary contrast.Source: §G.1 · p. 22

G.2 Full SearchQA comparison

  • HarnessMM added 18 exact matches to original HarnessMM in SearchQA while saving 22.75% of tokens, yet remained more expensive than No Skill and AutoSaddler.The complete study contains 118,860 outcomes and 119,768 actor calls.Source: §G.2 · p. 22

G.3 Cross-dataset QA transfer

Across the additional studies, outcomes vary by benchmark and protocol: QA transfer, AIME, and coding comparisons report distinct quality–cost patterns. Scope and evaluation boundaries matter, including incomplete or development-only populations.

G.3 Cross-dataset QA transfer

  • Recovery added 0.05–0.15 EM points over the no-recovery draft at higher cost; HarnessMM exceeded three external prompt controls in EM and F1, while No Skill and AutoSaddler used fewer tokens.These are single-seed point estimates; SearchQA gained 14 exact matches over original HarnessMM but lost 0.004 F1 points.Source: §G.3 · p. 23

G.4 AIME and the remaining quality–cost panels

  • The unchanged AIME domain incumbent scored 77.78% accuracy at 6.06 Tokens/task (k), compared with 61.11%, 66.67%, and 72.22% for the three controls.The archived constraint-ledger HarnessMM matched both optimized baselines at 72.22%, without a cost advantage over SkillOpt.Source: §G.4 · p. 24
  • The Terminal-Bench snapshot excludes unresolved verdicts from success denominators and does not provide a full-task ranking or complete official mean.Its plotted token values divide known totals by finished trials, not scored trials.Source: §G.4 · pp. 24–25
  • The registered GAIA2 four-method study has no complete test comparison; its ten scenarios with three repetitions are prior development results, and both custom repartitions use previously exposed public data.The development set is not substituted for the incomplete test comparison.Source: §G.4 · p. 25

G.5 Success-first coding search

  • The selected coding instruction gained four successes and used 39.68% fewer actor tokens than the corrected original harness, and gained two successes with 29.25% fewer than built-in compaction.It retained 14 training successes; the frozen instruction was to be evaluated separately on validation and TEST tasks.Source: §G.5 · p. 25
  • The 32Ki compaction candidate used 1068.67 actor Tokens/task (k), 55.72% fewer than the original harness, but solved 61.11% and lost three successes versus the instruction, so the quality guard rejected it.The 98,304-token successor also failed the 77.78% quality requirement.Source: §G.5 · p. 25

G.6 Qwen SDK original coding harness

  • The Qwen SDK parent solved 5.56% of the complete Terminal-Bench 2.0 TRAIN population at 4684.42 Tokens/task (k); validation and TEST remained sealed because it fell below nine required successes.Public TRAIN tool records included 11 host-directory errors and 70 repetitions of identical commands.Source: §G.6 · p. 26

G.7 Original HTTP coding harness

  • The DeepSeek-flash original HTTP coding harness solved 55.56% of the 18 Terminal-Bench 2.0 training tasks at 899.99 Tokens/task (k).Unknown historical request usage was excluded.Source: §G.7 · p. 26

H Industrial-harness RSI comparison

The industrial-harness studies evaluate recursive configuration search and guarded action execution under frozen policies and complete comparisons. The reported outcomes include rejected searches and negative or cost-increasing results, not only favorable efficiency cases.

H Industrial-harness RSI comparison

  • The grouped cross-fitted router admits only candidates with positive supported advantage; protected validation selects among admitted candidates and the seed before policies freeze for testing.Validation failures and test outcomes do not enter the proposer.Source: H Industrial-harness RSI comparison · p. 26
  • Both RSI searches rejected their two candidates and retained the seed; unchanged-harness tests recorded 10.42% versus 9.03% success for Codex and 8.68% versus 7.64% for DeepSeek.Paired differences were −1.39 and −1.04 points, with intervals spanning zero.Source: H Industrial-harness RSI comparison · p. 26
  • In the original-app TEST comparison, CoT/Jev achieved 49.65% success at 1874.98 actor Tokens/task (k), versus 56.94% and 825.73 for the original harness.CoT-only achieved 48.61% success at 1894.74 Tokens/task (k); all three arms include failures and fallbacks.Source: H Industrial-harness RSI comparison · p. 27
  • The CoT/Jev TEST arm executed 2,559 verified fast tool actions but achieved 49.65% success at 1,874.98 Tokens/task (k), below the original harness’s success and above its cost.The corresponding original-harness values were 56.94% and 825.73 Tokens/task (k).Source: H Industrial-harness RSI comparison · p. 27
  • Retained Codex selection scored 52.78% success at 708.73 actor Tokens/task (k), versus 52.43% and 648.71 for the original harness; this does not establish a HarnessMM efficiency gain.Both arms applied zero residual edits.Source: H Industrial-harness RSI comparison · p. 27
  • On the 18 DeepSeek coding TRAIN tasks, CoT/Jev raised success to 72.22% but used 30.37% more tokens than the original harness and 3.74% more than CoT-only.The 40% saving target remained unmet, and the archived protocol lacks a complete paired TEST result.Source: H Industrial-harness RSI comparison · p. 27
  • The complete TRAIN CoT/Jev arm executed 106 verified fast actions yet lowered success to 31.25% and raised cost to 1044.05 Tokens/task (k), versus 40.62% and 891.75 for the parent.The study proceeds to validation and TEST even after a negative training efficiency result.Source: H Industrial-harness RSI comparison · p. 28

I Exploratory quality–cost study

  • The exploratory quality–cost study uses frozen 20/20/60 partitions grouped before repetition, with training for candidate development and validation for checking frozen candidates.Its stated populations and cost definitions remain separate.Source: I Exploratory quality–cost study · p. 28

I.1 Six-benchmark Figure 2 protocols

The protocols distinguish development, validation, and test evidence across benchmark-specific populations. Results include complete harness comparisons, cost diagnostics, and follow-up evaluations, with validation selecting frozen candidates before protected testing.

I.1 Six-benchmark Figure 2 protocols

  • HarnessMM’s complete AIME actor cost was 5.90 Tokens/task (k), including 1.08 for shared authorship, versus No Skill’s 4.60 total.HarnessMM’s execution component alone was 4.82 Tokens/task (k), above No Skill’s 4.59; Table 15 separates input, output, and shared authorship.Source: §I.1 · p. 29, Table 15 · p. 29
  • 12.41% lower complete actor cost than No Skill was measured for HarnessMM on GAIA2; 146 of 10,424 native action attempts were verified fast executions.The comparison measures the integrated harness and does not isolate fast execution from context compression.Source: §I.1 · p. 30, Table 16 · p. 30
  • 76.23% EM was the best exact-union SearchQA TRAIN result, saving 61.82% of incumbent tokens but leaving a 0.825-point EM gap.The bootstrap EM-difference interval was [−1.89, +0.21] points; this family did not advance to validation or TEST.Source: §I.1 · pp. 30–31
  • 100.00% AIME accuracy was preserved by eliminate-then-evaluate at 18.63% higher cost, while the other two candidates lost one answer and cost more than No Skill.The answer-aware seven-policy oracle used 5.52 Tokens/task (k) against the 5.30 Tokens/task (k) target.Source: §I.1 · p. 31
  • 17.02% fewer actor tokens were reported for exact ledger references across 24 GAIA2 paired outcomes; the selected incumbent saved 11.18% of actor tokens on validation.Token-wire v3 and lazy schemas lost quality, while the validation diagnostics selected neither a method revision nor a paper claim.Source: §I.1 · pp. 31–32

I.2 Latest complete TRAIN follow-ups

  • 77.53% EM at 0.97 Tokens/task (k) was reported for SearchQA risk-220 on all 3,395 TRAIN questions, versus 77.05% at 2.29 for the incumbent.The observed EM difference was +0.471 points, with paired 95% interval [−0.471, 1.443].Source: §I.2 · p. 32
  • 76.76% EM at 0.99 Tokens/task (k) was reported for SearchQA risk-220 on validation, but its 19.08% saving missed the frozen 20% release threshold.Validation also showed a 6.951-point EM gain over No Skill; TEST remained sealed.Source: §I.2 · p. 33, Table 21 · p. 33

I.3 Six-condition instruction ablation

  • The instruction ablation compared four paragraph-presence conditions across complete 2,133-task rows, with only complete six-condition seeds reported.The underlying public task pool had prior exposure; replication seeds reuse task identities and are not independent task populations.Source: §I.3 · p. 33, Table 22 · p. 34, Table 23 · p. 34

J.1 Separating task rules from model calls

The follow-up analyses separate handwritten task rules, controller behavior, compression, and instruction effects from claims about HarnessMM optimization. They report mixed transfer results and document boundaries on evaluation and reproducibility.

J.1 Separating task rules from model calls

  • 92.39% of validation tasks were solved by the unchanged handwritten rules without skill text, actor initialization, or model fallback, versus 92.82% for rules plus actor.The three gains and zero losses yielded a 0.42-point difference at 1.08 additional Tokens/task (k); the comparison does not isolate learned instructions from initialization and fallback.Source: §J.1 · p. 34, Table 24 · p. 34

J.2 Archived action-controller comparisons

  • 100.00% success at each seed was achieved by the support-first handwritten controller on 134 unseen tasks, versus 67.16%, 67.91%, and 69.40% for its skills-only actor.These predecessor-controller interventions are separate from the instruction-only ablation; the 134 tasks, not 402 repeated executions, are the independent units.Source: §J.2 · p. 35
  • 95.08% success at 1.00 Tokens/task (k) was achieved by support-first execution on the custom test set, versus 92.55% at 9.91 for stall gating.The paired difference was 2.53 points and the token saving was 89.89%; both used the same instructions, selector, and 50-step cap.Source: §J.2 · p. 35

J.4 Compression and transfer failures

  • 70.54 EM at 1.90 Tokens/task (k) was reported for the 9B compact-context SearchQA adapter, versus 75.46 at 3.00 for fresh original HarnessMM.The adapter used 5,000 opened training-source questions; full source context was retained in later evaluations.Source: §J.4 · p. 35

J.5 Instruction-content and visual diagnostics

  • 84.82% success was reported for the full five-module response harness, versus 84.44% at 23.91% lower cost for the selected normalization/answer-type mask.No module passed the primary familywise criterion, and no average main effect or pair interaction passed the Holm-corrected secondary tests.Source: §J.5 · p. 36
  • 17 of 19 outcome-selected EM effects retained a nonzero sign in every block, but the effects were conditional and 13 nominated modules concerned normalization.The targeted replication retained the 19 tasks, two F1-only cases, and 32 stable controls; stable controls had zero EM edge effects.Source: §J.5 · p. 36

K Reproducibility

  • Startup PYTHONHASHSEED was unrecorded in 17 audited GAIA2 worker launches, while salted hashing, UUIDs, and simulation timing remained variable.A fixed model-server seed therefore does not establish identical tool trajectories.Source: K Reproducibility · p. 36

L Figure 3 evaluation settings

Figure 3 compares harness pairs under benchmark-specific populations, shared backbones, and frozen selection before testing. The reported results span HumanEval and MBPP, alongside industrial-harness comparisons.

L Figure 3 evaluation settings

  • Figure 3’s GAIA2 and Terminal-Bench panels use complete development populations, while MBPP and HumanEval panels use specified TRAIN/validation/TEST partitions.Candidates are selected on validation and frozen before TEST; the HumanEval comparison uses 98 test tasks and one attempt per task.Source: L Figure 3 evaluation settings · pp. 36–37
  • 90.82% Codex success was preserved on all 98 HumanEval test tasks while Tokens/task (k) fell from 66.72 to 47.02, a 29.54% reduction.DeepSeek Harness success rose from 71.43% to 85.71% while Tokens/task (k) fell from 96.28 to 11.62, an 87.93% reduction.Source: L Figure 3 evaluation settings · p. 37

L.1 Industrial harness comparisons

  • 97.28% success at 11.00 Tokens/task (k) was reported for the selected MiMo Codex successor on 257 MBPP test tasks, versus 94.55% at 10.45 for the earlier candidate.The same native task population and limits were used.Source: §L.1 · p. 37

L.2 Qwen DeepSeek Harness on 257 MBPP test tasks

  • 90.27% baseline success versus 91.83% HarnessMM success was reported on 257 MBPP test tasks, with total Tokens/task (k) of 53.98 versus 10.61.HarnessMM made 472 certified substitutions; CoT-only reached 93.39% success at 15.04 Tokens/task (k).Source: §L.2 · p. 37
Loading 2610.32813v1…