PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

Jens Lehmann, Sahar Vahdati

arXiv:2610.30499v1cs.AIcs.SE

TL;DR

This paper asks whether frontier models can build and improve the software harness that shapes their actions, rather than relying on human redesign. It studies autonomous harness construction and two revisions using ARC-AGI-3, then finds that development gains do not reliably translate into held-out gains.

  • Problem

    It is unclear whether a model can use execution experience to improve the software through which it acts, rather than relying on human engineers to redesign it.

    Source: §1 · pp. 1–2

  • Method

    Models build a harness with a fixed coding bootstrap, revise it using five development games, and evaluate snapshots on twenty held-out games while model weights remain fixed.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Across two revisions, mean held-out RHAE rises by 2.80 points (95% bootstrap interval [−1.44, 7.35]), while development improvement exceeds held-out improvement.

    Source: §5 · p. 6

  • Takeaways & Limitations

    Harness construction and reliable improvement are distinct capabilities; development gains and lower inference costs can coexist with held-out regressions.

    Source: §1 · p. 2, §9 · p. 9

    The study uses one benchmark, one set of feedback games, and two revisions, leaving other tasks and longer improvement sequences untested.

    Source: §8 · p. 9

Abstract

from arXiv · show

A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language models rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.

1 INTRODUCTION

The harness is the software layer that shapes how a model observes, acts, recovers, and spends inference, making its design part of an agent’s capabilities. This study asks whether frontier models can construct and revise that layer from execution experience, then tests whether those revisions improve held-out performance.

1 INTRODUCTION

  • A harness orchestrates model calls and shapes perception, memory, available actions, recovery, and inference expenditure.System behavior depends on both the harness and its underlying model or models.Source: §1 · p. 1
  • Harness changes can add tools, persistent memory, and coordination across model calls—capabilities that weight changes alone do not supply.The study frames this as a route for models to take on some engineering work that determines their capabilities.Source: §1 · p. 2
  • ARC-AGI-3 is a revealing test because human-steered harnesses have reached 100 RHAE on its public games, showing effective designs are known.The paper contrasts those results with lower scores from a simple benchmark harness and an off-the-shelf baseline.Source: §1 · p. 2
  • The study examines whether a model can build a complete interaction harness, learn from execution, and revise it without human edits.Models use a fixed coding bootstrap, five development games, and up to two revisions; snapshots are evaluated on twenty holdout games.Source: §1 · p. 2
  • Mean development improvement exceeded mean held-out improvement by 5.62 RHAE points, and cost reductions could accompany performance losses.The authors conclude that building a capable agent and improving it reliably are distinct abilities.Source: §1 · p. 2
  • The study contributes a framework for autonomous harness adaptation, a replicated all-snapshot protocol, and an outcome-blind analysis of revision episodes.It defines the harness adaptation gap as the difference between development-task improvement and held-out-task improvement.Source: §1 · p. 2

2 HARNESS ADAPTATION AS A CAPABILITY

The paper treats harness adaptation as changes to executable software while model weights remain fixed, with the deployed model–harness system as the unit of capability. It distinguishes constructing and revising a harness from selecting updates that improve held-out tasks, and evaluates performance alongside inference costs.

2 HARNESS ADAPTATION AS A CAPABILITY

  • The deployed agent is S_t = (M, H_t): a pretrained model paired with an executable harness, with M held fixed as the harness changes.The complete system, rather than the model alone, is the unit of capability measurement.Source: §2 · p. 2
  • The framework separates harness construction, interface and lifecycle repair, evidence-to-update mapping, and selecting updates that improve feedback-held-out tasks.The first three capabilities do not imply the fourth; a capable updater should also preserve working behavior and abstain when evidence is weak.Source: §2 · p. 3
  • A positive adaptation gap means improvement on feedback-generating tasks exceeds held-out benefit.In this ARC-AGI-3 study, mean RHAE over five development games and twenty held-out games estimates those quantities.Source: §2 · p. 3
  • The study records dollar-denominated gameplay and harness-development inference costs, and compares held-out performance with per-game deployment cost.Construction and revision expenditure is reported separately from deployment cost.Source: §2 · p. 3

3 RELATED WORK

Prior work automates self-refinement, prompt and program optimization, and agent or harness design, but often retains a human-specified artifact schema or computation graph. This study examines autonomous construction and revision of interactive-game harnesses, tracking both held-out generalization and cost.

3 RELATED WORK

  • Self-feedback and program-optimization methods revise responses, policies, prompts, or modules, but normally retain a human-specified artifact schema or computation graph.Examples include Self-Refine, Reflexion, CRITIC, OPRO, PromptBreeder, DSPy, MIPRO, TextGrad, and GEPA.Source: §3 · p. 3
  • Prior agent-design work searches optimizer construction, code-defined agents, workflows, runtime logic, and complete coding-agent repositories.Meta-Harness and Agentic Harness Engineering optimize executable harness source; HarnessCompass and RRSI study generalization beyond evolution tasks.Source: §3 · p. 4
  • Earlier evaluations report that successful harness updates or validation-selected evolution need not translate into test gains.SEAGym reports test success falling 14.5 percentage points despite completing 39 of 40 updates; other work reports small held-out gains or optimistic validation scores.Source: §3 · p. 4
  • ARC-AGI-3 systems demonstrate strong solving within human-steered harnesses, while this study shifts construction and revision of the reusable harness to the model.Workspace Optimization instead revises executable world models and strategies during play while keeping the reusable harness fixed.Source: §3 · p. 4

4 EXPERIMENTAL DESIGN

ARC-AGI-3 presents unfamiliar sequential grid worlds whose rules and goals must be discovered through interaction. The study has models construct and revise harnesses under a fixed protocol, then evaluates every frozen version on development and held-out games.

4.1 ARC-AGI-3: AN INTERACTIVE REASONING CHALLENGE

  • ARC-AGI-3 requires agents to explore unfamiliar grid worlds and infer game rules, action effects, and level goals that are not given in advance.Games use 64 × 64 color grids and may provide directional inputs, game-specific inputs, coordinate clicks, or undo.Source: §4.1 · p. 4
  • RHAE credits completed levels and discounts solutions using more environment actions than hidden human baselines, with greater weight for later levels.The study also records actions, completed levels, terminal states, and dollar-denominated model inference cost.Source: §4.1 · p. 4
  • Harnesses choose how to represent observations, manage context and memory, provide tools, organize reasoning, and allocate model calls.They may request single actions or sequences, execute discovered strategies with code, and reuse computations to reduce inference cost.Source: §4.1 · p. 4

4.2 AUTONOMOUS CONSTRUCTION AND REVISION

  • The model uses a fixed coding harness to build the task-specific harness; Figure 2 illustrates FT09, where tile-matching rules and later multi-tile controls must be discovered.The construction model can create and revise solver code, with local checks and mock-based tests.Source: §4.2 · pp. 4–5, Figure 2 · p. 5
  • A candidate must run and interact with two development games to qualify; failed candidates can receive execution reports and logs for up to ten repair cycles.Qualification checks execution and interaction within fixed action and level limits, not RHAE performance.Source: §4.2 · p. 5
  • Revision 1 receives source, traces, scores, actions, and inference costs from five development games; revision 2 follows the same process without human edits or guidance.Each revision may change any generated file or preserve its parent harness.Source: §4.2 · p. 5
  • All 27 runs complete construction and both revisions across nine model–effort conditions, with three independent constructions per condition.Every frozen version is evaluated on five development and twenty held-out games; held-out games do not inform construction, revision, or selection.Source: §4.2 · p. 5

5 QUANTITATIVE RESULTS

Two revisions increased mean development RHAE by more than held-out RHAE, while game-level outcomes and cost changes varied across lineages. The pooled results therefore show limited average transfer alongside substantial gains, regressions, and cost trade-offs.

5 QUANTITATIVE RESULTS

  • Held-out RHAE rose by 2.80 points after revision 2, with a 95% bootstrap interval of [−1.44, 7.35] and observed strict-sign changes of 15 gains, 12 losses, and no ties.The interval is compatible with no average benefit or a useful positive gain; the sign test gives p = .70.Source: §5 · p. 6
  • Held-out performance improved on 13 games and declined on seven, with the largest mean gain at 20.44 RHAE points and the largest decline at 15.03.Aggregate improvement can coexist with substantial regressions on particular games.Source: §5 · p. 6
  • The adaptation gap was 5.62 points (95% bootstrap interval [1.78, 9.77]), and development-based snapshot selection yielded a 4.53-point held-out gain versus 2.80 for always deploying H2.Mean deployment cost was $9.15 rather than $7.57 per game, and the selection rule remains to be evaluated prospectively.Source: §5 · p. 6
  • Nine of 27 revision-2 descendants improved both held-out RHAE and cost per game, while six gained RHAE at higher cost and nine became cheaper while losing RHAE.Mean cost fell by $2.72 per game, but the median change was −$0.54.Source: §5 · p. 6
  • Figure 4 plots observed H2 − H0 changes in held-out RHAE and inference cost per game for each model–effort condition and run.Each metric uses a common linear scale across panels, and categories use unrounded changes.Source: Figure 4 · p. 7
  • The experimental program corresponds to at least $18,129.27 under the frozen publication price schedule, including construction, revision, repair, qualification, and fresh selected-game inference.The appendix provides a cost breakdown, replicate-level trajectories, and checkpoint and game-sensitivity analyses.Source: §5 · p. 7

6 WHAT MODELS BUILD AND HOW THEY REVISE

The models produced diverse harness architectures and frequently reorganized them, but revision processes did not reliably translate into held-out improvement. Exploratory behavior comparisons are uncertain and do not establish why revisions succeed or fail.

6.1 WHAT HARNESS DESIGNS EMERGE, AND HOW DO THEY CHANGE?

  • Generated harnesses ranged from compact direct controllers to heuristic-first, hybrid, memory-rich, world-model, replay, planning, and inference-budget designs, with variation even within conditions.Architectures differed in interaction strategy as well as implementation.Source: §6.1 · p. 7
  • Twenty-two of 27 first revisions reoriented the harness, while second revisions included 12 reorientations, 12 extensions, and three preserves; no lineage replaced its harness family.Revisions repeatedly reorganized memory, perception, batching, replay, and resource allocation.Source: §6.1 · p. 7
  • One Sol-high revision raised held-out RHAE from 29.24 to 53.56 at essentially unchanged inference cost by changing action budgeting, memory management, and observations.In a Sol-xhigh lineage, changing response chaining and bounded memory coincided with RHAE falling from 43.95 to 21.99 as cost fell from $31.39 to $7.18 per game.Source: §6.1 · pp. 7–8

6.2 WHICH REVISION BEHAVIORS ACCOMPANY HELD-OUT IMPROVEMENT?

  • Revision-behavior coding had adequate reliability for scope (κ = 0.96), post-result adaptation (0.84), and planning (0.70), but several labels fell below the 0.67 threshold or were excluded.Planning was absent from 27 packets with missingness concentrated in several conditions and larger edits; updater verification was uniformly top-coded, leaving chance-corrected reliability undefined.Source: §6.2 · p. 8
  • Broad rewrites averaged +3.75 immediate held-out change versus −0.20 for contained changes, with a wide interval that includes zero.All 54 episodes inspected development evidence, targeted at least one eligible evidence-backed issue, and passed proximal operational checks, yet held-out outcomes varied.Source: §6.2 · p. 8
  • Reactive episodes averaged +2.48 immediate held-out change, compared with −0.76 when a post-edit result triggered another edit, revert, test change, scope change, or criterion-based stop.The −3.24-point contrast is uncertain, and differences in parent performance, repair effort, and attempted changes prevent conclusions about advantage or causal harm.Source: §6.2 · p. 8

7 DISCUSSION

Autonomous harness construction is substantial, but consistent held-out improvement and capability preservation remain distinct challenges. Comparisons with human-steered systems show headroom, while the experiment does not isolate a human-steering effect.

7 DISCUSSION

  • Autonomous harness building demonstrates substantial capability, but development gains and common evidence-guided revisions do not ensure consistent held-out benefit or explain revision outcomes.Strong initial harnesses can lose observed performance while weaker constructions improve substantially; observed regressions do not establish effects on expected performance.Source: §7 · p. 8
  • At the same Opus 5 xhigh setting, Tycho reached 100 RHAE on every public game, while revision-2 harnesses averaged 39.95 on twenty of those games.Both studies designate the same five development games, but their procedures differ.Source: §7 · p. 8
  • Every scored trajectory changed its implementation despite permission to retain the parent, making selection or abstention distinct from generating a revision.A post hoc development-based selection result illustrates this distinction; candidate revisions can be compared with a preserved baseline and rejected or rolled back when they do not justify replacement.Source: §7 · p. 9
  • Evaluating harness improvement requires tracking whether revisions generalize, preserve prior strengths, and justify their cost; sustained acceleration claims require longer runs, broader tasks, and updaters that can change.The experiment holds both the model and coding bootstrap fixed.Source: §7 · p. 9

8 LIMITATIONS

The study’s evidence is bounded by its benchmark and revision protocol, incomplete repeat evaluations, and limits on attributing score-cost differences or revision behaviors to independent causes.

8 LIMITATIONS

  • The study uses one benchmark, one set of feedback games, and two revisions, leaving behavior on other tasks and longer improvement sequences open.Because the games are public, some models may have encountered them during training.Source: §8 · p. 9
  • Three runs per condition leave uncertainty about small model differences, and unrepeated evaluations mean individual gains or regressions may partly reflect gameplay variation.A ten-run diagnostic showed gameplay can vary even with an unchanged harness.Source: §8 · p. 9
  • Score and cost are reported together, but the study does not separate model capability, harness design, and spending into independent effects.Inference expenditure varies across harnesses and models.Source: §8 · p. 9
  • Revision-behavior annotations used two LLM passes without evaluation against independent human annotations, and summarized records may lose context or event order.Different starting harnesses and repair targets mean observed associations cannot show that a revision behavior caused an outcome.Source: §8 · p. 9

9 CONCLUSION

The study demonstrates that models can build substantial harness capability, but their revisions convert execution experience into held-out gains unreliably. Development progress can overstate transfer, and lower inference cost may accompany lost performance.

9 CONCLUSION

  • Revision 2 raises mean held-out RHAE by only 2.80 points, while development gains overstate held-out improvement.The conclusion also notes that half of the cheaper final harnesses lose held-out performance.Source: §9 · p. 9
  • Human-steered systems show that stronger harnesses are possible for some of the same models.Source: §9 · p. 9
  • Strong task performance does not ensure that autonomous revisions generalize or preserve existing capabilities.The central challenge is making improvements accumulate across tasks while preserving what the system already does well.Source: §9 · p. 9

AI USE STATEMENT

The study combines a fixed experimental protocol with model-generated harness construction and revision, while keeping held-out games outside adaptation. It also reports the assistance used in the research and compares its protocol with prior systems.

AI USE STATEMENT

  • Language-model assistants supported infrastructure, review, literature search, interpretation checks, and editing, while the authors selected the protocol and data, verified analyses, and took responsibility.Source: AI USE STATEMENT · p. 10

A FULL EXPERIMENTAL PROTOCOL

  • Generated systems have no internet access, cross-game state, or access to other game instances, and the starting repository supplies no solver architecture or reusable task policy.The initial repository provides the task contract, engine and model clients, pricing documentation, and entry-point contract.Source: A FULL EXPERIMENTAL PROTOCOL · p. 14
  • The protocol evaluates five development games and twenty held-out games, which provide no feedback for construction, revision, or selection.It also records deployment and pricing-profile hashes and matches builder and runtime profiles within each study-condition row.Source: A FULL EXPERIMENTAL PROTOCOL · p. 14, Table 1 · p. 14
  • Each stage selects one candidate; a qualified initial harness is evaluated on development games, and its frozen source and feedback seed each of two revisions.Smoke qualification uses two development games, with 50 charged actions per game and a stop after two completed levels or a win; unsuccessful candidates may be repaired for at most ten cycles.Source: A FULL EXPERIMENTAL PROTOCOL · p. 14

B MINIMAL STARTING PROMPT

  • The build prompt asks systems to implement and test a complete ARC-AGI-3 solution autonomously, prioritizing high RHAE while using inference economically.It leaves solver architecture, visual representation, memory, and planning choices unspecified; evaluation uses one current game without internet or access to other games.Source: B MINIMAL STARTING PROMPT · p. 15

C RELATED-SYSTEM COMPARISON

  • The study compares protocol evaluation units, selection rules, and editable artifacts; unlike HUIB, one model profile here produces and uses updates.Every frozen-parent descendant is evaluated on held-out tasks without performance-based selection among qualified descendants.Source: C RELATED-SYSTEM COMPARISON · p. 15, Table 2 · p. 16

D ADDITIONAL QUANTITATIVE RESULTS

The additional results compare held-out performance and inference cost, report human-steered reference systems, and examine score variability and sensitivity to games and development feedback. A post hoc development-based checkpoint rule improves mean held-out RHAE over both the initial harness and always deploying revision 2, at higher deployment cost.

D ADDITIONAL QUANTITATIVE RESULTS

  • 24.54 is the initial held-out mean RHAE across 27 complete lineages, with scores ranging from 0 to 97.14.Table 3 reports completed trajectories; three logical costs were recovered from selected per-game token ledgers after aggregate report assembly failed.Source: D ADDITIONAL QUANTITATIVE RESULTS · p. 16, Table 3 · p. 16
  • 29.06 mean held-out RHAE from post hoc development-based checkpoint selection is +4.53 over H0 and +1.72 over always deploying H2, while mean deployment cost rises from $7.57 to $9.15 per game.The evaluation is post hoc and does not establish the performance of a prospectively specified selection policy.Source: D ADDITIONAL QUANTITATIVE RESULTS · p. 17, Table 5 · p. 17
  • At least $18,129.27 is the experimental program cost under the frozen publication price schedule, including construction, revision, repair, qualification, and fresh selected-game inference.Coding-bootstrap inference totals at least $1,352.34 and smoke qualification adds $646.84 across planned runs.Source: D ADDITIONAL QUANTITATIVE RESULTS · p. 18, Table 8 · p. 18
  • 100.00 RHAE is reported by human-steered systems on all 25 public games, while study revision-2 held-out means range from 15.43 to 41.94 across the listed model-effort conditions.Tycho also scores 100 on the same twenty held-out games; the Opus comparison matches model version and reasoning effort, while the Sol effort labels differ.Source: Table 4 · p. 17, D ADDITIONAL QUANTITATIVE RESULTS · p. 18

D.1 FIXED-HARNESS EXECUTION VARIANCE

  • 48.53 is the mean RHAE across ten evaluations of one unchanged harness, with a 3.99-point sample standard deviation and a 43.38–55.37 range.This is a post hoc diagnostic on one snapshot, not a random sample of harnesses.Source: §D.1 · p. 19, Table 10 · p. 19
  • 13 held-out games have positive mean endpoint changes and seven have negative changes; the largest gain is +20.44 and the largest decline is −15.03 RHAE points.Omitting r11l from the development summary reduces the observed development–held-out gap from 5.62 to 3.02 points, while other single-game omissions leave gaps of 4.68–7.84.Source: §D.1 · pp. 19–20

E HARNESS AND STRATEGY AUDIT

The audit finds substantial harness revision but no simple link between edit volume, strategy changes, and held-out improvement. Process coding and campaign audits document how revisions and evaluations were assessed, while noting limits on causal interpretation and cost accounting.

E HARNESS AND STRATEGY AUDIT

  • Median harness size rose from 2,161 to 2,604 to 2,912 generated Python lines, yet both large and small diffs occurred on either side of zero held-out change.Mean endpoint growth was 1,052 lines; Figure 5 plots lineage growth and selected diffs against held-out outcomes.Source: E HARNESS AND STRATEGY AUDIT · p. 20, Figure 5 · p. 21
  • First revisions most often changed operating strategy, while second revisions split evenly between extending capabilities and reorienting strategy.Changes included context persistence, replay, consultation cadence, action batching, and budget allocation.Source: E HARNESS AND STRATEGY AUDIT · p. 21
  • Strategy paths did not certify improvement: Sol-high replicates with identical paths ranged from a 33.18-point held-out gain to a 15.18-point loss.Opus-high and Opus-xhigh reoriented at every stage but also showed both gains and regressions.Source: E HARNESS AND STRATEGY AUDIT · p. 21
  • Sol-high replicate 1 rose from 29.24 to 53.56 held-out RHAE at nearly unchanged cost, while Sol-xhigh replicate 1 fell from 43.95 to 21.99 as cost dropped.Multiple components changed together in both cases, so neither isolates a causal mechanism.Source: E HARNESS AND STRATEGY AUDIT · pp. 21–22

F CAPABILITY CODEBOOK AND SAMPLING

  • The process codebook covered evidence access, diagnosis, probes, planning, edits, verification, results, adaptation, and resource allocation, using only visible events and reproducible artifact links as direct evidence.All 54 revision episodes were independently annotated under a frozen codebook; consensus was frozen before outcome linkage.Source: F CAPABILITY CODEBOOK AND SAMPLING · pp. 22–23
  • Exploratory process associations were unadjusted, and trajectory-profile shuffling may violate exchangeability when conditions differ.Planning was unobservable in 27 of 54 packets, with condition-dependent missingness, so observed-subset agreement does not support a planning–outcome conclusion.Source: F CAPABILITY CODEBOOK AND SAMPLING · p. 24

G CAMPAIGN INTEGRITY AND RECOVERY AUDIT

  • The source audit confirmed unchanged experimental controls across recorded versions, but explicitly did not constitute an independent experimental rerun.Retained-record checks covered all 27 complete submitted lineages.Source: G CAMPAIGN INTEGRITY AND RECOVERY AUDIT · p. 24
  • Across 2,130 attempts for 2,025 game–snapshot slots, exactly one clean attempt was selected per slot, but interruption causes were not fully established.The audit found 21 slots with fresh restarts; 17 selected attempts were fresh restarts.Source: G CAMPAIGN INTEGRITY AND RECOVERY AUDIT · p. 25

Related papers

Loading 2610.30499v1…