PaperlayerView exact revision on arXiv

Source-linked AI summary

Kepler: Auditable World Models for ARC-AGI-3

Wensen Wu

arXiv:2610.00834v1cs.AIcs.MA

TL;DR

Interactive agents must infer rules and objectives from unfamiliar environments, yet final scores alone do not reveal how they learned or whether their trajectories respected the intended experiment. Kepler uses executable world models with retrospective and conditional prediction checks to audit behavior. Two frozen release configurations reached 100.00 and 95.97 RHAE, while the paper reports important evaluation boundaries.

  • Problem

    Final scores can describe valid solutions, lucky trajectories, replay-selected outcomes, or agents that accessed information outside the intended boundary.

    Source: §1 · p. 1

  • Method

    Kepler checks agent-written transition models against complete prior history and tests their predictions against subsequent observations, recording counterexamples on mismatches.

    Source: §1 · p. 1

  • Results

    Two frozen release configurations reached 100.00 and 95.97 RHAE, with actions, tokens, costs, replay status, and limitations reported separately.

    Source: §1 · p. 2

  • Takeaways & Limitations

    At the top of the public set, score alone reveals little about how systems learned or whether their trajectories respected the intended experiment.

    Source: §1 · p. 1

    The same 25 public games served as development material and evaluation, so results establish neither held-out generalization nor private-set performance.

    Source: §6.4 · p. 13

Abstract

from arXiv · show

ARC-AGI-3 evaluates agents in interactive environments whose rules and objectives must be inferred from observation. We present Kepler, an open-source harness that represents hypotheses as executable world models and validates them through retrospective transition checks and conditional prediction checks. Under one frozen Claude Opus 5 configuration, Kepler obtained a server-verified 100.00 RHAE on all 25 public games, with no per-game model selection or score-conditioned reruns. On 181 of 183 completed levels, the final Opus attempt used no more actions than the corresponding median-human baseline. The retained board runs used 8,256 environment actions, of which 7,292 occurred in scored levels. Retained local provider-session records yield 858.0 million tokens, 97.37% cache reads, and a \$777.72 cost at September 1, 2026 API list-equivalent rates. We also report three evaluation failures: source-code leakage that produced an invalid perfect run, agents reconstructing a removed harness in a control condition, and autonomous repair masking a broken planner. A single-game observation case study showed that animation frames contained task-relevant information absent from settled text grids. Across the final Claude Opus 5 and GPT-5.6 Sol boards, 48 of 50 game-model cells reached 100. These results indicate that public-set score alone has limited discriminative value and motivate first-attempt, cost-conditioned, and verification-aware reporting.

1 Introduction

Kepler addresses the difficulty of distinguishing valid interactive-agent solutions from lucky, replay-selected, or boundary-violating trajectories. It combines executable world models and auditable checks with public-game results, while recognizing that those results do not establish held-out generalization or isolate the harness’s causal effect.

Problem and motivation

  • At the top of the public set, score alone reveals little about how a system learned or whether its trajectory respected the intended experiment.ARC-AGI-3 provides unfamiliar visual environments, legal actions, and no rule sheet or stated objective; replay scores do not establish first-exposure skill acquisition or protocol compliance.Source: §1 · p. 1

Approach

  • Kepler checks an agent-written transition model against complete prior history and tests predictions against subsequent observations, recording counterexamples on mismatches.The protocol evaluates externally committed beliefs and behavior rather than private chain-of-thought.Source: §1 · pp. 1–2

Evaluation and audit

  • Across 330 development and evaluation runs, trajectory audits found source access, contaminated controls, silent tool replacement, mutable instructions, and replay-transport failures missed by outcome-only evaluation.The paper reports an auditable evidence contract built around append-only evidence, complete-history retrodiction, conditional prediction checks, and mechanical replay.Source: §1 · p. 2

Capability and resource evidence

  • Two frozen release configurations reached 100.00 and 95.97 RHAE, with actions, tokens, costs, replay status, and limitations reported separately.The study also identifies a transient mechanic visible in retained animation frames but absent from settled-grid observations.Source: §1 · p. 2

Scope and limitations

  • The results establish neither held-out generalization nor the harness’s causal effect because evaluation reused the 25 public games and the intended control was invalidated by filesystem leakage.Release boards contain one retained run per game.Source: §1 · p. 2

2 Method

Kepler places an agent in a workspace where it records a game theory, tests that theory against observed transitions, and commits actions through guarded tools. Certification and replay checks are conditional, and one shared configuration is used across games.

2.1 The loop: observe, model, certify, plan, commit

  • The agent’s persistent world model grounds pixels into named objects and simulates actions; notes record checked or assumed claims, while the daemon appends transitions to a ledger agents are instructed not to edit.The ledger’s append-only behavior is not enforced as a filesystem immutability guarantee.Source: §2.1 · p. 3

2.2 The guarded channel

  • Before each non-reset action, commit.py compares a usable model prediction with the resulting observation; mismatches void the remaining plan and return a counterexample.Partial or empty predictions and simulation exceptions can allow execution to proceed, so prediction-based interruption is conditional rather than universal.Source: §2.2 · p. 3

2.3 Certify, then mechanically execute

  • The release writes per-level programs and mechanically executes them without further model decisions, checking program lengths against 1.3× the human baseline.Certification adds simulation and click-coordinate checks when the model exposes init_state, step_state, and outcome; otherwise it degrades to length checks.Source: §2.3 · pp. 3–4

2.4 Shared configuration and identifier checks

  • One configuration is used across games without game-ID dispatch, and a CI scan checks agent-visible files for literal game identifiers.The identifier check does not prove the absence of encoded game knowledge or model-training exposure.Source: §2.4 · p. 4

3 Named Experimental Stages

The named stages track frozen configurations and show how discipline rules, guard corrections, and clean-run execution affected reported outcomes.

3 Named Experimental Stages

  • The release configuration produced server-exact scores of 100.00 for Claude Opus 5 and 95.97 for GPT-5.6 Sol.It added recovery-path fixes, rendered-frame observation, and transport validation to the certify/replay stage.Source: §3 · p. 4

3.1 The discipline regression and the false-premise autopsy

  • The added-discipline stage dropped 3.77 points, while attempt-reset discipline coincided with sp80 falling from 82.16 to 14.29.The local development and final-segment replay procedure did not enforce the first-exposure protocol, so these scores are not official-leaderboard first-exposure results.Source: §3.1 · p. 5
  • Guard corrections recovered 2.50 points in the staged comparison by demoting fraction-based guards to warnings while retaining an absolute 3,000-action refusal.The authors describe this as a development-setup result, not evidence about matched official evaluations.Source: §3.1 · p. 5

3.2 The dead-code gate

  • A WIN short-circuit blocked certified restarts in three games; the certify/replay fix passed RESET-first plans through, and fresh post-freeze runs followed.Pre-freeze validation converted sc25 from 84 to 100 and g50t from 92 to 100, showing that execution machinery can suppress already-discovered solutions.Source: §3.2 · p. 5

4 Results

Across two frozen model configurations, 48 of 50 release-board game-model cells scored 100.0; the results also report replay qualifications, an sp80 observation case study, and separate resource measures.

4.2 The verified-card matrix

  • The superseded visual-replay board scored 94.42 server-side after two off-grid ACTION6 clicks passed locally but failed server replay; clean-run coordinate checks are not universal transport enforcement.The retained 100.00 board replayed exactly, but replay validates submitted trajectories, not every possible input or code path.Source: §4.2 · p. 6

4.1 Boards

  • Each release board used one frozen configuration across all 25 public games, with one run per game; an earlier score-conditioned rerun is labeled separately.Figure 3 highlights four games driving composite differences; sp80 collapses under forced-reset discipline, partly recovers under certify/replay, and reaches 100 after rendered-frame observation is added.Source: §4.1 · p. 5, Figure 3 · p. 6, Table 3 · p. 7

4.3 Public-set score saturation

  • 48 of 50 release-board cells scored 100.0 across two frozen model configurations, a final-board score convergence that is not learning speed, independent replication, or held-out generalization.The Opus board was single-configuration and server-verified-exact at 100.00; earlier certify/replay boards had 43 of 50 games at 100.0.Source: §4.3 · p. 6, Figure 4 · p. 7

4.4 An observation-channel case study

  • On sp80, a continuation with rendered animation frames identified a deflection rule and solved the final level in 57 actions, but no matched fresh text-only control isolates vision’s effect.The 19 earlier text-mode sessions shared accumulated notes and models, so they were not independent trials.Source: §4.4 · p. 7

4.5 Cost

  • The exact 100.00 Opus campaign used 858,041,926 raw tokens, 97.37% cache reads, and $777.72 in September 1, 2026 API list-equivalent pricing.Actual execution used Claude subscription quota; the dollar figure is a reproducible API-price comparison, not a billed charge.Source: §4.5 · p. 8
  • Kepler’s list-equivalent estimate was 74.0% below Tycho’s estimate, but Retrodict and baseline1 were cheaper at their reported scores.These are inference-price estimates, not billed charges or a controlled harness comparison; pricing, cache behavior, development costs, and run-selection methods differ.Source: §4.5 · p. 8, Table 5 · p. 9
  • The GPT-5.6 board used 2,429,083,433 raw tokens and cost $1,312.14 at September 1, 2026 list-equivalent rates.The estimate uses provider-side records after incomplete workspace CLI footer counters were found to omit most cached traffic and a long workspace.Source: §4.5 · pp. 8–9
  • The Opus board reports 8,256 retained-run actions and 7,292 scored-level actions; 181 of 183 completed levels used no more actions than the median-human baseline.These are final-attempt action counts, not learning-inclusive totals; the paper does not rank them against other systems’ differently defined counts.Source: §4.5 · p. 9

5 Evaluation integrity and failure cases

Kepler’s audits found that outcome scores and clean ledgers can miss serious evaluation failures. The incidents exposed source-code leakage, a control group that reconstructed the harness, a broken planner masked by agent repairs, and limits in network and workspace controls.

5 Evaluation integrity and failure cases

  • A source-reading agent read all 2,172 lines of the game, returned 100.00 without modelling it, and was voided and quarantined after a clean rerun scored 46.91, a 53-point drop.The structural fix moved environment files outside every workspace but did not make them unreadable; launch CLIs retained host filesystem access.Source: §5 · p. 10
  • Score verification recomputed 363 completed levels: 354 counts matched exactly, nine were conservative, and none inflated the score.These checks are evidence-bounded and cannot detect events the client did not retain.Source: §5 · p. 10

5.1 Incident 2: the control group rebuilt the harness

  • All six baseline agents accessed harness tools, so the ablation was harness-against-harness and the harness’s contribution to scores remains unmeasured.A valid control requires enforced boundaries excluding treatment tools, prior solutions, and engine source, with tests confirming access is denied.Source: §5.1 · pp. 10–11

5.2 The workspace-compression phenomenon

  • 26 of 26 GPT-campaign workspaces rewrote AGENTS.md, and twice the rewrite reached the repository-root file.The authors infer that writable instructions should be treated as mutable data.Source: §5.2 · p. 11

5.3 Outcome integrity does not imply tool integrity

  • The bundled bfs.py failed on every invocation, but agents silently replaced its search and continued solving, leaving outcome audits clean.Kepler added a smoke test for workspace tools and treats tool integrity and outcome integrity as separate release gates.Source: §5.3 · p. 11

5.4 Replay-transport seams, and what “no network access” precisely means

  • The agent made zero external requests, but the harness made a startup handshake through arc_agi’s anonymous-key call.The earlier description “no network access” was therefore imprecise; the agent’s only HTTP target was the localhost daemon.Source: §5.4 · p. 11

6 Discussion

The discussion separates solution discovery from replay execution and argues that public-set scores obscure important differences in discovery, cost, and verification. It describes an sp80 case where a missing mechanic, rather than insufficient search alone, constrained progress, while emphasizing the limits of the release evidence.

6.1 Discovery is the residual signal

  • 43/50 games reached 100 across two frontier models under certify/replay, versus 48/50 on the release boards; the remaining failures arose before a correct solution program existed.The comparison does not isolate replay’s causal contribution because adjacent harness and observation changes were not held constant.Source: §6.1 · pp. 11–12

6.2 Implications for ARC-AGI-3 evaluation

  • Final-attempt action efficiency omits the inference, exploration, and failed attempts preceding that attempt, so the paper recommends reporting discovery, execution, and replay separately.Similar replayed scores do not establish a shared architecture or discovery efficiency.Source: §6.2 · p. 12
  • With five public-set entries at ≥99 and four at 100, cost becomes a differentiating axis; the authors propose fixed-budget scores, standardized token and cost reporting, first-attempt evaluation, verification, and held-out-set headlines.Prediction-based plan interruption appears in four harnesses, but is a shared design pattern rather than evidence of causal reliability gain; Kepler’s missing-prediction fallbacks limit coverage.Source: §6.2 · p. 12

6.3 The sp80 observation failure and recovery

  • The sp80 agent searched roughly 4.1 × 10^8 configurations under an incompletely fitted model, while retained animation later exposed a deflection rule.Earlier frame counts also contained distinguishing clues, so the case does not establish that images were necessary; the authors propose measuring latency from prediction discrepancy to tested repair.Source: §6.3 · p. 12

6.4 Limits

  • A six-probe S5I5 test produced no useful counterexample, learned repair, or solving advantage, and was neither an independent-case evaluation nor a new discovery algorithm.The investigator knew the historical mechanic, and visible reset equality does not prove hidden-state equality.Source: §6.4 · pp. 12–13
  • The release evidence uses one public benchmark, one retained run per cell, and repeated development on the same 25 games; it establishes neither held-out generalization nor private-set performance.The harness’s causal contribution remains unmeasured pending a boundary-correct ablation, and provider training exposure cannot be ruled out.Source: §6.4 · p. 13

7 Related Work

Kepler is an independent implementation and evaluation of an executable-world-model approach, not a claim to invent its component mechanisms. The related work spans world-model and direct-interaction systems, while the paper emphasizes that unmatched protocols and bounded checks constrain comparisons and transfer.

7 Related Work

  • Executable world models, replay validation, checked execution, and goal discovery predate Kepler in OPINE-World [8], Tycho [11], Retrodict, Twin [16], and related work.The authors frame their contribution as release evidence and documented evaluation failures, not a new general repair algorithm or proof that replay fit guarantees competent play.Source: §7 · pp. 13–14
  • Early ∼99%-class reports used per-game best-of-2 fallback with score feedback; Kepler abandoned that score-conditioned selection and reports its initial loop without fallback as 97.78.The executable-world-model idea itself is established in published work, including Tycho [11] and baseline1.Source: §7 · p. 14
  • VISTA [9] and AVO use direct rendered-frame interaction rather than executable world models, but their observation channel is confounded with other system differences.The authors’ visual continuation inherited earlier learning, motivating a matched observation study rather than a benchmark-wide visual-advantage claim.Source: §7 · pp. 14–15
  • Prime Agent [10] reports three Opus runs at 94.99–95.50, while Kepler reports a higher single-board score; unmatched protocols do not support ranking capability or integrity.Prime Agent provides stronger run-to-run evidence with n = 1 release cells for Kepler.Source: §7 · p. 15
  • Stock Claude Code with an 11-line prompt reaches 96.2 pass@1 / 99.3 pass@2, a datapoint the authors say argues against attributing Kepler’s Claude score entirely to harness design.They cite it as further reason a valid control experiment is needed.Source: §7 · p. 15

7.1 Failure-to-check map

  • These checks do not establish empirical transfer beyond this benchmark, and exact replay depends on discrete observations and sufficiently reproducible transitions.Broader systems need domain-specific comparators and explicit stochasticity handling; syntactic trace checks are not semantic validation.Source: §7.1 · p. 15, Table 6 · p. 15

8 Reproducibility Statement

The paper releases code, scorecards, verification programs, and final-board traces to support offline score and ledger checks. Its reproducibility statement distinguishes mechanical score reproduction from bit-for-bit reproduction of live runs and documents limits of the published records.

8 Reproducibility Statement

  • The repository ships code, official scorecards, a release manifest, and verification programs, while the companion dataset contains the two final 25-game boards.The listed commands recompute scores, verify ledgers, audit integrity, replay a local fixture, and scan for literal game identifiers.Source: §8 · pp. 15–16
  • The dataset contains 50 run records and 58,098 environment events, and independent recomputation matches 100.00 for Claude Opus 5 and 95.9672 for GPT-5.6 Sol.Ledger checks cover all 363 completed levels; integrity scans apply to published records and a fixed pattern set.Source: §8 · p. 16
  • Published-ledger score reproduction is mechanical, but bit-for-bit live-run reproduction is not expected because of sampling and provider drift.Provider-session records used for token repricing remain private, and the 95 captured CLI logs provide uneven rather than complete event coverage.Source: §8 · p. 16

Related papers

Loading 2610.00834v1…