PaperlayerView exact revision on arXiv

Source-linked AI summary

Vestrum: Improving Agent Harnesses by Adapting Their Verification, Structure and Memory

Jayant Parashar, Eugene F. Douglass, William C. Bastian, Suchendra M. Bhandarkar

arXiv:2609.33822v1cs.AI

TL;DR

Long environment interactions make harness improvement costly, and execution traces offer diagnoses that may be hard to turn into changes useful beyond one episode. Vestrum converts those failures into scoped mechanisms, screens and evaluates them as bundles, and improves held-out performance across five settings.

  • Problem

    Long episodes can make evaluation costly, and turning one episode’s diagnosis into a change that remains useful elsewhere is difficult.

    Source: §1 · p. 1

  • Method

    Vestrum links task kinds and recognizable failure classes to scoped harness mechanisms, screens their generality, evaluates bundles, and retains search lessons.

    Source: §1 · pp. 1–2

  • Results

    Learned harnesses improve held-out performance across five settings and two baseline harnesses, while evidence-grounded verification helped across four settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Verification grounded in evidence was useful at intermediate commitments and for selective final-answer checking, whereas critics rebuilding finished answers broke more than they repaired.

    Source: §1 · p. 2

    One selected harness per setting was evaluated, so test repetitions do not measure variability across independent searches.

    Source: §5 · p. 9

Abstract

from arXiv · show

An agent harness controls how a language model accesses information, uses tools, preserves memory, and checks its work. Improving this software is costly when each evaluation requires a long interaction with an environment. We introduce Vestrum, a framework that turns failures in execution traces into scoped harness changes without training the task model. Its organizing overhypothesis is that tasks of a shared kind may exhibit recurring failures whose remedies transfer within that kind. Vestrum expresses failures as recognizable classes, proposes changes across verification, retrieval, decomposition, and knowledge synthesis, and screens their scope before evaluating them as a bundle. A persistent lessons file informs subsequent proposals. Across five settings and two baseline harnesses, the frozen harnesses improve held-out performance: UltraHorizon rises from 47.6 to 59.8 over GAM, Terminal-Bench 4 Hard from 63.7% to 70.3% of checks passed over Claude Code on eight held-out tasks at 1.03x test cost, and cell-type annotation agreement from 67.5% to 77.8% on held-out sections of one slide, alongside gains on LoCoMo and AMA-Bench. Across our searches, verification grounded in evidence helped both intermediate steps and final answers, at lower cost at intermediate steps, while critics asked to rebuild finished answers broke more than they repaired. On the three memory benchmarks, Vestrum also scores above the evaluated GEPA configurations in every paired evaluation.

1 INTRODUCTION

Vestrum treats harness revision as a scoped learning problem: recurring failures within a task kind may share remedies, but transfer must be screened and evaluated. Across five settings, its frozen harnesses improve held-out performance, while the searches highlight evidence-grounded verification as useful at intermediate steps and final answers.

Overhypothesis-guided harness adaptation

  • Vestrum’s overhypothesis is that tasks of a shared kind may have recurring failure classes whose remedies transfer within that kind.The overhypothesis guides categorization and scope but does not guarantee transfer.Source: §1 · pp. 1–2

Verification as an adaptable harness component

  • Harness verification can act at commitment points such as writing a memory, merging evidence, accepting a rule, or editing code.These are moments when later computation begins to depend on an intermediate result.Source: §1 · p. 2

Overhypothesis-guided harness adaptation

  • Vestrum connects task kinds and failure classes to scoped mechanisms across four design dimensions, screens their generality, and evaluates bundles while retaining search lessons.The design dimensions are verification, retrieval, decomposition, and knowledge synthesis.Source: §1 · p. 2

Evidence-grounded verification

  • Evidence-grounded verification helped across four settings, including checks that repaired 17 of 18 AMA-Bench notes and a selective final-answer check that repaired 6 and broke 4.Critics asked to rebuild finished answers broke more correct answers than they fixed; intermediate checks acted on only 12–13% of LoCoMo questions.Source: §1 · p. 2

Empirical gains across settings

  • Across five settings, learned harnesses improve held-out performance over their starting harnesses, with Vestrum also exceeding evaluated GEPA configurations on memory benchmarks.The GEPA comparison concerns task executions; learning-token use is higher on two benchmarks.Source: §1 · p. 2

2 OVERHYPOTHESIS-GUIDED HARNESS ADAPTATION

Vestrum uses development traces to diagnose recurring failure classes, propose scoped harness mechanisms, screen and evaluate bundles, and retain lessons for later search. Its overhypothesis guides where remedies might transfer, while evaluation—not screening—determines whether a changed harness is admitted.

2 OVERHYPOTHESIS-GUIDED HARNESS ADAPTATION

  • Each epoch diagnoses failures from development traces, proposes scoped mechanisms, evaluates bundles, and updates a persistent lessons record; the harness changes only when admission criteria are met.The general adaptation loop is used across five settings, with benchmark-specific interfaces for execution, scoring, and categorization.Source: §2 · pp. 2–3, §2.6 · p. 5

2.1 TASK KINDS, FAILURE CLASSES, AND SEARCH STATE

  • Task kinds are assigned before proposals and define possible transfer scope, while failure classes specify behaviors that may recur within a kind.Categories use task descriptions or benchmark categories and are not redefined to justify a proposal.Source: §2.1 · p. 3

2.2 FROM TRACES TO RECOGNIZABLE FAILURES

  • A language model reads a failed-check trace, localizes the apparent failure, and expresses it as a predicate recognizable in other traces.The failure predicate describes behavior rather than prescribing its remedy.Source: §2.2 · pp. 3–4

2.3 PROPOSALS ACROSS FOUR DESIGN DIMENSIONS

  • Proposals consider verification, retrieval, decomposition, and knowledge synthesis; each states its target class, applicable kinds, recognition predicate, and predicted check improvements.A language model scores proposals, and promising mechanisms are composed into a scope-restricted bundle before evaluation.Source: §2.3 · p. 4

2.4 GENERALIZATION SCREENING AND SCOPE

  • A reviewer checks whether a mechanism’s condition and content extend beyond its originating example, while lexical lint rejects task-specific names, symbols, paths, and constants.Passing screening makes a proposal eligible for evaluation but does not establish transfer; task-kind scope determines where it executes.Source: §2.4 · p. 4

2.5 BUNDLE EVALUATION AND ADMISSION

  • The current and proposed bundles are compared on development tasks; trace diagnoses guide pruning, and admission depends on bundle performance and any required validation.A Terminal-Bench example shows that restricting a mechanism to repair tasks allowed admission after its broader scope lost a check on another task kind.Source: §2.5 · p. 4

2.6 LEARNING ACROSS EPOCHS

  • Lessons record successes, failures, and repeated refusals for later proposals, separately from task memory, and the final harness is frozen before testing.The epoch procedure updates lessons after candidate evaluation and harness admission.Source: §2.6 · pp. 4–5

3 HELD-OUT PERFORMANCE AND EVALUATION RESOURCES

Vestrum’s frozen harnesses improve held-out results across agentic memory, coding, and cell-type annotation. The evaluations compare memory systems with GAM and GEPA, coding with Claude Code, and annotation against expert labels.

3 HELD-OUT PERFORMANCE AND EVALUATION RESOURCES

  • One harness per setting was frozen before testing, with test tasks disjoint from development and validation; memory systems were tested with a different task model than during search.Repetitions measured execution and judging variability of frozen harnesses, not variability across independent searches.Source: §3 · p. 5

3.1 AGENTIC MEMORY: ULTRAHORIZON, LOCOMO, AND AMA-BENCH

  • 47.6 to 59.8 on UltraHorizon; LoCoMo rises from 71.5% to 74.9%, and AMA-Bench’s second run from 64.2% to 74.2%, all over GAM.The LoCoMo gains were positive across three replicates; AMA-Bench improved in both generation runs.Source: Table 1 · p. 6, §3.1 · p. 6
  • Vestrum scores above GEPA in every reported paired memory evaluation, but the comparison evaluates configurations rather than isolating the search algorithm.Mean margins were 2.3 points on LoCoMo, 5.6 on AMA-Bench, and 11.9 on UltraHorizon.Source: §3.1 · p. 6
  • Vestrum uses about a quarter of GEPA’s estimated task-agent evaluations on LoCoMo and AMA-Bench and the same number on UltraHorizon.These counts measure task executions, not equal learning compute; Vestrum used more learning tokens on two benchmarks.Source: §3.1 · p. 6

3.2 TERMINAL-BENCH 4 HARD

  • 70.3% versus 63.7% of checks passed on eight held-out Terminal-Bench tasks, with test cost at 1.03× Claude Code.Four task means improved, four were unchanged, and none declined.Source: §3.2 · p. 6, Table 2 · p. 7
  • Removing the lessons file lowers development gains from 16 to 10 checks and validation gains from 11 to 6, at 1.6× cost.Removing the ceiling probe lowers validation gains to 5; removing failure classes, generalization screening, and task-kind scope as well leaves no validation gain.Source: Table 3 · p. 7, §3.2 · p. 7

3.3 PATHBENCH: RECONSTRUCTING EXPERT CELL-TYPE ANNOTATIONS

  • 77.8% versus 67.5% agreement with expert labels, and macro-F1 76.5 versus 64.8, on held-out PathBench sections.The held-out sections were other sections of the same slide; labels came from an expert-in-the-loop pipeline.Source: §3.3 · pp. 7–8, Table 4 · p. 8

4 WHAT THE LEARNED HARNESSES REVEAL

The learned harnesses alter available evidence, procedures, and action checks, with verification retained at points where later work depends on an intermediate result. Across the searches, evidence grounding and selective placement distinguish useful checks from broad answer rewriting.

4 WHAT THE LEARNED HARNESSES REVEAL

  • The learned harnesses make verification part of changes to evidence, workflow placement, and action conditions, rather than a standalone final check.A trace can motivate a change to the evidence available, the procedure that uses it, or the conditions under which the agent acts.Source: §4 · p. 8

4.1 VERIFICATION: PLACEMENT, EVIDENCE, AND SCOPE

  • Retained verifiers act at commitment points, checking intermediate records or withholding actions using evidence or harness-enforced prerequisites.Examples include a memory write, evidence merge, rule commitment, and first edit of a shipped file.Source: §4.1 · p. 8
  • Broad final-answer checking repaired 5 AMA-Bench answers but broke 9, while a LoCoMo answer-level checker improved 2 and regressed 6.A selective AMA-Bench checker repaired 6 answers against 4 regressions; the LoCoMo evidence-merge pipeline scored 0.796 overall.Source: §4.1 · pp. 8–9
  • The retained UltraHorizon gate replays logged observations before accepting a rule; an evidence-sufficiency gate at the same stage blocked every development episode and lost 10 points.Placement alone did not ensure a useful check; the evidence and acceptance condition also had to allow progress.Source: §4.1 · p. 9
  • Held-out gains were measured for complete harnesses; no verifier was evaluated alone at test.The appendix reports each retained check’s activity and failure record.Source: §4.1 · p. 9

5 DISCUSSION AND LIMITATIONS

Vestrum links recurring failures to scoped harness changes across four dimensions, and its evaluated harnesses improve held-out performance. The evidence remains bounded by selected settings and bundled verification evaluation.

5 DISCUSSION AND LIMITATIONS

  • Vestrum connects an overhypothesis about recurring failures to scoped changes across four design dimensions, with development histories showing verification’s dependence on evidence and scope.The framework turns execution failures into reusable changes to agent software.Source: §5 · pp. 9–10
  • The evidence covers one selected harness per setting, pilot-motivated design dimensions, one search per Terminal-Bench ablation variant, and PathBench sections from one slide.Test repetitions do not measure variability across independent searches; LoCoMo excludes unanswerable questions despite encouraging best-supported guesses.Source: §5 · p. 9
  • Verification was evaluated in bundles without test-time verifier ablations, leaving controlled component comparisons, independent search repetitions, and broader transfer for future tests.GEPA comparisons use a different editable surface and search model, and estimated evaluations and learning costs follow Appendix L accounting.Source: §5 · p. 9

ETHICS STATEMENT

The evaluation uses existing benchmarks and spatial transcriptomics measurements, including public CosMx data and expert reference annotations.

ETHICS STATEMENT

  • The study evaluates existing benchmarks and spatial transcriptomics measurements; LoCoMo uses synthetic dialogues and UltraHorizon simulated environments.The benchmark datasets are used under their released licenses.Source: ETHICS STATEMENT · p. 10
  • PathBench uses publicly available CosMx measurements from six sections of one HNSC slide and expert annotations from a bioinformatician’s pipeline.Evaluation records are keyed by section and cell index without patient identifiers.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

The appendix describes AI assistance and positions Vestrum within research on agentic memory, harnesses, long-horizon agents, architecture search, verification, and evaluation validity.

AI USE STATEMENT

  • AI assisted conceptual framing, methods, implementations, analyses, and manuscript work, while authors retained responsibility for study design, reported claims, and final content.AI did not generate synthetic evaluation data, spatial measurements, or expert labels.Source: AI USE STATEMENT · p. 10

A.1 AGENTIC MEMORY

A.2 AGENTIC HARNESSES

  • Vestrum belongs to trace-driven harness revision, organizing proposals by design dimension and task-kind scope while carrying admitted mechanisms and lessons across epochs.It can also revise recursive-model subcalls and the information prepared and retrieved for them (Zhang et al., 2025a; Zhou et al., 2022).Source: §A.2 · p. 15

A.3 LONG-HORIZON AGENTS

  • Vestrum shares failure-driven proposals and two benchmarks with long-horizon improvement work, but organizes remedies across four dimensions and evaluates composed bundles.Failure attribution remains difficult even when execution traces are available (Cemri et al., 2025; Zhu et al., 2025).Source: §A.3 · pp. 15–16

A.4 SEARCH OVER AGENT ARCHITECTURES

  • Unlike approaches that vary editable surfaces, search procedures, and feedback, Vestrum links localized failure predicates and task kinds to scoped mechanisms, persistent lessons, screening, and bundle admission.It uses a language model to localize failures and separately evaluates remedies; the paper reports no empirical comparisons against AdaMAST or HarnessFix.Source: §A.4 · p. 16

A.5 VERIFICATION IN AGENTS

  • Vestrum makes verification placement, evidence, scope, and permitted action editable; its record includes mixed outcomes for answer rewriting, selective checking, and intermediate gates.The LoCoMo checker improved two answers but regressed six, while gates unable to distinguish rival rules impeded progress.Source: §A.5 · p. 16

A.6 EVALUATION VALIDITY

  • Vestrum separates development, optional validation, and post-freeze test data, using fresh UltraHorizon validation seeds but making no formal reusable-holdout guarantee.Repeated candidate scoring can create an adaptive-analysis problem, and the table reports the evaluation splits.Source: §A.6 · pp. 16–17

B EVALUATION SETTINGS

The evaluation-settings material summarizes five settings, their data splits, grading units, and the test-time models used for GAM and Claude Code.

B EVALUATION SETTINGS

  • Table 7 summarizes the five evaluation settings, their data splits, and the units each benchmark grades.Development, validation, and test tasks are disjoint, and test tasks are run only after harness freezing.Source: B EVALUATION SETTINGS · p. 17, Table 7 · p. 17
  • At test, GAM uses Gemini 3.7 Flash, while Claude Code uses Sonnet 5 on Terminal-Bench and Opus 5 on PathBench.A check is the benchmark’s grading unit.Source: Table 7 · p. 17
  • Table 8 records how the shared evaluation loop was run in each setting.GAM starts the UltraHorizon, LoCoMo, and AMA-Bench evaluations; Claude Code starts Terminal-Bench 4 Hard and PathBench.Source: Table 8 · p. 17, B EVALUATION SETTINGS · p. 17

C TERMINAL-BENCH ABLATION

The Terminal-Bench ablations examine which parts of Vestrum’s loop contribute to useful, scoped changes; the frozen harness improves held-out checks while the paper details boundaries on generalization and supervision.

C TERMINAL-BENCH ABLATION

  • The full loop gained 16 development checks and 11 validation checks at 1.32× cost, versus 10 and 6 without lessons at 1.6×; removing the ceiling probe reduced gains to 3 and 5.The no-lessons candidate was not evaluated on the test set.Source: C TERMINAL-BENCH ABLATION · p. 18
  • Without failure classes, screening, and task-kind scope there was no validation gain; fixed classes collapsed failures together, while localization left 55% unassigned and trace-free edits obscured attribution.The unscreened mechanism also lost checks on a different task kind, and a repair round lowered another task’s score from 17 to 14 of 20 checks.Source: C TERMINAL-BENCH ABLATION · p. 18

D THE VESTRUM LOOP ON TERMINAL-BENCH 4 HARD: STEPS AND MAIN PROMPTS

  • In the reported epoch, the full bundle gained 16 checks and lost one on development, then gained 11 checks on validation before the harness was frozen.Inactive or harmful mechanisms were disabled or dropped before the pruned bundle was reevaluated.Source: D THE VESTRUM LOOP ON TERMINAL-BENCH 4… · pp. 19–20
  • On test, Vestrum passes 105.5 of 150 checks on average across draws, against Claude Code’s 95.5, at 1.03× test cost.Four task means improve, four are unchanged, and none declines.Source: D THE VESTRUM LOOP ON TERMINAL-BENCH 4… · p. 20

E OVERHYPOTHESIS, CATEGORIZATION, AND GENERALIZATION SCREENING

  • Task kinds define possible transfer scope and trace-readable failure classes identify behavior that may recur; screening tests a mechanism’s formulation, not whether it improves unseen tasks.On Terminal-Bench, proposals see test tasks’ kinds but not their solutions, tests, traces, or outcomes; procedural knowledge may be keyed to request terms, but development-task answer facts may not enter mechanisms.Source: E OVERHYPOTHESIS, CATEGORIZATION, AND… · p. 20

E.1 PERMISSIBLE SUPERVISION BY SETTING

  • Permitted supervision varies by setting: PathBench uses labels from two development sections for transfer to held-out sections of the same slide, while Terminal-Bench test solutions and outcomes remain unavailable to proposals.String lint and an LM reviewer do not establish semantic separation from development tasks.Source: §E.1 · p. 21

F NEAREST NEIGHBOURS: ADAMAST AND HARNESSFIX

  • Vestrum connects failure classes to task-kind scope, screens mechanism generality, and evaluates composed bundles; the comparison with AdaMAST and HarnessFix is methodological, not an empirical ranking.The related-work comparison identifies AdaMAST [Cemri et al., 2026] and HarnessFix [Chen et al., 2026a].Source: F NEAREST NEIGHBOURS: ADAMAST AND… · p. 21, Table 9 · p. 21

G VALIDATION PROTOCOL ON ULTRAHORIZON

  • Across four epochs and 21 fresh validation seeds, the loop considered 59 proposals and admitted two bundles, which together contain the UltraHorizon mechanisms.Each validation seed was used once, and only the decision and mean score returned to the loop.Source: G VALIDATION PROTOCOL ON ULTRAHORIZON · p. 21

H MEMORY BENCHMARK SETUPS

The memory benchmark sections describe the evaluation tasks, judging protocol, and detailed records for UltraHorizon, LoCoMo, and AMA-Bench. The supplied AMA-Bench comparison reports a Vestrum advantage over A-Mem in the final three-way run.

H MEMORY BENCHMARK SETUPS

  • UltraHorizon tests long-interaction discovery of environment rules, LoCoMo tests recall over extended conversations, and AMA-Bench tests memory after tool-using trajectories.LoCoMo evaluates multi-hop, temporal, open-domain, and single-hop questions; its adversarial unanswerable category is excluded.Source: H MEMORY BENCHMARK SETUPS · p. 22

I JUDGE PROTOCOL

  • Memory answers are graded by blinded Sonnet judgments against fixed rubric facts, with three independent passes and majority vote; LoCoMo pass agreement ranges from 0.947 to 0.955 across runs.AMA-Bench recorded 663 unanimous judgments out of 720.Source: I JUDGE PROTOCOL · p. 22

J.1 ULTRAHORIZON: FULL SEED RECORD

  • Table 10 reports each UltraHorizon seed’s mean score over 42 test episodes.Each seed contains the same 42 test episodes per system.Source: Table 10 · p. 22, §J.1 · p. 22

J.2 LOCOMO: FULL SEED RECORD

  • On LoCoMo, each run evaluates 956 test probes per system; across runs, Vestrum uses fewer calls than GAM but at higher recorded cost.The statistical unit is the conversation, and the planner split 12 of 152 development questions and 11 of 152 validation questions.Source: §J.2 · p. 22, Table 11 · p. 23

J.3 AMA-BENCH: FULL THREE-WAY AND REPLICATION RECORD

  • In the final AMA-Bench three-way run, Vestrum exceeds A-Mem by +0.179 strict, with 13/4/3 episode wins, losses, and ties and p = 0.0490 two-sided.The comparison uses the final three-way run’s 240 shared probes.Source: §J.3 · p. 23, Table 12 · p. 23
  • Table 13 preserves both paired run comparisons; in the second run used for the three-way comparison, Vestrum’s result against A-Mem is +0.179 strict, with 13 wins, 4 losses, 3 ties (two-sided p = 0.0490).Source: §J.3 · pp. 22–23

J.4 MODEL USAGE

  • The recorded usage includes 5,053 and 4,872 calls for UltraHorizon seeds 2 and 3, and 713 and 724 calls for the AMA-Bench Vestrum runs.LoCoMo used 61,894 calls and cost $163.30 across systems and seeds.Source: §J.4 · p. 23

K PATHBENCH GRADING

PathBench evaluates an agent’s cell-type assignments against a laboratory annotation across held-out sections of one slide. The reported section–class pair counts improve over Claude Code, while the endpoint remains agreement with an expert-in-the-loop reference.

K PATHBENCH GRADING

  • PathBench asks the agent to filter and cluster cells, then assign names from sixteen cell types; testing scores about 129,000 cells from four held-out sections.The development sections contain 55,366 reference cells, and no test-section label or measurement was read before the harness was frozen.Source: K PATHBENCH GRADING · p. 23
  • ARI and NMI assess clustering independently of label names, while accuracy and macro-F1 assess assignments; agreement is with expert-in-the-loop annotation, not independently established ground truth.Unresolved labels count as wrong, so declining to subtype earns no abstention credit.Source: K PATHBENCH GRADING · p. 23
  • Claude Code and Vestrum average 18.0 and 27.5 of 30 pairs, respectively, and on test average 43.5 and 51.5 of 60 pairs, respectively.Source: K PATHBENCH GRADING · p. 24

L GEPA COMPARISON AND RESOURCE ACCOUNTING

Vestrum scores above GEPA in every reported paired evaluation, though the comparison uses different search spaces, task models, token use, and stopping conditions. The appendix also lists admitted mechanisms across five settings and cautions that bundle admission does not establish each mechanism’s contribution.

L GEPA COMPARISON AND RESOURCE ACCOUNTING

  • Vestrum scores above GEPA in every reported paired evaluation; GEPA plateaued on UltraHorizon without improving GAM’s prompt.On AMA-Bench, GEPA improved over GAM by a pooled net +37 questions, while Vestrum’s further margin was concentrated in state abstraction, averaging 81% against 57%.Source: L GEPA COMPARISON AND RESOURCE… · p. 24, Table 15 · p. 25
  • Under the stated accounting, Vestrum used approximately a quarter of GEPA’s task-agent evaluations on LoCoMo and AMA-Bench and the same number on UltraHorizon.The totals use an allowance of 1.5 passes per epoch for pruning and retesting; this is an accounting convention, not an exact execution log.Source: L GEPA COMPARISON AND RESOURCE… · p. 25, Table 17 · p. 25
  • The comparison concerns task executions rather than total compute, and search spaces, task models, token use, and stopping conditions differ.Question-level sign-test values are optimistic because questions within a conversation or episode share evidence; the comparison also does not isolate structural changes or establish superiority over methods with the same editable surface.Source: L GEPA COMPARISON AND RESOURCE… · pp. 25–26

M ADMITTED MECHANISMS IN FULL

  • Terminal-Bench’s edit gate requires an invariant and probe script before the first shipped-code change, while repair procedures specify what to record.A scale decision records the chosen branch before execution.Source: M ADMITTED MECHANISMS IN FULL · p. 26
  • UltraHorizon’s harness organizes observations into evidence rows and rule-class notebooks, with contradiction analysis, probe plans, and a replay-before-commit gate where enabled.The gate requires replay before accepting a rule where it is enabled.Source: M ADMITTED MECHANISMS IN FULL · p. 26
  • LoCoMo’s planner decides whether to split questions; insufficient evidence triggers another retrieval round, followed by a claim judge at answer merge.Answer rules put strongest evidence first and require a best guess rather than refusal.Source: M ADMITTED MECHANISMS IN FULL · p. 26
  • AMA-Bench’s bundle preserves notebook history, checks entries during construction, and supplies deterministic event, action, and object-timeline views.The development failures implicated notebook entries used by later probes.Source: M ADMITTED MECHANISMS IN FULL · p. 26
  • PathBench’s knowledge-synthesis bundle supplies eleven sourced rules, marker and expression reference tables, and a nearest-neighbour helper for small unlabelled clusters.The rules begin by naming each cluster by its lineage; the bundle was admitted in one epoch with one iteration.Source: M ADMITTED MECHANISMS IN FULL · p. 26

N THE VERIFICATION RECORD

The verification record compares retained checks with alternatives across AMA-Bench, LoCoMo, UltraHorizon, and Terminal-Bench. Evidence-grounded checks at intermediate commitments could help, while broader audits of finished answers sometimes regressed; Figure 2 depicts admitted bundles, not individual mechanism effects.

N THE VERIFICATION RECORD

  • On AMA-Bench, a broad raw-trajectory recheck rewrote 82 of 120 development answers, repairing 5 and breaking 9, while a gated check was net positive with 6 repaired and 4 broken.The final design moved verification from the answer to the notebook; the gated check was not evaluated again.Source: N THE VERIFICATION RECORD · pp. 26, 28
  • On LoCoMo, the merge-only pipeline led alternatives by four answers net, while the retained claim judge ran on 12–13% of test questions and edited claims on 4–5%.The merge-only pipeline also included planner rules derived from the same conversation, so it split 12 questions while alternatives split 48 to 53.Source: N THE VERIFICATION RECORD · p. 28, Table 20 · p. 29
  • On UltraHorizon, the retained commit gate held answers in 12 of 15 diagnostic episodes, but a sufficiency gate held every development answer and lost 10 points.A gate needs evidence that separates rival rules to support progress in these runs.Source: N THE VERIFICATION RECORD · p. 29
  • Terminal-Bench experiments found that self-graded checks could mirror fixes or relabel failures, motivating checks supported by an independent draw or deterministic hook.The admitted edit gate’s separate effect was not measured.Source: N THE VERIFICATION RECORD · p. 29

O FIGURES

  • Figure 2 groups admitted mechanisms by design dimension and starting harness, but its scores summarize whole bundles rather than individual mechanism effects.The top row starts from GAM and the bottom row from Claude Code.Source: O FIGURES · p. 29, Figure 2 · p. 30
Loading 2609.33822v1…