PaperlayerView exact revision on arXiv

Source-linked AI summary

DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneck

Shubham Agarwal, Akshat Gupta, Mike Heddes, Dilara Soylu, Rui Meng, Patrick Tser Jern Kon, Jiefeng Chen, Mert Cemri, Shu Liu, Abe Gonzalez, Akanksha Jain, Bhavana Dalvi Mishra, Linfeng Song, Vilobh Meshram, Xiangru Tang, Jinsung Yoon, Aditya Parameswaran, Ion Stoica, Vahab Mirrokni, Burak Gokturk, Parthasarathy Ranganathan, Tomas Pfister, Nanyun Peng, Chun-Liang Li

arXiv:2610.17184v1cs.CL

TL;DR

Expensive true evaluations limit how many search iterations engineering optimization can afford. DuoDiscover has coding agents co-discover targets and cheap yardsticks, then checks yardstick fidelity against true evaluations. Across repositories, kernels, systems, and chip design, it reports gains including 1.65× ArchAgent v2’s gain on CPU microarchitecture in 16.7 days.

  • Problem

    Real engineering evaluations can take hours, limiting autonomous discovery to a handful of true evaluations.

    Source: §1 · p. 2

  • Method

    DuoDiscover uses coding agents to co-discover code targets and cheap yardsticks, climb yardstick scores, and test yardsticks against true evaluations.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    Across repositories, GPU/TPU kernels, systems, and chip design, DuoDiscover reports gains; on CPU microarchitecture it achieves 1.65× ArchAgent v2’s gain in 16.7 days.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Co-discovery provides a path toward scaling autonomous discovery across science and engineering.

    Source: §6 · p. 11

    When no faithful yardstick can be built, DuoDiscover falls back to expensive true-objective search, and warmup overhead can make it less advantageous under extremely short evaluation budgets.

    Source: §6 · p. 11

Abstract

from arXiv · show

Evaluation has been central to advances in AI, and it is equally critical to autonomous discovery. Yet scaling autonomous discovery to real-world engineering challenges faces a fundamental bottleneck: the cost of evaluating progress. In real settings, a single true evaluation, such as a full benchmark or serving trace, can take hours, so a search can afford only a handful of true evaluations. We present DuoDiscover, a search system that jointly discovers what to optimize and how to evaluate progress efficiently, through three mechanisms. First, each worker in a fleet of coding agents co-discovers an optimization target and a fast, tailored proxy evaluator, which we call a yardstick. The agent then hill-climbs the yardstick rather than the true evaluation. Second, since search can exploit gaps between a yardstick and the true objective, fidelity tests continually recheck each yardstick's ranking and invalidate any that drift. Third, the fleet is bottom-up, with no central decomposition of the problem. A lightweight orchestrator spawns and terminates agents, briefs running ones, and seeds new agents with others' confirmed work. DuoDiscover enables autonomous optimization from software to chip architecture. We demonstrate DuoDiscover on two repository-level benchmarks and four GPU and TPU kernel suites covering hundreds of kernels. On SWE-fficiency, it doubles the average speedup of human experts. On SOL-Exec, it ranks first on 121 of 235 kernels and in the top three on 212. On the heavily optimized vLLM and SQLite engines, DuoDiscover achieves 1.92$\times$ and 1.38$\times$ gains over the base code. On a CPU microarchitecture optimization task with evaluations of up to 28 hours, DuoDiscover surpasses both AlphaEvolve-based ArchAgent v2 and the best human design, with 1.65$\times$ ArchAgent v2's gain in 16.7 days versus more than 40 days, advancing the frontier of autonomous chip design.

1. Introduction

Expensive true evaluations limit autonomous discovery, while proxy-guided search requires finding both a useful target and a faithful evaluator. DuoDiscover co-discovers targets and yardsticks, checks yardstick fidelity, and uses a bottom-up agent fleet; reported results span software through chip architecture.

1. Introduction

  • True evaluations can bottleneck engineering search: vLLM serving evaluations take 85 minutes, allowing frontier coding agents only about a dozen evaluations in 20 hours.The paper frames cheaper evaluation as key to scaling autonomous discovery across science and real-world engineering.Source: §1 · p. 2
  • Proxy evaluators are target-specific, so useful search must find both a promising target and a cheap, faithful proxy; proxy drift can reward edits without real gains.A proxy that initially tracks the true objective can become unreliable as search exploits discrepancies.Source: §1 · p. 2
  • DuoDiscover pairs target and yardstick co-discovery with repeated fidelity tests and a bottom-up fleet coordinated by an orchestrator.The figure depicts cheap yardstick climbing alongside slower true-evaluation confirmation and yardstick validation or invalidation.Source: §1 · p. 2, Figure 2 · p. 3
  • Across repositories, GPU/TPU kernels, real-world systems, and chip architecture, DuoDiscover reports autonomous optimization; on CPU microarchitecture it achieves 1.65× ArchAgent v2’s gain in 16.7 days versus more than 40 days.The CPU evaluation takes up to 28 hours.Source: §1 · p. 2

2. DuoDiscover: Co-discovering targets and yardsticks

DuoDiscover treats optimization with expensive true evaluations as a search over both code regions and cheap ways to score progress. Independent agents choose targets and yardsticks, climb yardstick scores, and rely on true evaluations and fidelity tests to confirm gains and detect drift.

2.1. Problem: Optimization under expensive true evaluation

  • When true evaluations cost minutes to hours, agents have only tens of evaluations under a shared wall-clock budget and seek the state with the highest true score.A failed correctness check maps the true objective to a failure value.Source: §2.1 · p. 3

2.2. Co-discovery: Targets and their yardsticks

  • Each off-the-shelf LLM coding agent performs profiling, target selection, yardstick construction, and edits, while the orchestrator handles measurement and coordination.Agents choose their own code region; unlike AlphaEvolve, DuoDiscover does not require user-marked begin and end markers.Source: §2.2 · p. 3
  • Agents jointly choose a code target and a much cheaper yardstick whose edit ranking should match the true objective; yardsticks may score representative cases or use a cost model.For vLLM, an agent used a seconds-long decode-step-time model for scheduler prefill batching while the true evaluation took 85 minutes.Source: §2.2 · p. 4

2.3. Bottom-up search and coordination

  • The fast loop keeps edits that improve yardstick scores, while the slow loop uses true evaluations to confirm reported gains.The two-loop setup spends most search time on the yardstick.Source: §2.3 · p. 4
  • The orchestrator coordinates rather than centrally decomposes the problem: it terminates stalled agents, spawns agents on promising or uncovered targets, briefs workers, and transfers or combines confirmed work.Combined changes are kept only when a true evaluation confirms them.Source: §2.3 · p. 4

2.4. Fidelity testing: Validating yardsticks

  • After each true evaluation, the orchestrator retests yardsticks because search can exploit gaps between proxy scores and the true objective.A yardstick is validated when its ranking correlation clears the threshold, and invalidated if correlation later drops; the stated threshold is Kendall’s τ ≥ 0.5 over resolvable pairs.Source: §2.4 · pp. 4–5

3. Results

Section 3 evaluates DuoDiscover on repository and GPU/TPU kernel benchmarks, then on real-world software and CPU architecture tasks. It reports higher benchmark scores than all baselines and gains across the case studies, including a prefetcher design that outperforms prior human and system designs.

3.1. Benchmark results

  • DuoDiscover reaches 2.16× on FormulaCode, 21% above the strongest agentic baseline, and 15.02× on SWE-fficiency versus expert patches’ 6.90×.On the sampled SWE-fficiency tasks, it also doubles confirmed gains to roughly 12 per task and passes Claude Code’s 8-hour score after 2.5 hours.Source: §3.1 · pp. 5–6
  • Under the same wall-clock budget, DuoDiscover scores above every baseline on every repository and kernel benchmark.Table 1 reports the benchmark results under an 8-hour, four-agent budget, with stated exceptions for long FormulaCode tasks and TPU-Inference.Source: §3.1 · p. 5, Table 1 · p. 6
  • On SOL-Exec, DuoDiscover averages 79.74% SOL and ranks first on 121 of 235 kernels, second on 61, and third on 30.The reported leaderboard ranking is based on its state at submission.Source: §3.1 · p. 6
  • JAXBench reaches 4.62× over XLA, TPU-Inference 2.68× over tuned Pallas kernels, and FA4 11.97% geometric-mean speedup over its reference.The FA4 result covers 24 B200 configurations.Source: §3.1 · p. 7

3.2. Real-world case studies

  • For vLLM, DuoDiscover reaches 1.92× geometric mean versus Claude Code’s 1.07×; for FreeToken, it reaches 1.44× versus 1.05×.The FreeToken geometric mean combines 1.17× at batch 1 and 1.76× at batch 8.Source: §3.2 · pp. 7–8
  • SQLite reaches 1.38× versus Claude Code’s 1.12× with 17 versus 45 true evaluations; zstd reaches a 1.16× instruction ratio versus 1.05× with 7 versus 21 evaluations.The reported true-evaluation counts are DuoDiscover first, then Claude Code.Source: §3.2 · p. 8
  • On CPU prefetcher design, DuoDiscover gains 6.3% over the reference core, surpassing ArchAgent v2’s 3.8% and the best human design’s 3.5% in 16.7 days.ArchAgent v2 evaluated candidates over more than 40 days; DuoDiscover’s reported gain is the geometric mean across three configurations.Source: §3.2 · p. 8

4. Analysis

DuoDiscover improves search efficiency while maintaining confirmed gains, low run-to-run variance, and gains across coding-agent backbones. Ablations show that fidelity testing, co-discovery, and orchestrator actions contribute to the full system’s results.

Yardstick Quality

  • Yardsticks score edits 22× faster on repositories and 53× faster on kernels, with median Kendall τ of 0.66 and 0.76, respectively.Rank drift invalidates 41% of repository yardsticks and 45% of kernel yardsticks, but every reported gain is confirmed by a true evaluation.Source: §4 · p. 9

Search Efficiency

  • DuoDiscover builds yardsticks in its first hour and passes Claude Code’s score on repositories after a median of 2.5 hours.On kernels, it reaches Claude Code’s 69.12% score in 2.8 hours.Source: §4 · p. 9, Figure 4 · p. 9
  • 80.80% SOL is reached in 4.4M tokens and 54 true evaluations; DuoDiscover leads on 41 of 50 kernels.It matches Claude Code’s 69.12% in 2.8 hours with 0.93M tokens and 13 true evaluations, using 74% fewer tokens and 83% fewer true evaluations.Source: §4 · p. 9, Figure 4 · p. 9

Variance across Discovery Runs

  • Across three seeds per task, final scores stay within ±1.12% of the per-kernel mean and ±1.8% of the per-task mean, with 78% of kernel gains and 81% of repository gains reproducing.Claude Code’s corresponding variation is ±3.27% and ±5.21%.Source: §4 · p. 9

Component Ablations

  • 14.93× repository speedup and 80.80% SOL are the full system’s results after fidelity testing, co-discovery, and the orchestrator are added.The orchestrator’s addition is the largest single kernel step; removing any one action costs 0.63–1.01× repository speedup and 2.3–4.4 kernel points.Source: §4 · p. 9, Table 4 · p. 10, Table 5 · p. 10

Backbone Coding-Agent Generalization

  • 1.30× repository and 1.25× kernel gains with Antigravity/Gemini 3.8 Flash exceed the 1.16× and 1.17× gains with Claude Code/Opus 5.With DuoDiscover, Antigravity nearly matches Claude Code on repositories and surpasses it on kernels; the run is 5.1× cheaper.Source: §4 · p. 10, Table 6 · p. 10

5. Related work

Prior discovery work often treats evaluation as fixed, while related approaches use generated tests or multi-fidelity optimization to guide search. DuoDiscover instead has agents synthesize cheap, target-specific evaluators and validate or invalidate them during discovery.

Coding and discovery agents

  • Prior discovery systems span mathematics, algorithms, and systems optimization, but generally refine the search harness while treating the evaluator as fixed.Examples include FunSearch and AlphaEvolve, alongside later work on efficiency, mutation strategies, test-time reinforcement learning, and local minima.Source: §5 · p. 10
  • DuoDiscover is presented as the first discovery agent to find both solutions and cheap evaluators that are validated or invalidated during discovery.SIFT instead uses expensive evaluations only for the most promising solutions; prior AlphaEvolve workflows can specify cheaper checks in advance.Source: §5 · p. 11

LLM-generated tests and evaluators; Multi-fidelity optimization

  • Unlike LLM-generated correctness tests, DuoDiscover discovers cheap, target-specific yardsticks for optimization.Unlike common multi-fidelity methods based on surrogate models or adaptive resource allocation, its agents synthesize proxies and validate or invalidate them during search.Source: §5 · p. 11

6. Conclusion

DuoDiscover frames evaluation efficiency as part of autonomous discovery: agents co-discover targets and yardsticks in a bottom-up search system. Its conclusion spans repository benchmarks, kernel suites, and real-world case studies, while noting that expensive-objective fallback and warmup overhead limit its advantage in some settings.

6. Conclusion

  • Across two repository benchmarks, four kernel suites, and five real-world case studies, DuoDiscover demonstrates optimization from software to chip architecture.The conclusion argues that co-discovery opens a path toward scaling autonomous discovery across science and engineering.Source: §6 · p. 11
  • DuoDiscover’s two primary limitations are fallback to expensive-objective search when no faithful yardstick can be built and warmup overhead under extremely short evaluation budgets.The overhead comes from constructing yardsticks and running initial fidelity tests.Source: §6 · p. 11

A. Method details

The appendix defines the paper’s target, yardstick, co-discovery, fidelity test, and orchestrator, then details how yardstick fidelity is assessed and how benchmarks and baselines are documented.

A. Method details

  • A target is the code region an agent optimizes, a yardstick is its cheap scorer, and co-discovery means choosing both together.The orchestrator coordinates agents and runs the fidelity test against the true objective.Source: A. Method details · p. 17

A.1. Fidelity test details

  • Fidelity is assessed using Kendall’s τ over resolved state pairs, excluding yardstick ties; state pairs count only when true-score differences exceed the measured noise threshold.Noise is estimated from repeated true evaluations, with a minimum 1% separation; validation requires at least six resolved states, τ ≥ 0.5, and a one-sided permutation test rejecting independence at α = 0.05.Source: §A.1 · pp. 17–18

B. Benchmark and baseline details

  • The appendix lists each benchmark’s task type, score, comparison baseline, task count, budget, and true-evaluation time, and describes how baselines were run.The data-prefetcher study is an exception: it uses three agents, parallel traces, and baselines as reported rather than rerun.Source: Table 8 · p. 18, B. Benchmark and baseline details · p. 18

B.1. Benchmarks

The appendix specifies benchmark hardware, budgets, and baseline conditions, then reports results across repository and kernel suites. DuoDiscover leads the reported FormulaCode, SOL-Exec, and FA4 comparisons.

B.1. Benchmarks

  • The evaluation setup spans NVIDIA B200 and TPU v6e kernels and repository tasks on 16-vCPU/32-GB or 2-vCPU/4-GB nodes.Benchmark budgets are generally eight hours, with exceptions including 24 hours for some FormulaCode tasks and 10 hours on TPU-Inference.Source: §B.1 · pp. 18–19

C.1. FormulaCode

  • DuoDiscover beats the other baselines on 16 of 25 FormulaCode tasks, with 21% higher mean speedup across the selected tasks.Each task averages 264 performance workloads, and solutions are scored by geometric-mean speedup across those workloads.Source: §C.1 · p. 20

C.2. SWE-fficiency

  • Every DuoDiscover final patch passes tests on all 100 SWE-fficiency tasks.A failed test scores base speed, while 1.00 means no gain over the base code.Source: §C.2 · p. 20

C.3. SOL-Exec

  • DuoDiscover scores 79.74% mean SOL across 235 SOL-Exec kernels, versus 78.95% for the second entry, and ranks first on 121 kernels and in the top three on 212.Its lead comes from breadth across kernels; it is first on L1, L2, and FlashInfer and second on Quant.Source: §C.3 · p. 21

C.4. FA4

  • DuoDiscover is 11.97% faster than Flash Attention 4 across 24 configurations, ahead of KDA at 8.93% and Claude Code at 7.05%.The reported values are geometric-mean speedups over Flash Attention 4.Source: §C.4 · p. 21

C.5. JAXBench

  • JAXBench covers 17 priority kernels, each evaluated on 13 production-derived input shapes and scored by geometric-mean speedup over XLA.Candidates must pass a correctness gate requiring at least 98% of output elements to be within a data-type-specific tolerance of the XLA output.Source: §C.5 · pp. 21–22

C.6. TPU-Inference

The reported TPU-Inference suite results favor DuoDiscover overall, while the case studies describe search and solution details for vLLM, FreeToken, SQLite, and zstd.

C.6. TPU-Inference

  • DuoDiscover scores 2.68× across seven TPU-Inference kernels, ahead of Claude Code at 2.38×, PerfAgent at 2.14×, and OpenEvolve at 1.14×, and ranks first on five kernels.The reference is the optimized production Pallas implementation; the seven kernels include inference workloads from recent open frontier models.Source: §C.6 · pp. 22–23

D. Case study details

  • The data-prefetcher study is the case-study exception: it compares against ArchAgent v2 and the best human design, with a fidelity-test floor of two evaluated states.The other case studies use four Claude Code agents on Opus 5 as the baseline; fidelity tests use six resolved states on benchmarks.Source: D. Case study details · p. 23

D.1. vLLM

  • vLLM reaches a 1.92× geometric mean across four objectives, with p99 decode latency 2.94× lower, first-token latency 3.61× lower, and power 21% lower.The solution combines a per-step prefill cap, GPU clock control, and per-client fair scheduling; throughput is 1.01×.Source: §D.1 · p. 25

D.2. FreeToken

  • FreeToken reaches a confirmed 1.44× geometric-mean gain, while its batch-8 split-K kernel reduces the layer’s calls from 38.2 to 24.1 ms per decode step.The batch-1 gain comes from other edits; the expert-loop optimization saves nothing at batch 1 because each expert serves one token.Source: §D.2 · p. 26

D.3. SQLite

  • SQLite’s confirmed gain is 1.38×, against Claude Code’s 1.12×, after 17 true evaluations compared with 45.The combined edits pass the 13,249-test suite twice and preserve the unmodified build’s result hash.Source: §D.3 · p. 27

D.4. zstd

  • zstd reaches a 1.16× instruction-count ratio with identical output bytes, versus Claude Code’s 1.05×.The search evaluates instruction counts on 16 workload cells; a byte mismatch scores zero.Source: §D.4 · p. 28

D.5. Data-prefetcher design

The data-prefetcher search uses successive 12-hour agent runs seeded from prior results, while official evaluations proceed asynchronously. The appendix also describes additional evaluator and fleet-scaling analyses.

D.5. Data-prefetcher design

  • The final design reaches a 1.0631× overall speedup, exceeding ArchAgent v2 at 1.0382× and the best human design at 1.0352× (Table 2).Official evaluations ran asynchronously alongside subsequent runs and did not gate them.Source: §D.5 · p. 29
  • The design combines a rewritten L1D prefetcher, an extended L2 prefetcher, and a last-level-cache temporal correlator keyed by core on four cores.Agents moved from L1D-only work to L2 and LLC edits; adding 15 four-core mixes to the score raised MC from 1.0274× to 1.0393×, with FullBW and LimitBW unchanged.Source: §D.5 · p. 29

E. Ablation and scaling details

  • The appendix’s additional analyses cover a task-wide evaluator, fleet-size sweeps, and fleet-scaling curves over time.These analyses replace per-target yardsticks with a single task-wide evaluator in one ablation.Source: E. Ablation and scaling details · p. 29

E.1. Per-target yardsticks against a single task evaluator

The appendix tests task-wide evaluation against per-target yardsticks, examines fleet scaling, and analyzes yardstick quality and search outcomes.

E.1. Per-target yardsticks against a single task evaluator

  • A task-wide evaluator reduces repository scores from 14.93× to 12.98× and kernel SOL score from 80.80% to 71.36%.It passes fidelity tests on 21% of repository tasks and, where validated, costs 29% of a true evaluation; a task-wide workload trim can misrank edits across code regions.Source: §E.1 · pp. 29–30

E.2. Fleet scaling

  • At 16 agents, DuoDiscover reaches 85.96% SOL versus Claude Code’s 72.20%, and its lead widens at every tested fleet size.At one agent, the scores are 70.26% and 66.47%, respectively; the gap grows from 3.8 to 13.8 SOL across fleet sizes.Source: §E.2 · p. 30

E.3. Fleet scaling over time

  • DuoDiscover’s 16-agent fleet reaches its four-agent score of 80.80% in 3.4 hours, versus 7.6 hours for the four-agent run.The corresponding 8-agent run takes 5.0 hours; Claude Code’s 16-agent fleet reaches its four-agent score in 5.4 hours versus 6.8 hours.Source: §E.3 · p. 30

F.1. Yardstick quality

  • Yardstick fidelity is Kendall’s rank correlation, with validation requiring τ≥0.5 over at least six resolved states; shared true evaluations can serve multiple tests.A kernel run’s 54 true evaluations cover 11 validated yardsticks.Source: §F.1 · pp. 30–31
  • On kernels, 37.0% of DuoDiscover’s truly evaluated candidates are gains found on a yardstick, plus one combined edit, versus 21.1% for Claude Code.DuoDiscover spends 2.6 true evaluations per confirmed kernel gain versus 4.8 for Claude Code; unvalidated kernel yardsticks have median τ of 0.03.Source: §F.1 · p. 31

F.2. Search behavior

DuoDiscover’s search behavior combines agent-led target and yardstick discovery with fidelity checks and orchestrator-led exploration and exploitation. In an 8-hour SOL-Exec run, it matched Claude Code’s final score in 2.8 hours at 25.8% of the cost of Claude Code’s full run.

F.2. Search behavior

  • Agents pursued 15 targets per kernel, validating 11 yardsticks on average, and 9.2 targets per repository task, validating 5.3.Yardsticks used reduced inputs, profiling harnesses, filtered test subsuites, or static cost models; two of 20 repository tasks had no validated yardstick.Source: §F.2 · p. 31
  • DuoDiscover finished ahead of Claude Code on 41 of 50 kernels, level on 5, and behind on 4.Source: §F.2 · p. 32
  • Agents tried 285 candidates per kernel and 358 per repository task; yardsticks screened out 81% and 84%, respectively, before true evaluation.Fidelity checks invalidated 45% of validated kernel yardsticks and 41% of repository yardsticks after ranking drift.Source: §F.2 · p. 32
  • Confirmed gains were substantial rewrites: 98% of kernel gains changed over 100 lines, while repository patches changed a median of four to five files and about 800 lines.Source: §F.2 · p. 32
  • The orchestrator limited agents pursuing the same or related target to 5–15% on kernels and 18% on repositories, then shifted effort toward targets with confirmed gains.Unsuccessful targets consumed 19% of true evaluations on kernels and 20% on repositories.Source: §F.2 · pp. 32–33

F.3. Cost of a run

  • 2.8 hours matched Claude Code’s final 8-hour score at $79.60, or 25.8% of Claude Code’s full-run cost.A full 8-hour DuoDiscover run cost $375 versus $309 for Claude Code; GPU time for true evaluations is excluded from the cost table.Source: §F.3 · p. 33, Table 17 · p. 33
  • DuoDiscover’s four-agent run used 1,990 turns, versus Claude Code’s 1,640, with context compaction occurring 68 and 56 times, respectively.Each DuoDiscover turn averaged 2.2k generated tokens, 130k cache-read tokens, and 4.8k appended tokens.Source: §F.3 · p. 33

F.4. Examples of discovered targets and yardsticks

The examples show agents pairing target-specific yardsticks with edits, then combining confirmed improvements. They also show that some promising yardstick results fail true evaluation or cannot be merged directly.

  • 77.4% SOL was reached on the mixture-of-experts kernel by combining a tiled projection, fused activation, and in-place GEMM after resolving conflicting scratch-buffer reuse.The reference scored 9.8% SOL; an occupancy sweep for routing showed no true-evaluation gain and was dropped.Source: §F.4 · pp. 33–34
  • 71.5% SOL was achieved on grouped-query attention with rotary embeddings after a 128-by-32 unified tiling and split causal-pass loops raised one edit to 39% SOL.A reduced-input microbenchmark yardstick was validated against true evaluations at hour 1.1.Source: §F.4 · p. 34
  • 51.4% SOL was reached on the GEMM kernel from a 48% reference by sweeping tile sizes and pipeline depths across M ranges with a yardstick about 70× faster than true evaluation.The yardstick passed fidelity testing after six true evaluations.Source: §F.4 · p. 34
  • 4.45× over base code was confirmed for the SciPy FFT task, versus 3.89× for Claude Code, after three prime-length and divisibility optimizations were merged.Four of six yardsticks passed fidelity testing.Source: §F.4 · p. 34

F.5. Robustness

Robustness tests cover searches without reliable yardsticks, deliberately misleading yardsticks, noisy true evaluations, and sensitivity to test settings and random seeds.

  • On average, 4 of 15 kernel targets and 2 of 20 repository tasks had no validated yardstick, leaving search to use true evaluations alone until the orchestrator moved the agent on.Source: §F.5 · p. 35
  • A planted misleading yardstick was rejected within 3 to 5 true evaluations, while the fidelity test incorporated noise measured from the run to avoid treating timing jitter as a false gain.Source: §F.5 · p. 35
  • Sweeping α from 0.01 to 0.10 kept confirmed scores within a 1.4% band, and 78% of kernel gains and 81% of repository gains recurred in at least two of three runs.Source: §F.5 · p. 35
Loading 2610.17184v1…