PaperlayerView exact revision on arXiv

Source-linked AI summary

Praxist: From Experimental Artifacts to Solution Lineages

Jin Li, Ahmed Murtadha, Zhiyu Wang, Qiwen Chen, William Chen, Yifei Wu, Guan Wang, Andy L. Siy, Jiayi Yang, Mengsha Huang, Wenhao Li, Yixuan Liu, Shuailin Pan, Mingli Yuan, Sen Song, Yuhao Sun

arXiv:2608.25955v1cs.MAcs.SE

TL;DR

Autonomous R&D campaigns retain only a limited portion of evaluated experience in forms later attempts can build on. Praxist turns evaluated artifacts into typed, inheritable evidence and a solution lineage; on MLE-bench, it earns 60 medals (80.0%) across 75 tasks versus 55 (73.3%) for Claude Code on Claude Opus 4.8.

  • Problem

    Campaigns generate extensive evaluated experience, yet retain only a limited portion in a form that later attempts can build on.

    Source: §1 · p. 2

  • Method

    Praxist converts evaluated artifacts into typed, inheritable evidence and uses cohort synthesis to organize findings into inheritance lanes and agendas for later generations.

    Source: §1 · p. 3, §2.1 · p. 5

  • Results

    60 medals (80.0%) across all 75 MLE-bench tasks, including 49 gold, versus Claude Code on Claude Opus 4.8's 55 medals (73.3%) and 34 gold.

    Source: §4 · p. 26

  • Takeaways & Limitations

    Inspectable solution lineages can support reuse and extension by human scientists; the four case studies report those lineages in detail.

    Source: §4 · p. 27

    On 70 tasks with scores from both arms, Claude Code wins raw score on 36 versus PRAXIST's 33, so PRAXIST did not win most head-to-head score comparisons.

    Source: §3.2.3 · p. 13

Abstract

from arXiv · show

Autonomous R\&D agents now write, run, and improve executable artifacts under automated evaluation---but largely as laboratory instruments: shown on curated benchmarks, with gains that are hard to trace to a cause and costs well above what sustained engineering practice absorbs. The limitation is structural. Most systems treat each attempt as nearly self-contained, so logs, memories, and search trees record what happened without establishing which design element produced an improvement, whether its evidence survived validation, or how it recombines with others. Long campaigns therefore keep re-learning the same lessons. We introduce Praxist, a lineage-centered generational system that converts reproducible artifacts and evaluator outcomes into a typed evidence graph of findings, lane-structured frontiers, and agendas. Separating local artifact construction from cohort-level evidence synthesis lets later attempts inherit validated mechanisms, unresolved claims, and useful constraints, and leaves results attached to an inspectable lineage. On the standardized 75-task MLE-bench suite, the finalized official-grader results give Praxist 60 medals (80.0\%), 49 of them gold, against 55 medals (73.3\%) and 34 gold for a Claude Code baseline on Claude Opus 4.8---at a recorded model spend of US\$3,054 versus US\$38,370, roughly a twelfth of the cost. Four case studies---quantitative trading, LiDAR-inertial-visual SLAM, tokamak magnetic control, and rocket landing---carry the same process into open-ended engineering problems, improving on each task-native baseline in headline accuracy, survival, or resource cost, with the discovery path on record. Stronger artifacts at an order of magnitude less spend, each backed by an auditable lineage, are, to our knowledge, first brought together here: the operating profile production research requires, not the one a benchmark demonstration establishes.

1 Introduction

PRAXIST turns evaluated artifacts into inheritable evidence and an inspectable lineage, so later attempts can build on validated findings rather than treating each attempt as self-contained. Its evaluation spans all 75 MLE-bench tasks and four open-ended engineering case studies.

1 Introduction

  • Because compositional artifact search can prune mechanisms before their value emerges, long campaigns need to retain and recombine evaluated evidence rather than merely store it.PRAXIST frames this as evidence inheritance: selectively retaining reusable, artifact-grounded parts, as assembly theory describes.Source: §1 · pp. 2–3
  • PRAXIST separates cohort-level evidence synthesis from parallel artifact construction, turning evaluator outcomes into typed findings, frontier lanes, agendas, Gems, and an active lineage graph.Generations begin with design contracts allocated across mechanism family, intervention surface, and intent; the next agenda can continue, stop, validate, or explore.Source: §1 · p. 3
  • On 75 MLE-bench tasks, PRAXIST earned 60 medals (80.0%) and 49 golds versus Claude Code on Opus 4.8 with 55 medals (73.3%) and 34 golds.Across four open-ended case studies, it improved headline metrics in two domains, led on survival but not full-horizon precision in tokamak control, and cut visual-compute cost at unchanged accuracy in rocket landing.Source: §1 · p. 3
  • The evaluation compares outcomes, lineage process measures, and cost against local baselines across the benchmark suite and four open-ended case studies.The paper reports these settings as tests of PRAXIST’s results and its documented discovery process.Source: §1 · p. 3

2 Method

Praxist turns reproducible, externally evaluated artifacts into typed findings that are synthesized into inheritable evidence and agendas, while recording the research lineage alongside the final artifact.

2.1 Overview and Formalization

  • Praxist runs generational cohorts that turn reproducible artifacts into typed findings, promote evidence to a frontier, and use the frontier to direct later attempts.Evidence is inherited by operational role rather than as raw transcripts or scalar scores; global synthesis records what to build on, validate, avoid, or remember.Source: §2.1 · pp. 4–5

2.2 Inherited State and Design Allocation

  • Design allocation pairs selective inheritance with DIG contracts and QD diversity, directing peers to test specified mechanisms across distinct design cells.DIG fixes the intended intervention, evidence signature, validation hook, and forbidden changes before construction; QD distributes contracts across mechanism family, intervention surface, and intent.Source: §2.2 · pp. 6–7

2.3 Artifact Construction, Evaluation, and Findings

  • Evaluated artifacts become reusable, artifact-grounded findings labeled by type, evidence maturity, and recommended inheritance action, with failures retained as first-class evidence.The task’s external evaluator supplies outcomes and validity states; evidence stage is distinct from score, so a high preliminary score remains immature for promotion.Source: §2.3 · pp. 7–8

2.4 Global Synthesis: Frontier Update and Next Agenda

  • PI roles synthesize pooled findings into an agenda, while frontier lanes distinguish confirmed, candidate, diagnostic, and validation evidence and guide continue, stop, validate, or explore decisions.The PI panel uses complementary Builder, Skeptic, and Portfolio perspectives, with an External-validity role in high-stakes mode; task-defined lanes gate immature evidence from promotion.Source: §2.4 · pp. 8–9

2.5 Memory, Lineage, and Final Output

  • When enabled, periodic lane-balanced compression preserves durable lessons as bounded Gems, while an accumulated typed lineage links research objects and decisions to the final artifact.Gems are off by default and used only in the reported trading campaign; lineage is recorded during execution and explains why evidence became inheritable and how the outcome was reached.Source: §2.5 · pp. 9–10

3 Experiments

The experiments combine a standardized 75-task MLE-bench comparison with four open-ended R&D case studies evaluated by task-native measures. They report benchmark-wide medal gains and task-specific findings in landing, trading, tokamak control, and SLAM, with the latter results bounded by evaluation conditions.

3.1 Overview

  • Four open-ended case studies—rocket landing, quantitative trading, SLAM, and tokamak magnetic control—extend the evaluation beyond competition-style ML engineering using domain-native evaluators and task-grounded baselines.The rocket study also compares against Weco on the same task and success metric, but it is an external reference rather than a matched-model or matched-compute comparison.Source: §3.1 · p. 10

3.5 Case Study: SLAM

  • About 72% less evaluator-captured visual-path processing time across 14 sequences, with no trajectory-accuracy cost, came from observability-aware scheduling and geometry-aware map admission.The builds stamped poses differently, so accuracy columns are reported side by side rather than as a gain.Source: §3.1 · p. 11, §3.5 · pp. 19–20

3.2.1 Experimental Setup

  • MLE-bench covers 75 tasks graded by the official harness; PRAXIST’s finalized ledger excluded contaminated lineages, while baseline screening left five tasks unscored.PRAXIST used deepseek-v4-pro; the Claude Code baseline used Claude Opus 4.8. The PRAXIST campaign rejected 90,423 attempts and substituted clean submissions on nine tasks after excluding their previous best lineages.Source: §3.2.1 · pp. 11–12

3.2.2 Main Results

  • 60 medals (80.0%) and 49 golds for PRAXIST beat Claude Code + Opus 4.8’s 55 medals (73.3%) and 34 golds; spend was about US$3,054 versus US$38,370.PRAXIST exceeded the baseline on each tier: Low, 90.9% versus 81.8%; Medium, 81.6% versus 76.3%; High, 60.0% versus 53.3%. These are single-sweep outcomes, not multi-seed estimates.Source: Table 2 · p. 12, §3.2.2 · p. 12, §3.2.3 · p. 14

3.2.3 Analysis by Task Category

  • In a selected 19-task subset, PRAXIST scored better on 12 tasks and earned 11 golds versus Claude Code’s 5, though both systems medaled on 14.Across the 70 tasks with scores from both systems, Claude Code had the better raw score on 36, PRAXIST on 33, and one tied; PRAXIST’s five additional scores were unscored for the baseline.Source: §3.2.3 · pp. 12–13, Table 3 · p. 13

3.3 Case Study: Rocket (Reusable Rocket Landing)

  • 12,288/12,288 trajectories (100%) landed under the frozen protocol, versus 4.03% for the shared starting artifact and Weco’s reported 17.12%.The full-bank audit recorded 40,959, 40,959, and 40,960 successes out of 40,960 rows per bank; the banks were adaptively reused, so these rates are not real-world reliability estimates.Source: Table 4 · p. 16, §3.3 · pp. 16–17

3.4 Case Study: Quantitative Finance (Quant)

  • 53% calendar-time CAGR versus 23% for the paired baseline came from the discovered policy over 28 walk-forward quarters.The selected recurrent, execution-aware policy was a first-tier, single-seed artifact in the incubator lane with three hard constraint violations, not the cleanly promoted campaign artifact; under an additional 50 basis points per executed side, mean quarterly return remained positive at 7.09%.Source: §3.4 · p. 18, Table 5 · p. 19

3.5 Case Study: SLAM

  • COVSCHED conditions visual updates on LiDAR observability and a capped skip budget, while VMAP-DEDUP rejects nearby map re-observations.The schedule skips high-observability frames 74.9% of the time versus 0.3% for low-observability frames on nya_03, and caps consecutive skips at three; map candidates within 0.08 m in the same voxel and below 15° normal deviation are treated as re-observations.Source: §3.5 · pp. 20–22
  • COVSCHED’s lower raw mean APE RMSE than FAST-LIVO2 is not reported as an accuracy improvement, because pose-timestamp conventions confound the comparison.The raw means were 0.0937 m and 0.0501 m, respectively, but re-associating baseline poses under the COVSCHED rule leaves mean relative APE change at −0.09%. Coverage differs between arms (0.79× to 1.76×), and a same-source, same-timestamp paired rerun remains outstanding.Source: §3.5 · pp. 21–22, Table 6 · p. 21
  • 72.4% average reduction in amortized VIO processing time was measured across all fourteen SLAM sequences, with a 74.4% median reduction.The conservative aggregate proxy from LIO and VIO thread-wall workloads fell 22.1%; the asynchronous timing streams do not support interpreting the VIO reduction as end-to-end latency, throughput, or CPU-load reduction. Trajectory accuracy was not claimed to improve.Source: §3.5 · p. 21, Table 6 · p. 21

3.6 Case Study: Fusion (Tokamak Magnetic Control)

  • HybridJacobianPDV1 combines vertical PD and coil counter-drain, waypoint-ramped radial control, an ohmic-current loop, and sparse Jacobian-sign vectors.It closes nine axes versus six for the PCS-style baseline, using sparse sign vectors rather than dense damped pseudo-inverse feedback. A later matched control found randomly flipping half the identified signs scored 297 versus 298 aggregate steps, so the lineage recorded the signs as non-essential.Source: §3.6 · pp. 24, 26
  • The reported fusion precision comparison depends on horizon: full-horizon errors can be distorted by survival, while common-horizon scoring restores the expected controller ordering.Common-horizon precision excludes late-horizon steps where controllers differ most, so both metrics are reported. The reported seven-generation lineage’s survival-versus-precision frontier was later extended by controllers reaching up to 1,481 of 1,500 survival steps at worse full-horizon precision.Source: §3.6 · pp. 25–26
  • 1,264 of 1,500 steps survived under PRAXIST, versus 1,222 for the PCS-style controller and 1,039 for zero feedback; common-horizon WNRMSE p95 was 2.86 versus 2.99 and 4.89.The PCS-style controller completed more episodes (11 of 15 versus 10) and had lower full-horizon aggregate p95 (4.42 versus 4.65); none of the controllers passed the benchmark’s official classification rule. Both closed-loop controllers used target errors from the harness rather than probe observations.Source: §3.6 · pp. 24–25, Table 7 · p. 25

4 Conclusion and Discussion

PRAXIST turns evaluated artifacts into typed findings, lane-structured frontiers, agendas, and durable Gems so later attempts inherit evidence and retain an inspectable solution lineage. Its evaluation combines MLE-bench results with four case studies, while the conclusion points toward cross-campaign inheritance, slower or noisier evaluators, and deeper human–AI collaboration.

4 Conclusion and Discussion

  • 60 medals (80.0%) across all 75 MLE-bench tasks, including 49 gold, compares with 55 medals (73.3%) and 34 gold for Claude Code on Claude Opus 4.8.Of PRAXIST’s 60 medals, 49 are gold (81.7%).Source: §4 · pp. 26–27
  • The trading policy achieved a 53% walk-forward CAGR versus 23% for its paired all-eligible equal-weight baseline, while SLAM visual-path processing time fell 72.4% without costing trajectory accuracy.The SLAM result concerns evaluator-captured visual-path processing time and a resource-aware schedule replacing always-on visual updates.Source: §4 · p. 26
  • The tokamak controller achieved 3.4% higher aggregate survival and 4.4% lower common-horizon tracking error than a task-native MAST-U PCS-style controller in the same privileged-state harness.It did not outperform on the benchmark’s original full-horizon precision metric.Source: §4 · p. 26
  • The deterministic rocket controller landed 12,288/12,288 trajectories under a frozen first-contact protocol; Weco reports 17.12% from the same task, objective, and 4.03% starting artifact.Its lineage accumulated a fuel-commit governor, attitude-authority and phase-transition guards, and a closed-form constrained control allocator as typed evidence.Source: §4 · p. 26
  • PRAXIST’s generational cycle turns evaluated artifacts into typed findings, organizes them in lane-structured frontiers, and uses PI/Chair synthesis to produce agendas and compress durable lessons into Gems.The resulting lineage directs later attempts and records why the final artifact is credible; the four case studies detail their lineages, whereas MLE-bench reports graded outcomes per task, with ledger coverage varying by run duration.Source: §4 · pp. 26–27
  • The proposed next directions are inheritance across campaigns, extension to slower or noisier evaluators, and deeper human–AI collaboration that treats PRAXIST as an instrument for scientists.The suggested applications include executable-evaluator tasks in model and algorithm development, physical-system controller synthesis, simulation-driven engineering design, and quantitative strategy research.Source: §4 · p. 27

Code and Released Runs · Appendix

Praxist publicly releases its implementation, project page, and run archive, whose generational records make trajectories inspectable beyond aggregate scores. The appendix describes how task-family realization components vary across MLE-style and other tasks.

Code and Released Runs

  • The released runs preserve artifacts and evaluation records for inspection across generations.The archive also includes findings, frontier lanes, agendas, and lineage traces, allowing trajectories summarized in the paper to be inspected generation by generation.Source: Code and Released Runs · p. 27
  • Findings, frontier lanes, agendas, and lineage traces are included in the released generational record.These records support generation-by-generation inspection rather than relying only on aggregate scores.Source: Code and Released Runs · p. 27
  • The PRAXIST code is publicly available at github.com/sapientinc/praxist.Source: Code and Released Runs · p. 27
  • The PRAXIST project page is praxist.sapient.inc.Source: Code and Released Runs · p. 27
  • The released runs are available in a Google Drive archive.Source: Code and Released Runs · p. 27

Appendix

  • MLE-style tasks instantiate task-family components with competition context, submission artifacts, metric semantics, and grader outcomes.Other task families instantiate the components with their own artifact forms and evaluators.Source: Table 8 · p. 34

A Method details

Praxist’s method is specified as a generational state transition whose field-level contracts explain what information its roles represent, consume, and emit. The appendix focuses on methodological interfaces, not software mechanics, and uses MLE-style tasks only as an example.

A Method details

  • The state Sg = (Fg, Ag, Gg, Lg) links artifacts, evaluated findings, inheritance frontiers, and agendas for later generations.Artifacts ground attempts; findings interpret evaluated evidence; frontiers determine inheritance status; and agendas direct later generations.Source: A Method details · p. 34
  • Gems preserve durable lessons, while lineage records the path that produced the final artifact.These field-level contracts expand the objects using the main text’s symbols and vocabulary.Source: A Method details · p. 34
  • The appendix specifies represented information, consuming roles, and emitted outputs; command invocations, storage layouts, and release-specific engineering belong to implementation documentation or the experimental protocol.MLE-style tasks serve as a running example of a task-family realization, not as PRAXIST’s definition.Source: A Method details · p. 34

A.1 Task-Family Realizations

A task-family realization supplies the domain-specific context, artifact and evaluator definitions, role constraints, and evidence conventions needed to instantiate Praxist’s task-agnostic research cycle. This lets the same research-state objects support different evaluator-grounded tasks, provided evaluated artifacts yield traceable evidence.

A.1 Task-Family Realizations

  • A task-family realization instantiates the task-agnostic research cycle while allowing the same research-state objects to operate across evaluator-grounded tasks.It supplies the allowed task context, artifact form, evaluator semantics, role constraints, and evidence conventions.Source: §A.1 · p. 34
  • Realizations can define MLE competitions, proof-search tasks, software patches, or simulation bundles, with evaluated artifacts yielding traceable evidence.Examples specify the relevant data context, submission or artifact format, metric or checker, and evaluator outcome or report.Source: §A.1 · p. 34

A.2 Artifacts, Evaluation Records, and Findings … A.7 Lineage Trace

PRAXIST turns experimental artifacts and evaluator outcomes into typed findings, structured decisions, and inherited evidence, preserving a trace from prior work to the final reproducible artifact. Its lineage records how evidence shaped later experiments and supports end-to-end auditing.

A.2 Artifacts, Evaluation Records, and Findings

  • Each evaluated attempt links a reproducible artifact, task-grounded evaluation record, and interpreted findings that encode research claims with maturity and inheritance recommendations.These are conceptual records; task families may instantiate them with different concrete fields.Source: §A.2 · p. 35, Table 9 · p. 35
  • Committed artifacts alone support primary positive inheritance, while partial or failed attempts can yield diagnostics; findings may be positive, negative, diagnostic, uncertain, or procedural.Validity and evidence maturity are distinct from score: an invalid high score is diagnostic, not a confirmed improvement, and a high-scoring artifact may still need validation.Source: §A.2 · p. 35

A.3 Design Contracts and Diversity Allocation

  • The read-only Deep Innovation Gate makes intended experiments explicit, while Quantified Diversity allocates contracts across distinct design cells under cohort caps.Opening-generation caps are hard constraints; later generations use them as soft targets, so coverage is planned rather than guaranteed.Source: §A.3 · p. 36, Table 13 · p. 38, Table 14 · p. 39

A.4 PI/Chair Synthesis

  • Peers publish artifacts and findings, independent PI reviewers interpret evidence, and the Chair synthesizes an agenda with hypotheses, tests, peer contracts, validation targets, and dissent-to-experiment actions.The agenda includes exactly C peer contracts and may preserve justified minority high-upside ideas while updating claim boundaries and validation status.Source: §A.4 · pp. 36–37, Table 15 · p. 39

A.5 Frontier Status and Agenda Decisions

  • Promotion is stricter than finding extraction: it selects top k=2 primary-metric findings by default, may add task-anchor findings, and rejects non-promotable or insufficiently mature evidence.Preliminary evidence can remain in memory or diagnostics but cannot displace mature frontier evidence; lane-specific selection uses axes beyond global score.Source: §A.5 · pp. 37–38

A.6 Gems and Selective Memory

  • When enabled, Gems preserve durable lessons alongside current frontier evidence; among the reported campaigns, only trading used compression, recording three resets and four active Gems.Gems are drawn from frontier-balanced evidence and let later peers retain lessons after ordinary findings are archived, without carrying earlier full transcripts.Source: §A.6 · pp. 39–40

A.7 Lineage Trace

  • The typed lineage trace links artifacts, evaluations, findings, contracts, synthesis decisions, frontier updates, agendas, and Gems, recording how evidence became inheritable.The trace is stored across typed ledgers and generation-state files joined by shared identifiers, rather than as one serialized graph.Source: §A.7 · pp. 40–41, Table 21 · p. 42
  • The final output pairs a reproducible artifact with its lineage, exposing the evidence, failures, validation needs, Gems, and agendas that shaped its construction.The run writes the frontier, lineage ledgers, and a summary; the selection rule is applied at reporting time, so early-stopped campaigns still leave a complete frontier and lineage.Source: §A.7 · pp. 41–42

A.8 Resource Scheduling and Mature-Evidence Debt · A.9 Configuration Defaults and MLE-bench Budgets · A.10 Representative Prompt Contracts

These appendices document resource scheduling and its synthetic simulations, the finalized MLE-bench configuration, and representative role-based contracts for producing and synthesizing evidence across generations. The contracts emphasize durable lessons, compact active Gem sets, and provenance linking work to artifacts, findings, and frontier decisions.

A.8 Resource Scheduling and Mature-Evidence Debt

  • The scheduler pairs GPU utilization-and-memory admission with a mature-evidence quota and targets up to three mature-directed attempts per unit of debt while preserving exploration when capacity permits.In simulation, maturity requires full-stage execution, at least 0.75 completed work, and at least 0.80 evaluation coverage; production also gates launches on observed host pressure.Source: §A.8 · pp. 43–46
  • The Boolean signal and thin token finish −1.20 and −1.49 quota-success points below the debt controller, respectively, despite statistically indistinguishable hardware utilization in simulation.The performance gaps grow to 3.75 points under high CPU pressure and 4.43 points under high GPU-memory pressure.Source: Table 23 · p. 45, §A.8 · p. 45, Table 24 · p. 45

A.9 Configuration Defaults and MLE-bench Budgets

  • The finalized MLE-bench campaign used deepseek-v4-pro[1m], 12 peers, top-4 promotion, at most 20 generations, one job per GPU, and up to eight concurrent experiments per task.These are configured ceilings, not measurements of realized compute.Source: §A.9 · p. 46

A.10 Representative Prompt Contracts

  • Representative peer contracts direct agents to build reproducible artifacts from inherited research state, report stage-qualified outcomes and limitations, and preserve artifact references for later lineage synthesis.They also require raw task-grounded evaluator outcomes and preserve fragile claims as candidate or validation-needed evidence, while informative failures become diagnostic evidence.Source: §A.10 · pp. 47, 49
  • Additional contracts allocate diverse peer work, extract artifact-grounded findings, assign frontier inheritance status, and turn PI reviews into agendas and durable cross-generation Gems.The contracts specify each peer’s mechanism, intervention, lineage, expected evidence, and validation hook, and require findings to record evidence, maturity, limitations, and inheritance action.Source: §A.10 · pp. 49–50
  • Prompt contracts favor durable lessons over local details, keep the active Gem set compact, and preserve provenance to artifacts, findings, and frontier decisions.Source: §A.10 · p. 50

A.11 MLE-Style Task Example

The MLE-style example shows how task-grounded artifacts and evaluator outcomes become evidence, inheritance decisions, future agendas, durable lessons, and lineage. Other task families can vary artifact forms and evaluator semantics while retaining the PRAXIST cycle.

A.11 MLE-Style Task Example

  • In MLE-style tasks, a peer builds a reproducible submission artifact from competition context and data; evaluator outcomes feed findings, inheritance-status frontiers, agendas, durable Gems, and lineage.Findings preserve raw evaluator evidence, limitations, and inheritance recommendations; PI/Chair synthesis emits the next agenda.Source: §A.11 · p. 51
  • Other task families may use different artifact forms and evaluator semantics while retaining PRAXIST’s cycle of traceable evaluation, explicit inheritance, agenda-controlled work, selective memory, and lineage.Source: §A.11 · p. 51

A.12 Worked MLE-bench Research Trajectory

This worked trace follows the Jigsaw submission from a generation-0 matched BCE ablation through generation-1 diagnostics and ensemble branches, while distinguishing the selected trajectory from the later campaign-best attempt.

A.12 Worked MLE-bench Research Trajectory

  • ROC AUC rose from 0.98679 for two-seed BERT to 0.98723 with DistilRoBERTa and 0.98749 with independent RoBERTa-CLS; adding BiLSTM reduced the score by 0.00012.The selected lineage links the final finding to the three-model result and derives it from the two-seed BERT ensemble.Source: Table 28 · p. 51, §A.12 · p. 51
  • Generation 0 retained the matched BCE ablation after a 0.00037 improvement, and generation 1 reused preserved predictions after pod 6 completed the C07 null-model diagnostic.The trace includes a negative BiLSTM branch as well as the improving ensemble branches.Source: §A.12 · p. 51
  • The trajectory’s 0.98749 result is not the finalized Jigsaw entry: the release ledger reports a later integrity-clean campaign high of 0.9880.The ledger identifies that later result in Table 36.Source: §A.12 · p. 51

B Experimental Setup Details

The appendix makes execution conditions and resource accounting for the studies explicit, distinguishing configured ceilings from measured consumption and flagging gaps where released ledgers are incomplete.

B Experimental Setup Details

  • The appendix reports each study’s repetitions, hardware, time envelope, and PRAXIST campaign configuration so replication depth and headline-result costs can be read directly.It flags incomplete released-ledger coverage rather than implying uniform accounting.Source: B Experimental Setup Details · p. 52
  • Configured budgets are scheduling ceilings, while artifact wall spans bound elapsed time and include evaluation, synthesis, idle capacity, interruptions, and resume gaps.Early-terminated campaigns may consume far less than their configured budgets allow.Source: B Experimental Setup Details · p. 52
  • The appendix reports generations, peer sessions, optimizer updates or epochs, simulator steps, and evaluation rollouts separately, alongside distinct resource-accounting categories.Configured ceilings, archived wall spans, task-level utilization, and dollar-equivalent model spend are not added into a synthetic total; hardware attributions reflect the process-visible host.Source: B Experimental Setup Details · p. 52

B.1 MLE-bench · B.2 Rocket

The MLE-bench account documents one audited 75-task sweep and its model-spend ledger, while the Rocket study fixes the controller evaluation protocol and distinguishes canonical validation from a post-run coverage audit.

B.1 MLE-bench

  • The MLE-bench comparison used one locally run 75-task sweep per arm on a shared H100 pool, with one campaign configuration and cohorts of 12 peers per task run.PRAXIST’s audited release ledger retains each task’s single best integrity-clean official submission; the ledger records 272 completed generations, and roughly 1,157.5 summed task-run hours are not campaign wall time or GPU-hours.Source: §B.1 · p. 52
  • PRAXIST’s finalized model-spend ledger totals approximately US$3,054 across cache-hit input, cache-miss input, and output tokens.The ledger records 110,910,266,304 cache-hit input tokens, 4,602,586,778 cache-miss input tokens, and 1,108,180,075 output tokens; deepseek-v4-pro[1m] served as the research-agent model on every task.Source: §B.1 · p. 53

B.2 Rocket

  • Rocket evaluation freezes the Swordfish C05 six-degree-of-freedom plant and task contract, leaving candidates able to change only the controller.The plant uses RK4 with one 0.1 s substep for up to 900 steps; the task locks lateral reaction-control and grid-fin roll commands to zero, while pitch and yaw remain actuated through engine gimbaling and grid fins.Source: §B.2 · pp. 53–54, Table 30 · p. 54
  • Rocket landing banks comprise paired nominal and near-OOD positions plus a hard-OOD annulus, with complete evaluation using 4,096 rows per bank.Near-OOD scales all velocity components by 0.92–<0.98; hard-OOD jointly changes radius and velocity using scales of 1.02–1.08. The banks are fixed and adaptively reused, and hard-OOD does not isolate radius from velocity.Source: §B.2 · p. 54
  • Rocket success requires every landing gate to pass jointly at first leg-tip ground contact, and protocol validity also requires compliance with the forbidden-channel contract.The gates include horizontal error ≤5.0 m, bounded descent and horizontal speed, tilt and angular-rate limits, and a strict remaining-fuel fraction above 0.02; scoring excludes post-contact behavior.Source: §B.2 · p. 55
  • Rocket’s canonical private validation covers 13,312 evaluation units, whereas the separate post-run audit scans 122,880 landing trajectories without a roll suite.Canonical validation includes 12,288 landing trajectories and 1,024 roll-disturbance cases, and alone supports promotion and baseline comparison; audit v1 measures selected-controller coverage and is not promotion evidence.Source: §B.2 · p. 55

B.2.1 First-contact controller discovery, matched evaluation, and full-bank audit · B.3 Quant

The landing study reports a large matched-protocol success gain, then qualifies it with full-bank audits, narrow allocator attribution, and simulator limits. The quantitative-trading section distinguishes a T1 case-study selection from a clean, complete T3 campaign result.

B.2.1 First-contact controller discovery, matched evaluation, and full-bank audit

  • The controller uses a below-450 m fuel-commit governor and an initial-radius branch for r0 < 450 m to constrain descent and phase release.The governor replaces the near-hover target with a two-band descent reference when fuel falls below 80%; the radius branch requires vx > −15 m s−1 before phase release.Source: §B.2.1 · pp. 55–56
  • A closed-form, per-axis box-constrained quadratic-program allocator lowered COM sink P95 from 0.337891 to 0.326373 m s−1 and lateral-speed P95 from 0.065620 to 0.047199 m s−1.The one-key ablation also cut grid total variation by 99.5781%, while gimbal total variation rose 2.1277%; it does not attribute the lineage’s landing-success gain to the allocator.Source: §B.2.1 · pp. 56, 59, Table 33 · p. 60
  • Landing success rose from 495/12,288 (4.0283%) to 12,288/12,288 (100%), a 95.9717-percentage-point gain on the same frozen complete protocol.Both results use the same 12,288 trajectories and fixed initial-state rows; the baseline already achieved 100% first contact, so the gain concerns contact quality, not whether contact occurred.Source: §B.2.1 · p. 57, Table 31 · p. 58
  • Across 122,880 post-run audit trajectories, all reached first contact and passed vertical and fuel gates; exactly two failed the joint predicate on lateral speed alone.The scan covered every row in three fixed banks; it was not promotion evidence or a baseline comparison, and per-bank counts are the primary audit claim.Source: §B.2.1 · pp. 58–59, Table 32 · p. 59
  • The lineage moved from 4.0283% success to roughly 49% after early guidance changes, 98.3561% after generation-2 attitude retuning, and saturation by generation 11.These are cumulative results along one parent chain, not isolated effects of the changes named at each stage; only the allocator had a clean single-field ablation.Source: §B.2.1 · p. 60
  • Weco’s autonomous optimizer evaluated 793 controllers and reached 17.12% landing success from the same 4.03% starting artifact, but this was not a controlled cross-model comparison.Weco used different language models, wall-clock budgets, and a broader search space; its dashboard score is not measured on Praxist’s frozen 12,288-trajectory protocol.Source: §B.2.1 · p. 60
  • The audit measures a frozen low-order simulator on fixed banks, not real-world landing reliability, flight readiness, or hardware validation.It omits disturbances such as wind, sensor noise, and actuator failures; scoring stops at first contact, and the hard bank’s minimum successful fuel reserve was only 0.8908 percentage points above the gate.Source: §B.2.1 · p. 62

B.3 Quant

  • The reported quantitative policy stopped at T1 on one seed and 29 cells with three hard violations, while a separate generation-19 policy completed T3’s 145 cells with no hard violations and lower CAGR.T3 spans five seeds and 145 independently retrained cells; the clean confirmed-lane result is distinct from the T1 post hoc case-study selection, which was not eligible for clean promotion.Source: §B.3 · p. 62

B.4 SLAM

The SLAM campaign evaluates an observability-gated visual scheduler and map-admission filter against a baseline, reporting resource-efficiency results with important measurement and comparison caveats. Its evidence comes from accepted runs across fourteen sequences and a campaign spanning 43 generation boundaries.

B.4 SLAM

  • The comparison uses one accepted run per method–sequence pair across fourteen sequences, with no training seeds or run-to-run variance estimate and no pre-declared acceptance rule.Some sequences were replayed or rerun, and the accepted run—not necessarily the first—was used. Trajectory error is APE RMSE after rigid Umeyama alignment to Leica ground truth with 0.02-second association tolerance; visual-update counts come from launch logs.Source: §B.4 · p. 63
  • The champion uses PVTR_MODE=8 for observability-gated visual scheduling and VMAP_DEDUP=1 to reject map points within 0.08 m and 15° in normal in the same voxel.The dwell counter increases skip demand for persistently degenerate frames; saturation below one and a three-frame consecutive-skip cap prevent permanent visual starvation. The filter rejects rather than fuses candidates, so runs report admitted and retrieved point counts.Source: §B.4 · p. 63
  • COVSCHED’s ground-truth-associated sample ratio averages 1.03 (median 0.985) but ranges from 0.79× on eee_03 to 1.76× on nya_02, so full-sequence comparisons require coverage qualification.Short-window relative pose error over 1–30 s is mixed across sequences, and the component controls cover only one to three sequences. The rate-matched periodic-skip control also disables map admission, so it cannot isolate the scheduler signal from the filter.Source: §B.4 · pp. 63–64
  • 72.4% mean reduction in visual-path timing, 1.0% mean increase in LiDAR-path timing, and 22.1% reduction in summed internally timed thread-wall work were reported across fourteen accepted runs.The evaluator’s timing field is whichever thread last writes “Average Total Time”; skipped frames count as zero visual time, and scheduler eigenvalue computation is untimed. These figures are indicative proxies, not end-to-end latency, throughput, or CPU-load measurements.Source: §B.4 · p. 64
  • The campaign used DeepSeek V4 Pro with 8 peers per generation, up to 200 generations, two promotions per generation, and a 5-hour generation window.Its archived snapshot spans generations 0–42 over 145 hours 52 minutes. The specification declares no compute budget, Gem configuration, or evaluator seed list; a ledger request of 3,200 GPU-hours is stage-level, not a realized per-experiment budget.Source: §B.4 · p. 64

B.5 Fusion

Fusion precision rankings depend on the scoring horizon, and controller scores depend substantially on the environment lifecycle. The reported comparison and lineage include a compact evaluation protocol and a campaign that terminated before its configured cap.

B.5 Fusion

  • The comparison scored each controller on 15 episodes (5 scenarios × 3 seeds × 100 steps), with at most 1,500 survived steps; the three Table 7 runs took 28.05, 30.32, and 30.64 minutes.The selected artifact is synthesized Torch control code, not a trained policy; the reported comparison took less than two hours of runner wall time.Source: §B.5 · p. 64
  • On WNRMSE p95, PRAXIST leads on the common horizon except certification, while the PCS-style controller leads on three perturbed scenarios and the aggregate on the full horizon; neither meets the pass rule.On the full horizon, the PCS-style controller loses the certification scenario by an order of magnitude. The common horizon caps each episode at the zero-feedback baseline’s survival length; aggregate rows pool scored steps.Source: §B.5 · p. 65, Table 35 · p. 65
  • Seven generation result sets (generations 0 through 6) filled all five pod slots, recording 114 peer sessions and seven synthesis sessions before termination.The DeepSeek V4 Pro lineage was configured for at most 12 generations with five peers and two promotions per generation, but ended after about 18 hours and 15 minutes. The selected controller first appeared in generation 0, and the remaining six generations refined and tested it.Source: §B.5 · p. 65
  • 1,264 survived steps and 0.667 full-horizon completion came from fresh-reset/action-clipping evaluation; a second fresh validation gave 1,298 and 0.733, while default in-run evaluation gave 894 and 0.200.Table 7’s fresh-reset protocol builds and resets an environment for every episode and is applied identically to all three controllers, so the comparison is internally consistent despite lifecycle-dependent absolute scores.Source: §B.5 · p. 65

C Full MLE-bench Per-Task Results

The appendix presents the complete per-task comparison of PRAXIST and Claude Code + Opus 4.8 across all 75 MLE-bench competitions, alongside the integrity adjudication for the PRAXIST results. The ledgers use finalized scores and specify the hardware and task-tier denominators behind the comparison.

C Full MLE-bench Per-Task Results

  • 70 Claude Code + Opus 4.8 task scores remain accepted after the benchmark’s post-run screen, while five are not scored; its finalized 75-task cost ledger records US$38,370.Cost is reported as resource context and is not used for per-task medal assignment.Source: C Full MLE-bench Per-Task Results · p. 66
  • PRAXIST reports the highest canonical official clean attempt per task; its adjudication records 60 live_best_verified tasks, 6 fresh_lineage_reviewed_clean tasks, and 9 clean_fallback tasks.Across 90,423 rejected attempts, 85,960 were rejected under integrity_policy and 4,463 as canonical_invalid; clean fallbacks earned gold on five tasks, silver on two, and no medal on two.Source: C Full MLE-bench Per-Task Results · p. 66
  • Both arms used single-GPU NVIDIA H100 80GB jobs, with up to eight experiments in flight per task; per-tier rates use denominators of 22 Low, 38 Medium, and 15 High complexity tasks.The integrity adjudication applies to PRAXIST, separately from the benchmark’s post-run severe-cheating screen for Claude Code + Opus 4.8.Source: C Full MLE-bench Per-Task Results · p. 66
  • Tables 36–37 give the complete per-task comparison of Claude Code + Opus 4.8 and PRAXIST across all 75 competitions, grouped by task category.Scores use medal shading and bold the better score by task metric; the tables do not provide an aggregate result in these passages.Source: C Full MLE-bench Per-Task Results · p. 66, Table 36 · p. 67, Table 37 · p. 68

Related papers

Loading 2608.25955v1…