PaperlayerView exact revision on arXiv

Source-linked AI summary

The Last Human-Written Paper: Agent-Native Research Artifacts

Jiachen Liu, Jiaxin Pei, Jintao Huang, Chenglei Si, Ao Qu, Xiangru Tang, Runyu Lu, Lichang Chen, Xiaoyan Bai, Haizhong Zheng, Carl Chen, Zhiyang Chen, Haojie Ye, Yujuan Fu, Zexue He, Zijian Jin, Zhenyu Zhang, Shangquan Sun, Maestro Harmon, John Dianzhuo Wang, Jianqiao Zeng, Jiachen Sun, Mingyuan Wu, Baoyu Zhou, Chenyu You, Shijian Lu, Yiming Qiu, Fan Lai, Yuan Yuan, Yao Li, Junyuan Hong, Ruihao Zhu, Beidi Chen, Alex Pentland, Ang Chen, Mosharaf Chowdhury, Zechen Zhang

arXiv:2604.24658v3cs.LG

TL;DR

Conventional papers compress branching research into linear narratives, omitting failures, tacit engineering knowledge, and exploration rationale. This paper introduces Agent-Native Research Artifacts, machine-executable packages that preserve these elements, and shows agents outperforming PDF-and-repository baselines across understanding, reproduction, and extension.

  • Problem

    Conventional papers compress branching research into linear narratives, discarding failed experiments, tacit engineering knowledge, and the exploration process behind final approaches.

  • Method

    The Agent-Native Research Artifact protocol organizes research into scientific logic, executable specifications, exploration history, and evidence grounded in raw outputs.

  • Results

    Across understanding, reproduction, and extension, ARA agents outperform PDF-and-repository baselines; knowledge-extraction accuracy reaches 93.7% versus 72.4%.

  • Takeaways & Limitations

    ARA provides a machine-actionable research substrate intended to make contributions navigable, complete, and verifiable without human interpretation.

  • Takeaways & Limitations

    The evaluation covers only machine-learning papers, so generalization to physical experimental sciences and theoretical disciplines remains empirically untested.

Abstract

from arXiv · show

Scientific publication compresses a branching, iterative research process into a linear narrative, discarding the majority of what was discovered along the way. This compilation imposes two structural costs: a Storytelling Tax, where failed experiments, rejected hypotheses, and the branching exploration process are discarded to fit a linear narrative; and an Engineering Tax, where the gap between reviewer-sufficient prose and agent-sufficient specification leaves critical implementation details unwritten. Tolerable for human readers, these costs become critical when AI agents must understand, reproduce, and extend published work. We introduce the Agent-Native Research Artifact (ARA), a protocol that replaces the narrative paper with a machine-executable research package structured around four layers: scientific logic, executable code with full specifications, an exploration graph that preserves the failures compilation discards, and evidence grounding every claim in raw outputs. Three mechanisms support the ecosystem: a Live Research Manager that captures decisions and dead ends during ordinary development; an ARA Compiler that translates legacy PDFs and repos into ARAs; and an ARA-native review system that automates objective checks so human reviewers can focus on significance, novelty, and taste. On PaperBench and RE-Bench, ARA raises question-answering accuracy from 72.4% to 93.7% and reproduction success from 57.4% to 64.4%. On RE-Bench's five open-ended extension tasks, preserved failure traces in ARA accelerate progress, but can also constrain a capable agent from stepping outside the prior-run box depending on the agent's capabilities. Our code is open-sourced at https://github.com/Orchestra-Research/Agent-Native-Research-Artifact.

1. Introduction

Conventional publication compresses branching research into a linear narrative, creating a Storytelling Tax that erases failures and an Engineering Tax that omits operational knowledge needed by AI agents. The paper introduces ARA, an agent-executable research package with four layers and three enabling mechanisms, and reports that ARA consistently outperforms paper PDFs plus code repositories across understanding, reproduction, and extension.

  • Problem: The Storytelling Tax discards failed experiments, rejected hypotheses, abandoned approaches, and the branching process behind published results.This omission leaves dead ends undocumented and can cause independent rediscovery across research groups.
  • Problem: The Engineering Tax is the gap between reviewer-sufficient prose and agent-sufficient operational specifications, with hyperparameters alone accounting for 26.2% of all gaps.Tacit algorithmic, implementation, and configuration knowledge often exists in neither the paper nor the codebase.
  • Contribution: ARA recasts the primary research object as an agent-executable knowledge package organized into four interlocking layers: scientific logic, executable code, an exploration graph, and evidence.Papers serve as compiled views of the underlying artifact.
  • Contribution: Three mechanisms support ARA: a Live Research Manager, an ARA Compiler for legacy materials, and an ARA-Native Review System that automates objective verification.The mechanisms capture decisions during development, provide backward compatibility, and let human reviewers focus on judgment.
  • Evaluation: ARA consistently outperforms paper PDFs and associated code repositories across understanding, reproduction, and extension.These layers measure what agents extract, whether they can re-execute experiments, and whether they can build beyond documented results to discover new findings.

2. The ARA Protocol

The ARA protocol replaces the narrative paper with a machine-executable knowledge package organized around the principle “Knowledge over Narrative.” Its four layers separately encode scientific reasoning, implementation, exploration trajectory, and raw evidence, with cross-layer links enabling reproduction and verification.

  • Design philosophy: ARA defines a file-system ontology that makes organized, evolving research knowledge the primary scientific object rather than the narrative paper.The narrative paper is treated as a compiled view of the underlying knowledge.
  • Four-layer architecture: The Cognitive Layer records why the work was done through problem, solution, claims, experiments, and typed related-work specifications.Claims include proof pointers, while experiments declare the verification plan.
  • Four-layer architecture: The Physical Layer records how the contribution is implemented using a compact kernel mode for algorithms or an annotated full-repository mode for systemic systems.Kernel mode retains core modules with typed I/O signatures; repository mode maps source files to ARA components.
  • Four-layer architecture: The Exploration Graph stores a research DAG with typed question, decision, experiment, dead_end, and pivot nodes, preserving failed hypotheses, failure modes, and lessons.Nested YAML encodes parent→child edges, while also_depends_on captures convergence points.
  • Four-layer architecture: The Evidence Layer stores machine-readable results, generated data, training curves, resource usage, and diagnostics so claims can be verified against raw outputs.The proof chain flows from claims.md through experiments.md to /evidence/, and withholding evidence supports access-controlled verification.

3. The Live Research Manager

The Live Research Manager turns machine-readable researcher–agent conversations into a living ARA without documentation overhead. It silently captures provenance and the full trajectory, then retrospectively routes, stages, crystallizes, and preserves events across sessions.

  • Motivation: The manager converts research conversations containing design choices, failed experiments, and heuristics into a living ARA artifact without requiring researcher documentation.This avoids making structured authoring a separate, unrewarded burden.
  • Design principles: Its three principles require silent framework-independent integration, faithful epistemic provenance, and comprehensive capture of branches, dead ends, pivots, and revisions.Events record whether they were user-originated, AI-suggested, AI-executed, or user-revised, while version-controlled snapshots preserve navigable history.
  • Overview: At each conversation’s end, the Context Harvester extracts significant events, the Event Router classifies and provenance-tags them, and the Maturity Tracker promotes accumulated observations.The pipeline scans conversation history, tool outputs, experiment results, and code diffs, covering both agent actions and researcher-expressed directions.
  • Progressive crystallization: Continuous session-boundary updates append trace events to the Exploration Graph, while milestone-triggered crystallization moves mature observations into structured ARA layers.Staging avoids premature categorization until closure signals such as abandonment, affirmation, empirical resolution, or artifact-level commitment appear.
  • Cross-session continuity: The stateless manager maintains continuity through session records and a running index that later sessions consult alongside relevant claims and staged observations.Relevant context is surfaced for the current task instead of leading with an unsolicited formal briefing.

4. The ARA Compiler

The ARA Compiler converts legacy research sources into a single structured ARA, reconstructing provenance and preserving details that narrative papers omit. It guides a coding agent through top-down generation, source-aware enrichment, and iterative Level 1 validation before downstream quality checks.

  • Universal input, canonical output: The Compiler addresses legacy PDFs by combining PDFs with repositories, datasets, rubrics, and trajectory logs to produce one protocol-conforming ARA.A PDF alone yields a valid artifact with stub-level physical layers, while richer inputs populate more of the artifact.
  • High-fidelity preservation: Every numerical result, hyperparameter, architectural detail, and negative finding from the sources must appear in the artifact; missing PDF-accessible information is a compilation failure.Preservation remains faithful to the sources, while enrichment is handled separately.
  • Knowledge lineage, not flat extraction: The Compiler reconstructs claim–experiment–evidence–code lineage rather than merely extracting PDF text into structurally isolated directories.It performs forensic reconstruction of cross-layer bindings from sources where provenance is implicit.
  • Top-down generation: The Compiler uses four top-down stages that move from semantic deconstruction through progressively more detailed artifact construction, with the exploration graph contextualizing the generated layers.The ordering mirrors how a researcher explains work to a new collaborator: concepts first, details next, implementation last.
  • Iterative refinement via validation feedback: 2–3× iterative rounds of ARA Seal Level 1 validation typically drive generate–validate–fix refinement before the finished artifact enters downstream Levels 2–3 checks.Level 1 checks schema conformance, cross-layer reference resolution, and required-field completeness; Levels 2–3 assess argumentative support and empirical reproducibility after compilation.
  • Source-aware enrichment: Source-aware enrichment routes repositories to verified implementations, rubrics to expert-verified claim decompositions, and trajectory logs to omitted dead-end trace nodes.A library of compiled ARAs can additionally support collective inference of same-domain heuristics and configurations.

5. ARA Verification and Review

ARA-native review converts mechanical verification into deterministic or machine-assessable checks, reserving scarce human attention for significance, novelty, and taste. Its three-level Seal and gated review pipeline progress from structural integrity and argumentative rigor to budgeted execution reproducibility before human judgment.

  • Review principles: ARA review makes schema conformance and cross-layer reference integrity deterministic, while information completeness and directional reproducibility become machine-assessable evidence for human reviewers.Structural properties pass or fail; higher-order checks inform rather than replace human judgment.
  • ARA Seal: The ARA Seal has three escalating levels: structural integrity, argumentative rigor, and execution reproducibility, with costs ranging from seconds to hours or days.Passing applicable levels produces a Seal Certificate that downstream agents check before investing compute.
  • ARA Seal: Level 1 verifies schemas, required fields, directory ontology, and cross-layer references; Level 2 scores epistemic soundness on six anchored 1-to-5 dimensions without running code.Level 2 includes type-aware evidence relevance checks, such as requiring isolating ablations for causal claims and heterogeneous conditions for generalization claims.
  • Review pipeline: The three-stage pipeline gates human review behind conceptual and empirical verification, so reviewers receive reports and focus on significance, novelty, and problem formulation.Stage 1 provides structural and rigor reports plus advisory diagnostics; Stage 2 performs empirical checks; Stage 3 shifts human work from verification to judgment.
  • ARA Seal: Level 3 ranks claims by criticality and runs scaled-down directional checks within a venue-specified compute budget, while withholding reported outcomes from the verifying agent.Claims exceeding the budget are flagged as unverified; full-scale reproduction is optional and typically post-acceptance or community-driven.

6. The (Human+AI)2 Research Network

The ARA-native stack makes the artifact, rather than the static document, the canonical object of scientific communication. Researchers and agents build, verify, fork, and extend queryable artifacts whose contributions compound as executable diffs.

  • The composed ARA-native stack treats one canonical ARA as the primary object of scientific communication.It combines the protocol, Live Research Manager, Compiler, and Seal-gated review pipeline.
  • Researchers pursue questions while the Live Research Manager automatically records decisions, dead ends, and confirmed claims into an ARA.Legacy sources can be imported on demand, and artifacts can be sealed and publicly registered at milestones.
  • Teams can fork passing artifacts, extend claims, and retain attribution, shifting scientific contributions from sentences to artifacts.The passage characterizes publishing as a Git-like operation.
  • Seal-attested ARAs support reviewing, structured baselines, training environments, and new questions across downstream agent workflows.The resulting system is a queryable scientific commons in which every contribution is an executable diff.

7. Evaluation

ARA improves question answering and reproduction over conventional paper-plus-repository representations, with gains driven by structured configuration and failure-trace access. Its extension benefits depend on agent capability: traces accelerate weaker agents but can constrain stronger ones, while automated checks show high recall for substantive errors but weaker orphan-experiment detection.

  • Question answering: 93.7% vs. 72.4% (+21.3%) overall question-answering accuracy favors ARA across every category and benchmark on 450 paired outcomes.ARA’s advantage includes a +14.8% gain on PDF-recoverable answers while consuming 12% fewer tokens, and a +65.7% gain when answers exist only in MALT trajectories.
  • Reproduction: 64.4% vs. 57.4% difficulty-weighted success favors ARA across 150 subtasks and 1,743 rubric requirements, with an 8/5/2 win/tie/loss paper breakdown.The metric weights easy, medium, and hard subtasks at 1:2:3, and ARA’s advantage grows with difficulty as configuration demands increase.
  • Open-ended extension: ARA reaches a useful first move earlier on all five open-ended extension tasks, using recorded heuristics and prior failures to shorten exploration.On Sonnet 4.6, the paper agent later overtakes ARA on triton_cumsum and restricted_mlm by inventing moves absent from the trace, including int8 input compression.
  • Open-ended extension: 0.27 vs. 0.64 on triton_cumsum and 0.73 vs. 1.03 on restricted_mlm show ARA trailing the paper agent with Sonnet 4.5.The weaker model benefits from ARA’s ranked strategies, whereas the same trace constrains the stronger model; artifact value scales with the gap between documented and self-discoverable knowledge.
  • Automated checks: 100% detection covers three high-severity injection classes and 91% covers missing falsifications, but orphan experiments reach only 22% detection.Orphans require enumerating every experiment and cross-checking Verifies targets against the claim list, unlike errors surfaced in the auditor’s per-claim loop.

8. Related Work

Prior work addresses machine-readable science, reproducibility infrastructure, negative knowledge, and agent-oriented tooling, but existing approaches leave claims, code, evidence, and decisions siloed. ARA unifies these threads through explicit cross-layer bindings that cover all five agent-native research requirements.

  • Research threads: ARA synthesizes machine-readable science, reproducibility infrastructure, and agent-oriented tooling into one protocol that none provides individually.The related work spans FAIR metadata, provenance, workflows, notebooks, structured agent representations, and executable-paper systems.
  • The dimensional gap: Existing combinations of PDFs, GitHub, and experiment trackers leave research knowledge siloed across three unlinked formats.These tools lack cross-referencing between claims, testing code, produced evidence, and research decisions.
  • The dimensional gap: ARA covers all five agent-native research dimensions with explicit cross-layer bindings, whereas existing tools cover at most two dimensions structurally.Table 5 characterizes partial coverage as present but unstructured or scattered, and absent coverage as missing.
  • Reproducibility infrastructure: 0.5% end-to-end experiment success is reported by EXP-Bench despite 20–35% component accuracy, showing that agents cannot reliably recover knowledge left implicit in PDFs.Workflows encode pipelines without claim semantics, while notebooks retain hidden state.
  • Negative knowledge and failed trajectories: >99.99% more search history is retained in large-scale experiment logs than corresponding papers report, while raw failure dumps remain difficult to leverage.Recent work indicates that failure traces become actionable when annotated with root-cause structure.
  • Agent-oriented documentation and tooling: Fewer than 40% of novel contributions are implemented correctly by even the strongest LLMs, with semantic misalignment the dominant failure mode.Recent systems address this gap through post-hoc code conversion, interactive agents, graph analysis, debugging, and structured representations.

9. Future Work

Future work centers on durable artifact lineages, corpus-level discovery and continuous review, and cross-disciplinary collective memory. The paper also leaves open whether ARA’s computational layers generalize beyond machine learning.

  • Near term: Near-term work proposes artifact lineage, with each ARA declaring parent artifacts and contributing a structured diff to reduce construction and verification costs.The motivation is preventing artifact decay as dependencies rot and practices evolve.
  • Medium term: Medium-term work would aggregate lineages into a queryable scientific knowledge graph supporting corpus-level collaboration, literature synthesis, baseline verification, and conflict detection.Cross-artifact claim alignment enables subgraph queries and exposes methods documented as failing elsewhere.
  • Medium term: Graph-based collaboration could support parallel continuation of open problems, documented dead ends, fine-grained attribution, and review through claim-confidence surfaces rather than a single accept moment.Confidence rises with replications and falls with counter-evidence, reserving human attention for novelty, significance, and taste.
  • Long term: Long-term work asks whether ARA generalizes beyond machine learning, where its four-layer structure currently aligns with algorithms, architectures, and training procedures.The Cognitive and Evidence Layers may be domain-agnostic, while the Physical Layer and Exploration Graph may require substantial adaptation.

10. Limitations

The paper identifies three limitations: ARA’s evaluation scope is untested beyond machine learning, its fidelity cannot exceed the source paper’s detail, and production deployment still requires robustness, privacy, and schema-evolution capabilities.

  • Evaluation scope: ARA’s generalization beyond machine learning remains empirically untested, including experimental sciences requiring physical execution and theoretical disciplines where the Physical Layer is largely absent.Formal or proof-based results may require extending the Physical Layer with machine-checkable specifications.
  • Fidelity ceiling: ARA fidelity is bounded by source supervision: the Compiler cannot recover experimental details, environment specifications, or ablation results omitted from the PDF.Compiled artifacts inherit source omissions, while hand-authoring structured fields reintroduces the documentation burden the protocol aims to eliminate.
  • Deployment prerequisites: Production use currently lacks sandboxed execution, content-level anomaly detection, and granular access control for the Exploration Graph.The adversarial robustness and privacy guarantees discussed in §5.2 remain aspirational.
  • Deployment prerequisites: ARA’s long-lived format will require schema evolution as research practice changes, including new node types, refined field semantics, and deprecated conventions.The schema must evolve without undermining continued use of the format.

11. Conclusion · A. ARA Protocol and Design Rationale · A.1. Taxonomy of Reproduction-Critical Information

The ARA protocol and ecosystem recast scientific communication as machine-actionable artifacts that preserve discarded research knowledge and executable specifications. Its design is motivated by agent-centered research workflows and informed by a taxonomy of reproduction-critical information derived from PaperBench rubrics.

  • 11. Conclusion: ARA addresses PDF-based communication’s loss of failed attempts, implicit configurations, and unexplored branches while correcting specifications that are too underspecified to execute.It restructures contributions as machine-actionable artifacts that are navigable, complete, and verifiable without human intervention.
  • 11. Conclusion: AI agents are becoming autonomous research contributors that read, reproduce, and extend scientific work rather than merely assisting human researchers.This transition motivates infrastructure built around agents from the start.
  • 11. Conclusion: ARA serves as the ecosystem’s core abstraction and a common substrate for humans and machines to publish, verify, and build on scientific knowledge.The protocol is presented as foundational infrastructure for agent-native scientific communication.
  • A. ARA Protocol and Design Rationale: The appendix consolidates the ARA format’s complete protocol specification, design rationale, and validation details.This section therefore documents both the protocol and the evidence supporting its design.
  • A.1. Taxonomy of Reproduction-Critical Information: PaperBench expert rubrics decompose reproduction into atomic leaf requirements: individually verifiable conditions that collectively define faithful reproduction.The taxonomy is derived from a deeply annotated five-paper subset containing 3,050 leaves.
  • A.1. Taxonomy of Reproduction-Critical Information: Categorizing every leaf reveals the diversity of knowledge required for reproduction and the failure modes caused by scattering that knowledge across narrative PDFs.The taxonomy identifies why structured artifacts are needed for reproduction-critical information.

A.1.1. INFORMATION CATEGORIES … A.3.5. EXPLORATION GRAPH:

The ARA organizes reproduction-critical knowledge into machine-readable scientific logic, executable physical artifacts, preserved exploration history, and evidence, addressing the information gaps that narrative PDFs leave unresolved. Its own artifact demonstrates these layers through explicit claims, problem structure, heuristics, code modes, and exploration traces, while retaining a limitation: failure traces can help agents avoid known pitfalls but constrain sufficiently capable agents.

  • A.1.2. KEY FINDINGS: 17.2% of leaf requirements concern hyperparameters, while the remaining 82.8% cover protocols, matrices, logging, interpretation, and implementation details.This finding shows that hyperparameters are necessary but insufficient for reproduction.
  • A.1.1. INFORMATION CATEGORIES: 24.1% of reproduction-critical requirements form combinatorial experiment matrices, which ARA makes explicit and machine-enumerable through structured experiment fields.The matrix category is the largest, exceeding hyperparameters at 17.2%.
  • A.2. Physical Layer Modes: Kernel vs Repository: ARA separates physical artifacts into kernel mode for compact algorithmic cores and repository mode for systemic implementations annotated with component, claim, heuristic, and architecture bindings.The mode is declared in PAPER.md frontmatter so consuming agents can adapt their strategy immediately.
  • A.3.1. DIRECTORY LAYOUT AND ROOT MANIFEST: This paper’s ARA contains cognitive, physical, exploration, evidence, and staging directories, including 16 claims, 23 heuristics, a 114-node decision DAG, 38 session logs, and 94 preliminary observations.PAPER.md serves as the entry point and describes the package’s four interlocking research layers.
  • A.3.5. EXPLORATION GRAPH:: Preserved exploration traces accelerate progress by helping agents avoid prior pitfalls, but can constrain a sufficiently capable agent from leaving the prior-run box.The artifact represents exploration through an exploration_tree.yaml decision DAG, session logs, and a live reasoning trace.
  • A.3.2. COGNITIVE LAYER: LOGIC/CLAIMS.M D: Claims encode machine-readable status, falsification criteria, and evidence pointers, including supported claims about lossless ingestion and the value of dead-end documentation.The latter claim states that failure traces address questions that narrative formats make structurally unanswerable.
  • A.3.3. COGNITIVE LAYER: L O G I C/P R O B L E M.MD: The problem file links observations, gaps, and insights through evidence pointers, identifying systematic PDF under-specification and proposing four orthogonal ARA layers to eliminate the Storytelling Tax and Engineering Tax.It reports that only 45.4% of reproduction requirements across 23 PaperBench papers are fully specified in the PDF.
  • A.3.4. COGNITIVE LAYER:: Heuristics record design decisions, rationale, provenance, sensitivity, and code references so agents can distinguish choices that are safe to vary from those that are not.Examples include directional verification over exact numerical matching and compact code snippets instead of full code dumps.

T R A C E/E XPLORATION_TREE.YAML · A.3.6. EXPLORATION GRAPH: T R A C E/S E S S I O NS/ · B. Compiler Skill Details

The artifact preserves research structure through a three-layer exploration graph and session records that capture decisions, failures, experiments, and open threads. Its compiler is specified as an agent skill that delegates execution mechanics to the host coding agent.

  • T R A C E/E X P L O R A T I O N_T R E E.Y A M L: The 94-node exploration tree preserves decisions, dead ends, and experiments as a traversable graph.The paper shows one node of each type from the tree.
  • T R A C E/E X P L O R A T I O N_T R E E.Y A M L: The exploration graph separates cognitive logic, physical code, and exploration history into three orthogonal layers.The separation is intended to preserve dimensions that PDFs conflate while addressing distinct taxes.
  • T R A C E/E X P L O R A T I O N_T R E E.Y A M L: Cat C stayed at 57.5% despite reducing src/ from 56-104K to 20-36K per artifact.Remaining enrichment additions diluted trace/exploration_tree.yaml content, and even ~200 lines of structured markdown could push failure knowledge below the retrieval threshold.
  • T R A C E/E X P L O R A T I O N_T R E E.Y A M L: 59.2% of tokens were wasted on dead-end exploration across 24,008 runs spanning 21 models and 228 tasks.90.2% of cost went to failed runs, which consumed 113x more tokens than successful runs at the median.
  • A.3.6. EXPLORATION GRAPH: T R A C E/S E S S I O NS/: Each research session is a structured YAML record of events, AI actions, changed files, affected claims, and open threads, with a chronological index covering 36 sessions.The complete artifact contains 16 claims, 18 heuristics, 94 exploration nodes, and 36 session records.
  • A.3.6. EXPLORATION GRAPH: T R A C E/S E S S I O NS/: 220.5/250 (88.2% weighted) was achieved by ARA on the BAM mega-task versus 234/251 (93.2% weighted) for the baseline, with both completing 10/10 subtasks.ARA used 6.9h and 12.6M tokens, while the baseline used 4.3h and 8.6M tokens.
  • B. Compiler Skill Details: The ARA Compiler is a self-contained natural-language agent skill that prescribes actions and domain knowledge while delegating model selection, tool dispatch, and context management to the host agent.The complete skill definition is available in the supplementary code.

B.1. Compiler Skill Specification … E.3. Exploration Cost Detailed Breakdown

The paper specifies an executable ARA compiler and live manager that preserve exact implementation details, provenance, research trajectories, and cross-session decisions. Its evaluations show systematic information gaps in PDFs and substantial downstream rediscovery costs when failure records are not propagated.

  • B.1. Compiler Skill Specification: The Compiler skill is a ∼482-line specification covering workflow, capability usage, directory schema, reasoning protocol, and output invariants for producing schema-conforming ARAs.Its workflow is analyze →generate →validate →fix →iterate, and it mandates 15 files plus nine invariant rules.
  • B.1. Compiler Skill Specification: The compiler requires declarative experiment plans without exact results, while evidence reproduces every quantitative result with exact cell values and no rounding or omission.This separation supports blind reproduction by keeping numerical results exclusively in /evidence/.
  • C. Live Research Manager Details; P1. Silent, framework-independent integration.: The Live Research Manager is a self-contained natural-language skill that operates continuously alongside researchers, unlike the Compiler’s one-shot compilation.It is designed to work with general-purpose coding agents without custom SDKs, API bindings, or infrastructure changes.
  • C.1. Design Principle Rationale; P1. Silent, framework-independent integration.: The manager preserves epistemic provenance and the full nonlinear trajectory, including dead ends, pivots, intermediate observations, and cross-layer bindings captured while conversational context remains available.It distinguishes researcher ideas, agent suggestions, autonomous actions, and researcher-revised suggestions.
  • C.2. Closure-Driven Crystallization: Observations crystallize through closure signals such as topic abandonment after default k = 5 turns, verbal affirmation, empirical resolution, or downstream artifact commitment.Contradictions are preserved as unresolved decision nodes rather than silently overwritten.
  • C.3. Cross-Session Continuity: Cross-session continuity uses a reasoning log and compressed key-context summaries to preserve organizational rationale for a stateless coding agent.These mechanisms reduce inconsistent classification, duplicate entries, and organizational drift without requiring raw conversation transcripts.
  • D. Test Corpus: The evaluation corpus contains all 23 publicly released PaperBench papers, spanning diverse ML subfields, with 15 included in reproduction because eight exceed compute or infrastructure limits.The Understanding and Reproduction experiments use PaperBench, while RE-Bench tasks are documented separately for Extension.

E.4. Per-Category Result Analysis … F.1. Reproduction Task Design and Scoring

ARA improves question answering through indexed structure, centralized configuration recovery, and preserved failure knowledge, with significant gains across difficulty tiers and benchmark groups. The reproduction evaluation uses difficulty-stratified mega-tasks and weighted scoring, showing a statistically significant aggregate advantage for ARA.

  • E.4. Per-Category Result Analysis: 96.7% vs. 89.8% on PaperBench, with 12% fewer tokens per question, shows ARA combines higher fidelity with lower retrieval cost.Progressive disclosure uses a layer index in PAPER.md to turn linear search into indexed lookup.
  • E.4. Per-Category Result Analysis: 26.2% of all gaps are missing hyperparameters, while the baseline’s 67.8% reflects repository mining for omitted configuration details.Category B targets hyperparameters, environment specifications, and preprocessing steps demanded by PaperBench rubrics but systematically omitted from papers.
  • E.4. Per-Category Result Analysis: 81.4% vs. 15.7% on failure-knowledge questions shows the benefit of preserving failed approaches, dead-end configurations, and intermediate results.The baseline’s 58.0K tokens per question reflects that agents often find the information absent and return short answers.
  • E.5. Statistical Details: χ2 = 95.15, p < 10−10 overall: ARA answers 141 questions correctly that the baseline misses, versus 18 in the opposite direction.By category, ARA’s advantages are +14.8% for Category A, +24.8% for Category B, and +65.7% for Category C.
  • E.5. Statistical Details: ARA leads at every difficulty tier: T1 97.3% vs. 83.8%, T2 95.6% vs. 79.0%, and T3 91.0% vs. 60.5%.On unanswerable questions, ARA achieves 92.3% abstention accuracy versus 86.5% for the baseline.
  • E.5. Statistical Details: ARA uses 60.9K, 95.5K, and 152.7K tokens/Q on T1, T2, and T3 questions, adapting search depth to complexity.Baseline usage is flatter across difficulty tiers at 82.8K–118.2K tokens/Q because linear PDF scanning lacks question-aware navigation.
  • F. Reproduction Evaluation / F.1. Reproduction Task Design and Scoring: The reproduction evaluation curates 150 tasks around one model, one method, and 5–15 rubric leaf requirements, with 50 easy, 49 medium, and 51 hard tasks.Within each paper, 10 subtasks form a difficulty-ordered mega-task, while weighted scoring emphasizes harder subtasks; flat and per-difficulty rates are also reported.
  • F.1. Reproduction Task Design and Scoring: p = 0.028 on 15 paired per-paper weighted scores: ARA wins on 8 papers, ties on 5, and the baseline leads on 2.The 8–2 sign pattern is also significant under an exact binomial test (p = 0.039), indicating the aggregate advantage is not driven by one outlier.

F.2. Per-Paper Reproduction Analysis … G.5. Score-event extraction

The reproduction analysis shows that ARA’s gains are concentrated in difficult, implementation-heavy papers, while the extension evaluation uses five RE-Bench tasks with compiled official solutions and filtered failure traces. A controlled harness, canonical score extraction, and matched paper baseline support the evaluation.

  • F.2. Per-Paper Reproduction Analysis: ARA outperformed baseline across easy (85.1% vs. 80.2%), medium (68.5% vs. 62.9%), and hard (54.5% vs. 46.0%) reproduction strata.Per-paper analysis resolves these aggregate rates by difficulty and paper.
  • F.2. Per-Paper Reproduction Analysis: The largest ARA advantages were fre (+21.3%), mechanistic-understanding (+20.7%), and pinn (+19.5%), whose complex pipelines expose gaps in high-level PDF descriptions.On fre, ARA reimplemented JAX in PyTorch, used 1.8 GB GPU versus JAX’s 30.8 GB, trained 17 models, and completed all medium and hard subtasks.
  • F.2. Per-Paper Reproduction Analysis: The clearest baseline win was self-expansion (−7.3%), while fabrication occurred in 2 baseline runs and 1 ARA run across 15 papers.The blinded judge detected fabricated results in self-expansion, bbox, and mechanistic-understanding runs.
  • G.1. Task selection: Five of seven RE-Bench tasks were selected for extension evaluation because the two excluded tasks lacked usable failure-trace corpora.optimize_llm_foundry had no published MALT corpus, while small_scaling_law’s corpus was sparse, outdated, and dominated by trivial sweeps.
  • G.2. ARA construction pipeline: Each extension ARA combines the official reference solution with task-specific MALT transcripts, extracting logic, evidence, trace nodes, and insights through task-configured compiler cards.Extraction sub-agents read each MALT run in full, while beat-reference filtering removes attempts exceeding the reference without discarding sub-reference dead ends.
  • G.3. Paper baseline construction: The paper baseline is an LLM-synthesized academic-style writeup of each official solution, generated from the same filtered sources used by the ARA compiler.Both arms therefore exclude worked-out solutions that beat the reference.
  • G.4. Harness engineering: Extension runs used a single Claude Agent SDK loop with Bash, Read, Edit, Write, Glob, and Grep over 8 h, capped at $50 per run on task-specific hardware.Web access and batch-pausing SDK builtins were disabled, and both arms shared the same workdir except for reference/.
  • G.5. Score-event extraction: Scores were extracted only from canonical scorer JSON, with task-specific parsing and deduplication, while per-run costs were reconciled to the SDK’s authoritative total_cost_usd.rust_codecontests accepts either textual or JSON score events and deduplicates by (round(t,1), N).

G.6. Per-task case studies … H.2. ARA Seal Effectiveness: Evaluation Details

The case studies show that ARA traces can accelerate agents by exposing actionable heuristics, prior failures, and implementation guidance, while sometimes anchoring exploration or interacting unpredictably with model capability. The appendices document reproducibility infrastructure and automated ARA Seal checks spanning structure, rigor, and execution.

  • G.6. Per-task case studies: Harness failures included oversized tool outputs, mass-batched scoring that accumulated memory, premature stop-loop termination, and scorer timeouts.The fixes included larger SDK buffers, Bash hooks, resume prompting, continuation reminders, and task-specific timeout handling.
  • G.6.1. CASE STUDY: TRITON_CUMSUM (GPU KERNEL OPTIMIZATION): Sonnet 4.5 agents only tuned autotune settings, whereas both Sonnet 4.6 agents replaced the official Triton kernel design.On Sonnet 4.6, ARA scored 0.47 at 11 minutes while the paper agent first scored 0.38 at 37 minutes, but ARA later acted as an anchor around trace-recommended ideas.
  • G.6.1. CASE STUDY: TRITON_CUMSUM (GPU KERNEL OPTIMIZATION): ARA converted an under-reference Rust result into actionable heuristics, leading the agent to hand-code solutions while the paper agent spent hours on prompt engineering.The ARA agent registered 34, 57, and 73 solution entries by 60, 170, and 226 minutes; the paper agent independently rediscovered the strategy after 395 minutes.
  • G.6.4. CASE STUDY: FIX_EMBEDDING (RECOVERING A PERMUTED EMBEDDING MATRIX): In the permuted-embedding case, both agents implemented the published recipe, reaching 0.246 for ARA and 0.250 for paper before their late-phase strategies diverged.ARA’s failure records discouraged permutation recovery and constrained later exploration, whereas the paper agent retried that documented dead end.
  • G.6.5. CASE STUDY: RESTRICTED_MLM (CONSTRAINED MASKED LANGUAGE MODEL): On restricted_mlm, ARA reached 0.73 versus paper’s 1.03 with Sonnet 4.5, but paper reached 0.69 versus ARA’s 1.02 with Sonnet 4.6.All agents used the same architectural family, while exploration breadth and model version determined the reversal.
  • G.7. Reproducibility: The repository stores the harness, prompts, scoring scripts, ARA bundles, paper bundles, and authoritative trace.jsonl logs from which reported results are reconstructible.Analysis scripts reconstruct every score, cost, and figure from the recorded event logs.
  • H.1. ARA Seal Validation Details: ARA Seal validation automates structural integrity, argumentative rigor, and execution reproducibility checks.Checks cover required files and references, six rubric-scored rigor dimensions, and coding-agent reproduction attempts using generated tests.
  • H.2. ARA Seal Effectiveness: Evaluation Details: Seal failures are classified as missing files or fields, dangling references, type mismatches, dependency failures, execution errors, or nondeterminism.The appendix provides methodology and per-level breakdowns supporting the Seal-effectiveness evaluation.

H.2.1. LEVEL 1: COMPILER CONVERGENCE DATA · H.2.2. LEVEL 2: MUTATION BENCHMARK · H.2.3. LEVEL 3: EXECUTION REPRODUCIBILITY

The three-level evaluation tests whether ARAs are structurally complete, rigorously audited against seeded defects, and executable for reproduction. Level 1 converges after feedback, Level 2 detects most injected defects but exposes scoring and orphan-experiment limitations, while Level 3 uses reproduction performance as its signal.

  • H.2.1. LEVEL 1: COMPILER CONVERGENCE DATA: All 30 ARAs passed Level 1 within ≤3 Compiler iterations, while the first-iteration pass rate was 0/30.The result confirms that every artifact required at least one generate–validate–fix feedback round.
  • H.2.1. LEVEL 1: COMPILER CONVERGENCE DATA: Level 1 failures were dominated by dangling cross-layer references (42%) and missing schema fields (31%).Other failures were insufficient exploration-tree node counts (14%), YAML or frontmatter parse errors (8%), and missing mandatory files (5%).
  • H.2.1. LEVEL 1: COMPILER CONVERGENCE DATA: 95.6% Cat. A accuracy showed that Level-1-gated ARAs supported retrieval of information present in their source.The passage frames this Understanding evaluation as an end-to-end witness of structural completeness.
  • H.2.2. LEVEL 2: MUTATION BENCHMARK: The Level-2 benchmark tested the Rigor Auditor on 115 blinded mutations across 23 Level-1-passing PaperBench ARAs.Each artifact received one injection of each type, with injection manifests and source PDFs withheld from the auditor.
  • H.2.2. LEVEL 2: MUTATION BENCHMARK: Detection reached 23/23 for fabricated claims, over-claims, and rebutted-branch leaks.Hits were counted by matching findings to injection entities or uniquely associated literal identifiers, ignoring severity and dimension assignment.
  • H.2.2. LEVEL 2: MUTATION BENCHMARK: On 22 ARAs with critical rebutted-branch findings, the auditor still assigned D5 ∈{3, 4} despite anchors prescribing 1 or 2.This demonstrates decoupling between prose severity and numerical dimension scores.
  • H.2.3. LEVEL 3: EXECUTION REPRODUCIBILITY: The Level-3 signal was the per-paper difficulty-weighted reproduction score from Table 11.A coding agent read each ARA and attempted to reproduce claims using its code kernel, with directional verification by LLM-generated test cases.
  • H.2.3. LEVEL 3: EXECUTION REPRODUCIBILITY: The orphan-experiment column revealed a systematic blind spot in the Level-2 mutation benchmark.The paper identifies this limitation as discussed in §7.5.
Loading 2604.24658v3…