PaperlayerView exact revision on arXiv

Source-linked AI summary

Externalizing Research Synthesis and Validation in AI Scientists through a Research Harness

Zijian Wang, Hanqi Li, Ziyue Yang, Zijian Hu, Shenghan Zuo, Yunzhe Zhang, Da Ma, Danyu Luo, Chenrun Wang, Jing Peng, Tiancheng Huang, Sijia Guo, Huayang Wang, Zichen Zhu, Senyu Han, Yilu Cao, Bo Chen, Xin Chen, Kai Yu, Lu Chen

arXiv:2606.18874v3cs.AI

TL;DR

AI scientists can automate research while leaving the reasoning from evidence to claims implicit and difficult to audit. XCIENTIST externalizes synthesis and validation into governed artifacts and, across three case studies, preserves traceable mechanisms through testing, repair and bounded claims.

  • Problem

    Automated research systems generate ideas and runnable experiments, but their decisions, mechanism fidelity and evidential support remain difficult to audit.

    Source: §1 · p. 4

  • Method

    XCIENTIST uses structured literature evidence, an ideation-validation-evolution loop, executable contracts, ablation and repair traces, and claim-boundary audits.

    Source: §1 · p. 4, §2 · p. 6

  • Results

    Across memory, graph forecasting and PINN tasks, XCIENTIST preserves attributable trajectories from mechanism design through validation, diagnosis and repair.

    Source: §4 · pp. 15–16

  • Takeaways & Limitations

    AI scientists should be evaluated not only by final artifacts but also by whether their research trajectories remain grounded, executable, testable, repairable and attributable.

    Source: §4 · p. 15

    The evaluation is case-based and depends on paper-graph fidelity, runnable repositories, available datasets and meaningful benchmark protocols.

    Source: §4 · p. 16

Abstract

from arXiv · show

AI systems can increasingly automate scientific workflows, but the reasoning that links prior evidence, generated ideas, experiments and final claims often remains implicit inside model inference. Here we introduce Xcientist, a research harness that externalizes research synthesis and experimental validation into inspectable, contract-governed processes. Xcientist organizes literature evidence, idea states, implementation plans, ablation records and repair traces as persistent research artifacts, so that generated mechanisms can be grounded, executed, tested and revised without losing their evidential basis. We identify claim drift as a failure mode of automated research, where runnable artifacts no longer support the mechanism originally claimed. Across training-free memory systems, graph-structured traffic forecasting and multi-scale physics-informed neural networks, Xcientist preserves traceable trajectories from problem formulation to mechanism design, validation and bounded revision. These results suggest that AI scientists should be evaluated not only by their final artifacts, but by whether their synthesis and validation processes remain attributable, inspectable and scientifically accountable.

1 Introduction

XCIENTIST addresses the difficulty of auditing AI-generated scientific reasoning by externalizing research synthesis and experimental validation into inspectable structures and governed processes. Across three scientific domains, it preserves traceable paths from evidence and mechanism design to testing, repair and bounded claims.

Motivation

  • Automated research systems can generate ideas, code and experiments, but their intermediate decisions and evidential basis often remain difficult to audit.This includes whether prior evidence shaped an idea, whether implementations realize proposed mechanisms, and whether gains support the claimed intervention.Source: §1 · p. 4

Approach

  • XCIENTIST externalizes research synthesis and experimental validation so they can be inspected, controlled and reproduced.The framework replaces tacit knowledge and unconstrained inference with explicit structures and governed execution.Source: §1 · p. 4, Figure 2 · p. 4

Research synthesis

  • Its evidence graph makes methods, baselines and datasets explicit and queryable, enabling bounded idea searches guided by novelty and feasibility criteria.These structures support evidence-grounded reasoning and verifiable gap identification rather than reliance on opaque model weights.Source: §1 · pp. 4–5

Experimental validation

  • Contract-governed execution requires each implementation, evaluation, ablation and repair step to produce evidence satisfying predefined inputs, operations, artifacts and acceptance criteria.Structured repair traces preserve the basis for experimental decisions and metric interpretation.Source: Figure 2 · p. 4, §1 · p. 5

Evaluation

  • Across memory systems, graph forecasting and PINNs, XCIENTIST exposes how mechanisms are formed, tested, diagnosed and repaired rather than evaluating only final artifacts.The cases support evaluating AI scientists by whether their trajectories remain grounded, executable, testable, repairable and attributable.Source: §1 · p. 5

2 XCIENTIST Architecture Overview

XCIENTIST is a three-layer research harness that turns literature evidence into inspectable research trajectories through constrained ideation, validation, repair and claim auditing. Its central design evaluates the accountable process connecting evidence, mechanisms, executable experiments and scientific claims rather than only the final artifact.

Research Harness

  • The Research Harness converts automated research from free-form model outputs into a governed sequence of evidence intake, idea construction, validation, repair and claim audit.Ideas are proposed from structured evidence, translated into executable plans, tested through checkable artifacts and bounded before becoming claims.Source: §2 · pp. 5–6

Paper Graph Infrastructure

  • The Paper Graph Infrastructure parses full-text papers into schema-bound records organized as a method-evolution graph for evidence retrieval and claim support.It provides the evidence substrate for synthesis rather than relying on transient context or latent model memory.Source: §2 · p. 5

Ideation and validation

  • Idea generation searches, scores and fuses candidate mechanisms under explicit novelty, feasibility and verifiability criteria before experiment validation.Validation then turns selected mechanisms into staged implementation, execution, ablation and repair contracts.Source: §2 · p. 5

Ideation-validation-evolution loop

  • The ideation-validation-evolution loop repairs ideas when checkable experiments expose defects and preserves links between evidence, mechanism design, implementation artifacts and claims.This makes mechanism evolution inspectable rather than treating ideas as final model outputs.Source: §2 · p. 6

System User Interface

  • The System User Interface exposes workflow lanes, artifacts, traces and approval decisions so users can review, intervene in and audit intermediate states.A unified backend records the process while the frontend makes it visible and controllable.Source: §2 · p. 6

Design significance

  • XCIENTIST shifts evaluation from final generated artifacts to accountable research trajectories that couple structured knowledge, idea evolution, contracted validation, repair and claim-boundary auditing.This system-level coupling distinguishes it from prior systems that automate only individual workflow components.Source: §2 · p. 6

3 Results

XCIENTIST evaluates research trajectories rather than isolated scores, using external evidence, structured idea evolution, validation contracts and repair traces to address claim drift. Across memory, graph forecasting and PINN tasks, the case studies show bounded, inspectable mechanism development with both successes and explicit limits.

3.1 Motivating example: Claim drift in automated research

  • Claim drift occurs when plausible proposals and runnable artifacts no longer preserve evidence attributing results to the claimed mechanism.AI-Scientist-v2 exhibited semantic, experimental and mechanistic drift across memory, graph-forecasting and PINN runs.Source: §3.1 · pp. 7–8, Table 2 · p. 8

3.2 Task 1: Training-free memory system for LLM agents

  • Memory-design repair replaced drift-prone or brittle mechanisms with a slotted evidence reranker, soft facet enhancement and fallback behavior for unreliable metadata.The repairs targeted structural risks rather than merely adding components.Source: §3.2 · pp. 9–11
  • The memory system improved overall textual F1 from 0.306 to 0.391 while reducing average token length from 2844.1 to 1017.2, a 64.2% reduction.The final trajectory converged on atomic evidence, constrained retrieval and a compact performance–cost trade-off.Source: §3.2 · pp. 9–11

3.3 Task 2: Graph-structured spatio-temporal forecasting

  • Graph forecasting repaired an inert hard orthogonal projection by replacing it with proposal-space innovation coverage that preserved diffusion complementarity without input-space basis constraints.Removing the projection changed clean MAE by less than 0.12% relative (∆= +0.0019), motivating the redesign.Source: §3.3 · pp. 11–12
  • Iteration 4 achieved the best graph-forecasting performance, with average MAE 1.556 and horizon-12 MAE 1.908, while using shorter training and adding no core-mechanism parameters.Removing the coverage cell increased clean MAE by 0.0042 and worsened block-40% degradation from 1.060 to 1.101.Source: §3.3 · pp. 12–13

3.4 Task 3: Multi-scale physics-informed neural networks

  • PINN average rank improved from 6.33 in v1 to 2.33 in v6, with v6 obtaining the best mean relative L2 error on heat1d_multiscale and pinnacle_heat.The reported errors were 0.0672 ± 0.0118 and 0.4307 ± 0.0139, respectively.Source: §3.4 · p. 14
  • The PINN results do not show unconditional dominance: v6 remained behind strong external baselines on heat2d_multiscale.MMPINN reached 0.00661 ± 0.000995 and MultiscalePINNs reached 0.0261 ± 0.00331 on that benchmark.Source: §3.4 · p. 15
  • The PINN trajectory converged from expressive residual decompositions toward a fixed complement contract and Fourier-enriched, testable scale-separation mechanism.The design assigns global structure to a stabilized coarse branch and restricts residual branches to derivative-consistent fine-scale corrections.Source: §3.4 · pp. 13–15

4 Discussion

XCIENTIST makes automated scientific reasoning inspectable by preserving evidence, mechanism design, validation, repair, and claim-boundary decisions as linked artifacts. Its case studies show bounded successes and failures while exposing claim drift as a central accountability problem.

4 Discussion

  • The paper argues that AI scientists should be assessed by whether their research trajectories remain attributable, inspectable, and revisable, not only by their final artifacts.This accountability layer complements systems focused on paper generation, hypothesis generation, or data-driven discovery.Source: §4 · pp. 15–16
  • XCIENTIST externalizes the path from literature evidence to idea states, implementation contracts, validation stages, ablations, repair traces, and final claims.Validation outcomes remain evidence for revising mechanisms rather than serving only as a terminal check.Source: §4 · p. 15
  • Across three tasks, XCIENTIST preserved bounded trajectories: compact slotted retrieval for memory, repaired graph forecasting after an inert projection, and mixed PINN results across heat benchmarks.The final PINN version achieved the best mean relative L2 error on heat1d_multiscale and pinnacle_heat but remained inferior on heat2d_multiscale.Source: §4 · pp. 15–16
  • Claim drift occurs when implementation or validation no longer supports the mechanism originally asserted, including semantic, experimental, or mechanistic drift.XCIENTIST links evidence, mechanism design, contracts, ablations, and claims to reduce this failure mode.Source: §4 · p. 16
  • The evaluation remains case-based and depends on paper-graph fidelity, runnable repositories, available datasets, and meaningful benchmark protocols.The authors call for claim-level provenance, quantitative process metrics, and broader, less curated evaluations.Source: §4 · p. 16

5 Methods

XCIENTIST implements a research harness that turns literature synthesis, idea search, experiment validation, reporting, and claim auditing into governed, artifact-producing stages. Its workflow uses structured idea states, validator-backed contracts, isolated workspaces, and iterative repair.

5 Methods

  • Figure 7 presents three coupled layers: a paper-graph evidence substrate, a research harness linking synthesis through validation, and an interface exposing traces and approvals.The harness grounds, executes, ablates, repairs, and audits candidate mechanisms before claims are written.Source: Figure 7 · p. 17
  • The paper graph infrastructure parses full text, extracts schema-bound evidence, resolves entities, and supplies structured material for downstream research workflows.The resulting graph contains approximately 72,000 core method nodes, 250,000 baseline nodes, 63,000 dataset nodes, and 1.15 million typed edges.Source: §5 · pp. 17–18
  • The framework adopts DeepSurvey for evidence-grounded literature synthesis, using retrieval expansion, full-text keynotes, thematic analysis, comparison tables, and plan-then-write drafting.Its outputs provide a reusable analysis substrate for ideation and reporting.Source: §5 · p. 18
  • Idea generation searches structured idea states with multiple Idea Taste Modes, Monte Carlo Tree Search, editable component operators, symbolic ablation memory, and constrained fusion.Each state records problem framing, mechanisms, risks, expected advantages, and a preliminary validation plan.Source: §5 · p. 18
  • Validation enforces validator supremacy, separation of concerns, and workspace encapsulation through staged preparation, implementation, baseline runs, component ablations, and convergence checks.The scheduler requires validator-backed passes, runnable artifacts, and complete component-to-evidence mappings before declaring convergence.Source: §5 · pp. 18–19
  • Reporting converts workspace artifacts into prose through a write-audit-repair loop that checks source fidelity, research integrity, citation readiness, and claim drift.Claims are bounded or revised when evidence cannot attribute outcomes to the originally proposed mechanism.Source: §5 · p. 19

A XCIENTIST Architecture Implementation

The architecture builds a heterogeneous method-evolution graph from full-text evidence and integrates it with synthesis, ideation, validation, and reporting modules. Its open, traceable schema preserves experimental context for both literature analysis and claim-grounded execution.

A.1 Paper Graph Infrastructure

  • Existing paper-level citation and retrieval systems omit method-level comparisons, datasets, conditions, and results needed for evolution tracing and evidence-grounded ideation.XCIENTIST addresses this by decomposing papers into contribution-level method entities linked to shared baselines and datasets.Source: §A.1 · p. 23

A.1.1 Corpus Construction

  • The graph is built from nearly 50K Semantic Scholar computer-science papers spanning major CS domains and neighboring fields.Corpus construction provides the source base for the heterogeneous evidence graph.Source: §A.1.1 · p. 23

A.1.2 Full-text Structured Parsing

  • Full-text parsing preserves baseline details, component descriptions, experimental conditions, and failure modes that abstracts often omit.MinerU merges rendered Markdown with structured content lists so tabular evidence remains available to extraction.Source: §A.1.2 · p. 23

A.1.3 Schema-bound Evidence Extraction

  • Schema-bound extraction separates ideation resources from graph data while retaining flexible natural-language fields and quote anchors for traceability.The two blocks capture scientific content alongside baseline and dataset comparison structure.Source: §A.1.3 · p. 24, Table 3 · p. 24

A.1.4 Graph Construction with Entity Resolution

  • The heterogeneous graph contains core, baseline, and dataset nodes connected by methodological, comparison, and evaluation edges with local experimental evidence.Entity resolution merges repeated baselines and datasets by Semantic Scholar Paper ID, yielding approximately 385K nodes and 1.15M typed edges.Source: §A.1.4 · p. 25, Table 4 · p. 25

A.1.5 Integration with Downstream Modules

  • Graph records support synthesis by tracing method lineages and identifying gaps, while baseline, dataset, component, and quote fields constrain validation and claim attribution.The same evidence substrate supports survey artifacts, idea states, ablation design, repair planning, and claim-boundary audits.Source: §A.1.5 · pp. 25–26

A.2.1 Overview

  • DeepSurvey converts graph-grounded literature into structured surveys whose keynotes, relations, comparisons, and paragraphs remain anchored to source papers and passages.Analysis precedes writing, and evidence traceability supports downstream verification.Source: §A.2.1 · p. 26

A.2.2 System Architecture

XCIENTIST combines graph-backed literature synthesis, structured ideation, and citation-controlled drafting into a staged research workflow. Its idea-generation component explores multiple grounded search preferences, uses memory-guided MCTS, and fuses candidates through inspectable, repaired synthesis.

A.2.3 Graph-backed Literature Retrieval

  • Graph expansion, hybrid filtering, full-text keynotes, clustering, and multi-perspective analysis produce a structurally grounded and comparison-ready evidence corpus.The analysis includes typed relation graphs, comparison tables, guided Q&A, code analysis, and inter-cluster patterns with source attribution.Source: §A.2.3 · pp. 26–27, §A.2.4 · p. 27, §A.2.5 · p. 28

A.2.2 System Architecture

  • DeepSurvey builds a five-stage survey pipeline that retrieves, structures, relates, drafts, and iteratively refines literature evidence.Its graph-backed retrieval and full-text keynotes create the analysis substrate before outline-driven generation and multi-granularity refinement.Source: §A.2.2 · p. 26, §A.2.3 · p. 26, Figure 9 · p. 27

A.2.6 Outline-driven Drafting and Citation Enforcement

  • Outline-driven drafting assigns papers to sections and verifies citations against localized evidence, while retries and refinement loops preserve claim-source alignment.Each writing unit is constrained to its assigned paper subset, and paragraph-level checks trigger regeneration when citation consistency or authenticity fails.Source: §A.2.6 · p. 28, §A.2.7 · p. 28

A.3.2 Idea Taste Modes

  • Idea Taste Modes reshape search preferences over the same grounded problem, supporting complementary novelty, transfer, engineering, and evidence-first directions.The modes alter the search objective, edit-operator bias, and mechanism-instantiation guidance rather than acting as superficial prompt variants.Source: §A.3 · p. 28, §A.3.2 · p. 29, Figure 10 · p. 29

A.3.1 Workflow Stages

  • XCIENTIST treats ideation as staged movement from evidence and problem understanding to controlled mechanism search, refinement, and selection.This decomposition is intended to reduce problem drift by replacing unconstrained brainstorming with a controlled transition toward mechanism design.Source: §A.3.1 · pp. 28–29, Table 5 · p. 29

A.3.3 Memory-Guided MCTS

  • Memory-guided MCTS expands interpretable component-level edits and uses retrieved search experience, validation hints, and multi-dimensional evaluation to guide candidates.Vector memory supplies local edit patterns and failure corrections, while symbolic memory derives from component-wise ablations and supports mechanism-centered novelty assessment.Source: §A.3.3 · p. 30

A.3.4 Idea Fusion

  • Cross-mode fusion selects a coherent host thesis, retains only supporting components, records synthesis metadata, and accepts repairs only when the composite score improves.The fused proposal must expose one dominant mechanism while preserving scope and producing a defensible validation story.Source: §A.3.4 · pp. 30–31

A.4 Experiment Validation

XCIENTIST validates experiments through independent contracts, staged gates, and inspectable workspace state rather than narrative progress. Preparation, execution, ablation, and convergence advance only when validator-backed evidence satisfies explicit conditions.

A.4 Experiment Validation

  • Validator supremacy, separation of concerns, and workspace encapsulation make empirical evidence—not model output—the basis for accepting experimental progress.Planning, execution, and verification are separated, while each experiment runs in a versioned, scoped project environment.Source: §A.4 · pp. 31–32

A.4.2 Master Scheduling and Convergence Judgment

  • The master scheduler routes work according to validated workspace state and declares convergence only after every phase passes, blockers clear, and all components have ablation evidence.It prioritizes code self-containment, implementation, standard experiments, and canonical-component ablations, rejecting partial or textually justified progress.Source: §A.4 · p. 32, Table 6 · p. 32, §A.4.2 · p. 33

A.4.1 Preparation: Building the Execution Surface

  • Preparation uses sequential contracts and worker-validator repair loops to establish verified repositories, environments, datasets, models, and authoritative handoff artifacts.The handoff files record exact execution resources and preserve the canonical component list as the ground truth for later phases.Source: §A.4.1 · pp. 32–33

A.4.3 Code Enablement and Ablation Exposure

The code-enablement phase turns the proposal into self-contained runnable code whose canonical components can be independently disabled and traced to their sources.

A.4.3 Code Enablement and Ablation Exposure

  • Implementation requires a full real-data smoke test and self-contained project code before experiments can proceed.Each planned step specifies inputs, write roots, and verification commands, ending with an integration test over the prepared dataset and model.Source: §A.4.3 · p. 33
  • Every canonical component must support isolated ablation without changing other components, enabling measurement of each component’s marginal contribution.The code also documents each variant’s method context and records provenance when source code is copied.Source: §A.4.3 · p. 33
  • The validation artifact tree organizes the experiment’s executable and evidential outputs for inspection.The supplied figure identifies the artifact tree specifically at the experiment-validation stage.Source: Figure 11 · p. 34

A.4.4 Standard and Ablation Science

XCIENTIST turns validation, reporting, interface operation, and literature review into inspectable workflows governed by explicit artifacts, contracts, audits, and persistent run state. Across these stages, the system preserves traceability from implementation and evidence collection through repair, communication, and bounded deployment.

A.4.4 Standard and Ablation Science

  • Each component is validated through ordered baseline and full-method execution, while ablation science disables one component per step to establish complete marginal- contribution coverage.Validators require execution artifacts, metrics, confidence estimates, and method-context descriptions before accepting each step.Source: §A.4.4 · p. 35

A.4.5 Iteration Integration and Final Evidence

  • The scheduler grounds iteration decisions in validated reports and emits a canonical ablation file, while iteration-limit termination is explicitly recorded instead of being presented as convergence.The final deliverable records per-component results, metric values, confidence levels, and method-context analyses.Source: §A.4.5 · p. 35

A.5 Report Writing

  • Report writing uses a write-audit-repair loop that checks technical claims against source code, validates citations against downloaded papers, and generates figures from explicit method descriptions.The terminal report is a scored, audited artifact built from source code, validated ablations, iteration reports, and selected literature evidence.Source: §A.5 · p. 35, §A.5.3 · p. 37, §A.5.4 · p. 37

A.6 System Interface

  • The system interface exposes topic-centered runs through review, ideation, and experiment lanes, with snapshots, streamed events, persistent artifacts, and approval interactions supporting inspection and recovery.Backend orchestration converts requests into run objects and stage plans, while events and artifacts preserve progression, failures, and outputs for later analysis.Source: §A.6.2 · p. 39, §A.6.3 · p. 39, §A.6.4 · p. 39

A.6.5 Per-run Runtime Materialization

  • Per-run workspaces isolate outputs and preserve replayable execution context, but the current deployment remains single-host and intentionally favors bounded operational complexity over distributed scale.Local filesystem persistence and single-host orchestration limit cross-host artifact management, scaling, and isolation.Source: §A.6.5 · p. 40, §A.6.7 · p. 40, §A.6.8 · p. 41

B Literature Review Case Study

  • DeepSurvey builds an inspectable AutoSurvey evidence substrate by graph-expanding from seed papers, filtering candidates semantically and with an LLM re-ranker, and producing an approximately 100-paper corpus from 8 seeds.Graph traversal discovers structurally connected papers that flat semantic retrieval may miss.Source: B Literature Review Case Study · p. 42, §B.1 · p. 42, Table 9 · p. 42

B.2 Full-text Keynote Extraction

  • Full-text keynote extraction exposes methodological mechanisms and limitations absent from abstracts, including overgeneralization as AutoSurvey’s dominant error mode rather than missing citations.The keynote compresses a 1,015-word source passage to 105 words while retaining deeper diagnostic content.Source: §B.2 · p. 42, Table 10 · p. 43

B.3 Clustering and Multi-perspective Relation Analysis

DeepSurvey combines thematic clustering with relation graphs, comparison tables, and guided questions to analyze literature from multiple perspectives. The resulting synthesis distinguishes explicit evolutionary modeling from more flexible iterative refinement as a central planning trade-off.

B.3 Clustering and Multi-perspective Relation Analysis

  • DeepSurvey groups AutoSurvey papers into thematic clusters and applies relation graphs, comparison tables, and guided Q&A to support layered cross-paper analysis.The relation graph encodes typed citation relationships such as foundation, extension, and substitution.Source: §B.3 · p. 43, Table 11 · p. 43, Table 12 · p. 44
  • Explicit hierarchical structures offer stronger organization and lineage modeling at higher cost, whereas iterative outline refinement is simpler and more adaptive but less explicit about research evolution.The comparison frames planning as a trade-off between explicit evolutionary modeling and flexible refinement.Source: §B.3 · p. 43
  • Figure 12 compares subsection refinement with the relation graph, providing the visual basis for examining these alternative structural-planning strategies.The comparison should be read alongside the textual distinction between explicit lineage modeling and iterative refinement.Source: §B.3 · p. 43, Figure 12 · p. 44

B.4 Code Repository Analysis

Across repository analysis, evidence-grounded drafting, field diagnosis, and MCTS ideation, XCIENTIST converts research synthesis into scoped, inspectable structures that constrain mechanisms and revisions.

B.4 Code Repository Analysis

  • Repository analysis finds a dominant modular, multi-agent architecture while deep understanding, evaluation alignment, long-context handling, and domain generalization remain implementation gaps.PyTorch with Hugging Face Transformers is the convergent stack, while LangChain and LlamaIndex dominate orchestration.Source: §B.4 · pp. 44–45

B.5 Outline-driven Drafting and Citation Enforcement

  • DeepSurvey produces focused, comparative outlines grounded in assigned evidence, whereas AutoSurvey includes off-topic sections and lacks comparative synthesis.The analysis substrate also supports actionable limitations and future directions, while paragraph-level citation checks trigger retries when claims exceed assigned evidence.Source: §B.5 · pp. 45–46, Figure 13 · p. 45, §B.6 · p. 46

C.2 Analysis of Field Structure

  • For training-free agent memory, XCIENTIST separates representation, retrieval, packing, efficiency, and reliability, then uses field consensus and ablation evidence to reject unnecessary auxiliary mechanisms.The resulting proposal is Family-Scoped Expected-Utility Retrieval for Atomic Note Memory, combining bounded retrieval, family reliability, provenance, temporal fit, and deterministic rank-and-pack.Source: §C.4 · p. 48, §C.2 · p. 48, §C.3 · p. 48

C.5 Monte-Carlo Tree Search

  • MCTS refines the root idea through defect-driven edits, retaining failed branches and selecting a traceable sequence that culminates in FamilyMarginalUtilityEstimator.Component retrieval additionally supports novelty comparison, reuse, and cross-domain mechanism transfer rather than surface-level title matching.Source: §C.5 · pp. 49–50, §C.6 · p. 51

C.7 Idea Fusion

XCIENTIST fuses candidate ideas by selecting a compact mechanism, preserving mature infrastructure, rejecting unnecessary coordination, and recording each design decision and repair.

C.7 Idea Fusion

  • The fused thesis is Analytic Family-Calibrated Rank-and-Pack for Training-Free Atomic-Note Memory, centered on calibrated family-aware read-time ranking and a fast closed-form null model.Bounded retrieval, metadata enrichment, quota decay, and microhandles remain supporting infrastructure rather than competing thesis mechanisms.Source: §C.7 · pp. 51–52
  • Fusion resolves conflicts by using bounded-pool reliability adjustment, rejecting HierarchicalMemoryRouter, and replacing raw provenance counts with shrinkage-based analytic calibration.The resulting method is a smaller mechanism with a clearer chain: bounded retrieval, analytic family calibration, and calibrated packing.Source: §C.7 · p. 52
  • The first repair raises the score from 3.99 to 4.04, and the final candidate is retained because later boundary changes do not improve it.Each selected or rejected component has an explicit role or reason, preserving an auditable fusion trajectory.Source: §C.7 · p. 52

D Experiment Validation Case Study

XCIENTIST validates a contract-defined memory mechanism through matched, validator-backed implementation and experiments, while constraining conclusions to supported baseline-versus-full-method evidence and generating an evidence-grounded report.

D Experiment Validation Case Study

  • The idea contract decomposes Training-Free Slotted Evidence Retrieval into four canonical components, each required as a separately ablatable module.The contract targets immutable atomic notes, deterministic enrichment, and capacity-capped slotted retrieval without learned memory-evolution modules.Source: D Experiment Validation Case Study · p. 53

D.1 From Idea Contract to Runnable System

  • All nine implementation steps pass validator checks, and the runtime exposes full, baseline, and ablation configurations through corresponding modules and switches.This makes implementation itself part of the audit trail from conceptual component to executable artifact.Source: §D.1 · p. 53, Table 16 · p. 53

D.2 Standard Science Validation

  • Overall F1 rises from 0.306 to 0.391, a 1.278× relative improvement, while average token length falls from 2844.1 to 1017.2, a 64.23% reduction.The comparison uses the same LoCoMo subset, embedding model, answer generator, and verified dataset identity, with improvements across all reported question categories.Source: §D.2 · p. 54, Table 17 · p. 54

D.3 Controlling Claim Boundaries

  • The validated claim is limited to the matched standard comparison because retained ablation artifacts are diagnostic and the replacement table contains only Baseline and Full Method rows.XCIENTIST therefore avoids promoting outdated or incompatible component-ablation percentages as headline results.Source: §D.3 · p. 54

E Report Writing Case Study

  • The resulting report links problem framing, named implementation modules, experimental outputs, figures, literature references, and future directions into a structured technical narrative.Its central framing presents deterministic slotted evidence retrieval as an alternative to heavier memory-evolution machinery.Source: Figure 15 · p. 56, §E.1 · p. 56

E.2 Implementation-Grounded System Narrative

The report grounds its system narrative in named source-code modules and explains retrieval as a bounded process that packages evidence into explicit slots for downstream question answering.

E.2 Implementation-Grounded System Narrative

  • Each conceptual claim is tied to a named module with a defined role in the write or read path, turning an abstract method into an inspectable implementation.The architecture includes AtomicMemoryStore, ModularAtomicNoteEnricher, UltraSparseFacetHandleIndex, SlottedEvidenceReranker, and DedupMinorityAwareProvenanceAdjudicator.Source: §E.2 · p. 56, Table 18 · p. 57

E.3 Retrieval-Centered Technical Narrative

  • The read path enriches notes with embeddings, entities, temporal tags, QA keywords, and context descriptors before bounded ANN and lexical screening.Candidates are assigned to support, provenance, temporal, and conflict slots under explicit capacity constraints, linking representation to evidence packaging and QA improvement.Source: §E.3 · p. 57

E.4 Experimental Evidence Integration

This section presents XCIENTIST’s contract-governed integration of experimental metrics into auditable reports, emphasizing faithful, component-complete evidence synthesis and validation. It also details source-fidelity auditing and minimal verified repairs to ensure generated claims remain grounded in the underlying code, data, figures, citations, and implementation.

E.4 Experimental Evidence Integration

  • The full method reaches 0.391 overall F1 versus 0.306 for the baseline, a 27.8% relative improvement, while reducing token usage 64.2% from 2,844 to 1,017 tokens per query.XCIENTIST checks these metrics and category-level scores against the source experimental data rather than leaving them as unverified prose.Source: §E.4 · p. 57

E.5 Post-Generation Audit

  • The post-generation audit verifies source-code fidelity, parameter alignment, citation authenticity and completeness, experimental consistency, figure placement, and writing quality, finding no mismatches requiring substantive repair.The audit checks that referenced functions, key parameters, and cited papers match the implementation and article body.Source: §E.5 · pp. 57–58, Table 19 · p. 58

F Key Instructions and Prompt Contracts

  • The prompt contracts require structured, evidence-grounded outputs for paper-graph extraction, baseline and dataset relations, cross-paper synthesis, idea refinement, MCTS expansion and evaluation, code enablement, and science validation.They emphasize recall, exact quotations, independent insight, bounded mechanism-level changes, strict JSON, declared workspace constraints, provenance, real execution evidence, and contract-based validation.Source: F Key Instructions and Prompt Contracts · pp. 59, 60, 61, 62, 64
  • Experiments must compare baselines with the full method on the same prepared real dataset and test exactly one ablation per canonical component.Components must be taken from idea.json.components with their exact names, explanations, and order; smoke, debug, and subset runs do not replace final full-dataset evidence.Source: F Key Instructions and Prompt Contracts · pp. 65–66
  • Workers must execute one science contract through the real command chain, preserve raw evidence, and retain exact component identity and method context for ablations.Validators judge raw evidence and artifacts, reject undeclared targets or non-real data, and return structured PASS, PARTIAL, or FAIL judgments.Source: F Key Instructions and Prompt Contracts · p. 65
  • The convergence contract requires code to pass before standard science and standard science before ablations, with machine-readable status preferred over narrative summaries.Completion cannot be declared when implementation flaws invalidate conclusions, and final ablation evidence must cover all canonical components in the same order without omissions.Source: F Key Instructions and Prompt Contracts · p. 66
  • The audit verifies implementation details, architecture, experimental values, citations, references, and figures against source materials and raw artifacts.Research-integrity checks flag fabricated statistics, unsupported conclusions, missing evidence, and uncited references; confirmed critical issues receive minimal, precise repairs while ambiguous or stylistic changes are skipped.Source: F Key Instructions and Prompt Contracts · p. 66

Related papers

Loading 2606.18874v3…