PaperlayerView exact revision on arXiv

Source-linked AI summary

Artifact-centered Claim-aware Observability for Autonomous Scientific Agents

Xiangyu Yin, Ming Du, Michael H. Prince, Mathew J. Cherukara

arXiv:2608.18312v1cs.CLcs.CYcs.HC

TL;DR

Scientific-agent audits need to expose artifacts and claim–evidence relations, not only model calls or run logs. This paper proposes a compact, portable observability profile and concludes that it complements existing telemetry and provenance standards with an inspectable research graph.

  • Problem

    Existing tracing, experiment-tracking, and archival systems do not make claim, evidence, evaluator, and lineage relations portable first-class audit objects.

  • Method

    The paper proposes an append-only event trace profile covering individuals, operators, fitness records, lineage, archives, runs, streams, and steering commands.

  • Results

    The proposed profile defines an audit target in which reviewers can recover each claim’s evidence, introducing operator, and accepting or rejecting evaluation from pipeline traces.

  • Takeaways & Limitations

    Artifact-centered, claim-aware traces provide a compact semantic layer that complements existing telemetry and provenance standards for inspecting generated research.

  • Takeaways & Limitations

    Verification quality still depends on evaluators, domain rules, and human judgment, while domains must define the granularity of material individuals.

Abstract

from arXiv · show

Autonomous scientific agents now increasingly propose ideas, write code, run experiments, analyze results, and even draft papers. Observe and audit those agents are necessary but logging every model call is not enough, scientists also need to inspect the artifacts and claims that the systems produced and their relations. This is driven by the fact that failures in scientific agent systems are often distributed across several objects. A manuscript claim may cite the wrong evidence, a search process may select a degenerate candidate, a laboratory novelty claim may depend on an unstated rule, or a multi-agent plan may change without a visible trigger. Existing tracing, experiment tracking, and archival provenance tools are valuable, but their native objects do not make these scientific audit relations first-class. We argue that autonomous scientific systems should emit portable, claim-aware artifact lineage as a minimum audit layer. We propose a compact observability profile organized around individuals, operators, fitness records, lineage, archives, runs, streams, and steering commands. In this profile, scientific claims are ordinary individuals with explicit evidence bindings and verification records. The profile is intended as a semantic layer that complements current telemetry and provenance standards. Execution details can remain in OpenTelemetry. Final packages can export to PROV-O or RO-Crate standards.

1 The audit unit is the artifact

The section argues that autonomous scientific systems should treat artifact lineage and claim–evidence bindings as first-class observability records, rather than relying on span trees and run logs alone. This semantic layer complements existing tracing, experiment tracking, and archival provenance while enabling scientific audit questions during execution and richer archival packages afterward.

  • The audit unit is the artifact: The minimum portable audit trail includes candidate artifacts, derivation operators, evaluator outputs, archive or selection decisions, human steering events, and claims with inspectable evidence bindings.These records make artifacts and their scientific relations explicit observability objects.
  • The audit unit is the artifact: Scientific-agent failures often span multiple objects, including coding errors, inconsistent numerical results, unsupported claims, and disputed novelty or yield claims.The section motivates artifact-centered auditing by describing failures that are not localized to one model or tool call.
  • The audit unit is the artifact: Runtime spans record which calls occurred, but they do not identify claims, bind them to measurements, or reveal reuse of invalidated results.Claim-aware artifact observability instead records artifacts, lineage, evaluations, and evidence links.
  • The audit unit is the artifact: The proposal adds a semantic audit layer to tracing, experiment tracking, and archival provenance rather than replacing those systems.Execution can remain represented by conventional traces while scientific inspection focuses on artifacts, lineage, evaluations, and evidence.

2 Why current infrastructure can miss scientific audit objects

Current tracing, experiment tracking, provenance, archival, and version-control systems expose partial infrastructure views but use primary units that do not match scientific audit questions. Autonomous-system audits instead require portable relations among claims, evidence, evaluators, lineage, branches, and interventions.

  • Infrastructure mismatch: Existing infrastructure records spans, runs, artifacts, metrics, model lineage, provenance, final-object derivations, and file changes, but not scientific audit relations as primary objects.Tracing captures prompts, completions, tools, timing, tokens, and call structure; trackers capture runs, parameters, metrics, artifacts, and sometimes model lineage.
  • Infrastructure mismatch: Scientific audits ask which candidate contains a claim, what evidence it used, which evaluator accepted it, and which branch selected it.These questions differ from asking which call happened next, which run produced a metric, how an object was derived, or which files changed.
  • Audit scenarios: Paper-writing agents can produce numerical claims that contradict experiment logs, making the claim, cited evidence artifact, extracted value, and verification result the audit object.The supplied scenario explicitly distinguishes this audit object from the prompt that generated the sentence.
  • Audit scenarios: Evolutionary or tree-search systems can appear to improve while populations collapse into near-duplicates exploiting benchmark quirks, requiring lineage and fitness auditing.The passage identifies lineage and fitness as the relevant audit objects for detecting this failure mode.
  • Portability gap: Lab-specific metadata conventions do not provide reviewers or archives a portable way to find unsupported claims, trace evidence, identify intervention branches, or track reused archive members.Parent IDs and evidence may be stored in spans, commits, database rows, filenames, citations, notebooks, snippets, or table coordinates.

3 Audit questions reveal observability requirements

Scientific audit questions span provenance, epistemic support, fitness, and control history, so observability must represent these relations distinctly rather than relying on file provenance alone. Exposing claims and their risks enables targeted review without replaying entire agent runs.

  • Audit questions: Observability must distinguish provenance, evidence, and control history because scientists ask which artifacts produced outputs, which sources support claims, and which instructions changed searches.These relation types describe derivation, epistemic support, and steering history.
  • Audit questions: File provenance is insufficient because derivation, evidential support, fitness, and steering can diverge across generated text, code candidates, and revised plans.A sentence may derive from a draft without data support; a candidate may inherit a strong parent while its score is unstable; and a revision may follow an instruction without preserving its constraint.
  • Audit questions: Claim-aware traces let reviewers inspect high-risk claims, such as numerical comparisons or novelty assertions, without replaying the entire run.Review can focus on final claims with missing evidence and other targeted risks instead of thousands of execution spans.

4 Claim-aware observable event trace profile

The profile defines a compact, append-only event trace whose derived views preserve inspectable scientific history. It treats claims as ordinary, evidence-bound artifacts while complementing execution telemetry and export standards.

  • Event trace: Append-only traces preserve the events behind archive removals, claim revisions, and plan changes instead of treating dashboard state as the scientific record.Derived views include archive membership, lineage paths, evaluator dashboards, claim graphs, and worker utilization.
  • Core abstractions: The profile comprises five scientific abstractions and three structural records, including individuals, operators, and fitness records.Individuals can be programs, manuscripts, tables, protocols, plans, samples, molecules, figures, claims, or agent versions; operators represent transformations or actions.
  • Portable representation: Domain payloads remain reference- and hash-addressable, while every scientifically relevant transformation is represented as an operator over individuals.This yields stable meanings for parentage, evidence, evaluation, selection, and intervention across domain-specific payloads.
  • Claim inspection: Claims are ordinary individuals with evidence references, optional bindings, and verifier-produced fitness records, making unsupported or contradictory evidence relations queryable.Evidence bindings can record fields read, values extracted, comparisons made, and reasons evidence supports or weakens a claim.
  • Claim inspection: Separate claim, evidence, and verification records allow later systems to invalidate claims without rewriting history, while accommodating automatic, model-based, theorem-proving, human, or laboratory verifiers.A claim is a stable review handle, not a guarantee of truth; contradictions are captured in new fitness records.
  • Standards and packaging: The profile complements OpenTelemetry, PROV-O, RO-Crate, and agent-native research artifacts by preserving a live claim-aware log from which final packages and derivation graphs can be compiled.Execution details can remain in OpenTelemetry, while PROV-O mappings connect individuals, operators, agents, and parent-child relations.

5 Proposed requirements for the community

The paper proposes five minimum observability requirements so autonomous scientific agents can expose auditable artifacts, lineage, evaluations, claims, evidence, and human decisions. These requirements preserve implementation flexibility while making the units needed for scientific review queryable.

  • Minimum requirements: Systems should assign stable individual IDs and metadata to audit-relevant artifacts, including claims, tables, figures, experiment logs, code versions, plans, and selected candidates.Each artifact should include its kind, payload reference or hash, creator operator, timestamp, and status.
  • Minimum requirements: Lineage events should record parent and child individuals, stream, tool or agent, and interpretable parameters for refinements, reviews, selections, and plan updates.This makes the lineage graph replayable and queryable.
  • Minimum requirements: Evaluator outputs should attach to the artifacts they judge, such as code candidates, samples, claims, drafts, or claim sets, rather than floating only at run level.The requirement covers benchmark scores, novelty decisions, and reviewer scores.
  • Minimum requirements: Generated scientific statements should be represented as claim individuals linked to evidence artifacts and, when possible, the exact value or field used.Verification then becomes a normal evaluation step.
  • Minimum requirements: Human steering and archive decisions should remain visible as logged events, while systems may choose their own database and user interface.The requirements expose the units on which scientific review depends without prescribing one implementation.

6 A minimal worked example

A minimal worked example shows how claim-aware artifact graphs expose mismatches between scientific claims and evidence without requiring exhaustive model-call inspection. The same representation generalizes claims across domains as research artifacts with provenance, evidence, and evaluation.

  • Worked example: A claimed 4.2-percentage-point accuracy improvement can conflict with experiment logs, yet the proposed profile represents the event sequence as a graph of records.Span-only traces may bury the claim in a completion while leaving evidence as a tool-output file path.
  • Inspection workflow: Reviewers can query unsupported or contradicted claim individuals, inspect evidence bindings, and follow lineage to drafts and extraction operators through stable artifact IDs.Developers can still inspect model-call spans, and provenance archives can export the final graph.
  • Cross-domain generalization: The pattern applies across novelty, proof, and protein-design claims, each represented as a research artifact with provenance, evidence, and evaluation.Bindings may include samples, diffraction patterns, database entries, thresholds, theorem statements, proof-checker outputs, structure predictions, assays, and acceptance rules.

7 Trace shapes and example encoding

The profile uses one artifact-centered vocabulary across diverse autonomous scientific trace shapes, preserving common audit units while accommodating writing, search, self-modification, multi-agent coordination, and laboratory workflows. These records form an exchange layer beside existing logs, supporting review of lineage, claims, evaluations, revisions, and interventions.

  • Shared trace vocabulary: Six trace shapes—linear refinement, multi-phase pipeline, tree search, evolutionary population, self-modifying artifact, and closed-loop discovery—share the same artifact-centered vocabulary.Across these shapes, the audit units remain artifacts, derivations, evaluations, selections, evidence links, and interventions.
  • Linear refinement and multi-phase writing: Writing traces emit revision drafts, claim-extraction operators, verifier fitness records, and typed handoff artifacts linking final claims to their originating phase.Handoffs can include research questions, code repositories, experiment logs, analysis summaries, drafts, and internal reviews.
  • Tree search and evolutionary populations: Tree search and evolutionary systems encode states or candidates as individuals, expansions or variation as operators, results as fitness records, and survivor sets or Pareto fronts as archive events.These records reveal whether improvement arises from diverse lineages or concentrated ones.
  • Self-modifying and multi-agent systems: Self-modification treats prompts, policies, evaluators, and agent code as individuals, while version-deriving operators and streams preserve acyclic event logs and role-specific coordination.Streams can distinguish a PI, coder, runner, critic, instrument, or human supervisor.
  • Closed-loop laboratories: Laboratory traces represent protocols, samples, outputs, measurements, and claims as individuals, allowing delayed evidence to connect later claims to samples, instrument files, extraction operators, and acceptance criteria.This supports claims about improved yield or novel materials even when measurements arrive after planning ends.
  • Exchange layer: The same compact records can accompany existing logs and hashed or URI-referenced payloads, enabling queries about candidate lineage, unsupported claims, evaluator disagreement, archive churn, plan revisions, and human interventions.The live system may retain its own databases and orchestration runtime while the artifact-centered log serves as an exchange layer.

8 Discussion & Limitations

The profile is intended to support practical audit evaluation while addressing deployment overhead, portability, integrity, redaction, and scope. It exposes evidence, lineage, evaluation, and steering objects without preventing failures or guaranteeing evaluator correctness.

  • Audit coverage: Auditors should recover each reported claim’s evidence, introducing operator, and accepting or rejecting evaluation from traces of scientific-agent pipelines.The baseline should be the best available existing records rather than an intentionally weak comparison.
  • Reviewer utility: Reviewer utility should be measured by time to identify unsupported claims, agreement on failure causes, and reconstruction steps performed outside the trace.The profile is useful only if it reduces review burden rather than adding an ignored artifact.
  • Trace benchmarks: A useful benchmark would pair agent-generated research artifacts with gold audit labels and provide reviewers with schemas, logs, manifests, validator output, and query examples.This package is weaker than full reproducibility but stronger than a prose assurance.
  • Overhead and portability: Tiered logging can limit overhead by using IDs and hashes for most artifacts, richer evidence bindings for selected claims, and domain-specific expansion when audits require it.The same run should export to OpenTelemetry, PROV-O, RO-Crate, and agent-native archives without losing claim, lineage, fitness, archive, or steering relations.
  • Integrity and redaction: Tamper evidence and privacy controls can distinguish missing payloads from missing events and make silent rewriting of the scientific record harder.Proposed mechanisms include append-only streams, signatures, hash chains, redaction records, and access policies; they do not make a bad evaluator correct.
  • Other Limitations: Artifact-centered observability does not prevent hallucination, fabrication, unsafe laboratory action, or benchmark overfitting, but makes the objects needed to detect and investigate failures explicit.Verification quality still depends on evaluators, domain rules, and human judgment, while bad novelty criteria or biased benchmarks become visible objects.

9 Conclusion

Autonomous scientific systems need observability centered on artifacts and the relations among claims, evidence, and evaluation. The proposed minimum audit layer is a compact profile of eight object types and relations.

  • The primary audit object is a candidate artifact together with its claims, evidence, and evaluation relations.
  • Artifact-centered, claim-aware traces are proposed as a minimum observability layer for scientific agent systems.
  • The compact profile comprises individuals, operators, fitness records, lineage, archives, runs, streams, and steering commands.

A Event contract and structural invariants

The profile defines a compact event contract for portable, inspectable artifact-centered traces, while allowing domain payloads to remain in external stores. Its structural invariants preserve resolvable, acyclic audit relations and explicit claim and evaluator accountability.

  • Event contract: The event stream preserves stable identifiers, typed relations, hashes or URIs, and version information needed to reconstruct the audit graph.Domain payloads may remain in notebooks, databases, repositories, instrument stores, or archives.
  • Structural invariants: References must resolve to earlier event objects or be marked as external imports, ensuring that the trace’s links remain inspectable.This is the first of five stated structural invariants.
  • Structural invariants: Derivation edges must be acyclic by artifact version, including when the system modifies its own code, prompt, or evaluator.The invariant applies even to self-modifying system components.
  • Structural invariants: Every final scientific claim requires an evidence binding, a failed verification record, or an explicit unsupported status.This makes claim support or non-support explicit in the audit trace.
  • Structural invariants: Evaluator outputs must identify both the evaluator version and the individual they judged.The invariant ties each evaluation result to its evaluator configuration and judged object.

B Inspection queries … Use of Generative AI Assistance

The paper defines artifact-centered observability through schema-level inspection queries, compact tracelets, and portable review packages that preserve scientific dependencies and claim semantics. It also describes ethical safeguards and discloses limited editorial use of generative AI.

  • B Inspection queries: Schema-level audit queries make common scientific audit questions mechanically answerable across JSONL, SQL, graph, version-control, and experiment-tracking implementations.The queries specify required records rather than a particular database language.
  • B Inspection queries: A system that logs prompts and tool calls but cannot answer the inspection queries provides execution telemetry, not artifact-centered scientific observability.The queries serve as an acceptance test for implementations.
  • C Compact tracelets for common failures: Partial tracelets expose the few records needed to inspect common failures without reproducing the full runtime history.These compact fragments are presented as diagnostic examples in Table C.1.
  • D Packaging, export, and redaction: A portable review package can contain an event stream, payload manifest, validator output, and inspection queries while separating event identity from payload access for redaction.This separation allows private data to be removed without erasing the scientific dependency graph.
  • D Packaging, export, and redaction: Exports should preserve artifact identifiers and map profile objects to OpenTelemetry spans, PROV entities, activities, and agents.RO-Crate can package payloads, manifests, validator reports, and query examples, while tabular producers can expose OpenLineage facets.
  • D Packaging, export, and redaction: Claim semantics should remain explicit because evidence bindings, verifier versions, archive membership, and steering effects are lossy when compressed into free-text metadata.These fields can be preserved directly or in companion files.
  • Ethics Statement: The work targets accountability for generated claims, human steering, and evidence use, while recommending access control, redaction, and signing for sensitive trace data.The authors emphasize that trace data may expose laboratory, personnel, or proprietary information.
  • Use of Generative AI Assistance: Claude Opus 4.8 provided editorial assistance by condensing, anonymizing, and reformatting an earlier manuscript draft; the authors reviewed the text and retain responsibility.The assistance did not introduce a new autonomous agent or real-world deployment.
Loading 2608.18312v1…