PaperlayerView exact revision on arXiv

Source-linked AI summary

From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents

Zijie Zhuang, Changxin Lao, Pengbo Xu, Hanwen Xu, Ruochen Yang, Yingzhi He, Peng Zhang, Jiangxia Cao, Yusheng Huang, Guohong Mu, Jian Liang, Ruiming Tang, Shuang Yang, Zhaojie Liu, Wenwu Ou, Kun Gai

arXiv:2608.05235v1cs.IRcs.AI

TL;DR

Completed research trajectories do not by themselves establish which claims their evidence supports. The paper verifies consequential artifacts and qualifies claims across rounds before retaining records; its results include non-monotonic adaptation and positive online lifts from two complete-workflow candidates.

  • Problem

    A completed trajectory may contain unsupported artifacts, invalid or confounded experiments, and later modifications that obscure earlier findings, leaving unclear which claims are supported.

    Source: Introduction · p. 1

  • Method

    The framework combines bounded verification with cross-round qualification of intervention claims, retaining supported findings with provenance and applicability boundaries.

    Source: Introduction · p. 2

  • Results

    Across paper-to-target adaptations, final states often lose earlier gains; two complete-workflow candidates yielded positive online lifts over frozen production baselines.

    Source: Conclusion · p. 8

  • Takeaways & Limitations

    A trajectory is a source of candidate evidence requiring explicit verification, rather than experimental knowledge by itself.

    Source: Conclusion · p. 8

    Online case studies show candidate-level performance, not framework-level attribution, absent a parallel plain-agent or no-memory arm.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings and retain the resulting trajectories to guide later decisions. Yet a completed trajectory is not automatically evidence: generated artifacts may be unsupported or incomplete, executed rounds may be invalid or confounded, and later modifications may obscure earlier findings. We study \textbf{trajectory-to-evidence conversion}, asking what a completed research process has actually established. We introduce an evidence-grounded framework that couples bounded verification of consequential artifacts with post-execution claim qualification. A context-isolated generate--verify--repair process checks artifacts for evidence violations and missing downstream requirements before release. After execution, validity and attribution checks consolidate evidence across rounds, qualify intervention-level claims as actionable repairs, diagnostic guards, or withheld findings, and preserve admitted claims as auditable records with explicit provenance and applicability boundaries. A hybrid LLM-assisted controller subsequently applies, defers, or rejects records based on available target evidence. Record audits characterize which claims survive qualification, while downstream diagnostics identify affirmative applicability judgment as a bottleneck for the tested controller. Across paper-to-target adaptations, later rounds often improve on the first, while final rounds frequently underperform an earlier best, exposing non-monotonic trajectory evolution. Candidates produced through the complete workflow also yielded positive online lifts relative to deployed baselines.

Introduction

Research agents produce and retain extended experimental trajectories, but those trajectories do not by themselves establish which conclusions the evidence supports. The paper frames trajectory-to-evidence conversion as qualifying and preserving supported intervention claims.

Introduction

  • Research agents conduct iterative experiments whose retained trajectories include proposals, code states, logs, measurements, failures, and revisions.Prior experience is retained to guide later decisions.Source: Introduction · p. 1
  • A completed trajectory may contain unsupported artifacts, invalid comparisons, confounded results, abandoned hypotheses, or later modifications that obscure earlier gains.Even valid findings are bounded by their observed mechanism, target condition, and evaluation protocol.Source: Introduction · p. 1
  • Existing agent systems use feedback and retained history to guide optimization, while leaving qualification of persistent experimental claims largely implicit.The framework combines bounded artifact verification with post-execution checks of validity, attribution, and applicability before preserving findings with provenance and conditions.Source: Introduction · pp. 1–2
  • The paper frames trajectory-to-evidence conversion as identifying intervention-level claims supported by evidence across rounds, rather than treating a completed round or final state as knowledge.Only findings supported under explicit conditions are preserved as records.Source: Introduction · p. 1
  • The framework’s contributions include cross-round evidence qualification and identify non-monotonic adaptation and unreliable affirmative applicability decisions as bottlenecks.The tested controller’s decisions are evaluated separately against reference applicability and post-execution outcomes.Source: Introduction · p. 2

Related Work

Prior research agents execute or reproduce workflows, and language agents retain experience to guide later decisions. This paper distinguishes evidence qualification from workflow execution, memory reuse, and experimental lineage.

Research Agents and Automated Experimentation

Trajectory Memory and Experience Reuse

Framework Overview

The framework verifies consequential outputs before handoff, then qualifies evidence across executed rounds. Only qualified claims become frozen records for downstream decisions.

Framework Overview

  • Before consequential outputs move downstream, a verification harness checks available evidence, constraints, and downstream requirements; producers repair identified issues, and unresolved outputs are withheld after the attempt limit.This process applies during source adaptation.Source: Framework Overview · p. 2
  • After execution, rounds addressing the same intervention and locus are grouped to determine which claims their evidence supports and under what observed conditions.Only qualified claims become records, which are frozen before downstream reuse.Source: Framework Overview · p. 2
  • A controller applies, defers, or rejects each frozen record using the target mechanism, conditions, and available observations.Applied Repairs become method contracts, while applied Guards constrain or redirect subsequent planning.Source: Framework Overview · p. 2

Problem Setting

The setting builds reusable records from source adaptation trajectories and evaluates them in a separately initialized target episode. Adaptations must preserve their defining mechanism, and records carry explicit evidence and applicability boundaries.

Problem Setting

  • A paper-to-target adaptation changes a target setting while preserving the paper’s defining mechanism; the setting includes the initial repository, data, evaluation utility, and implementation and deployment constraints.An adaptation that removes the defining mechanism does not count as successful.Source: Problem Setting · p. 2
  • An agent may dispatch up to B machine-executed experiments, each producing a code state, runtime observations, and a measured outcome; trajectories can include invalid experiments, retries, and non-monotonic results.The resource limit counts machine experiments, not agent-side verification.Source: Problem Setting · p. 2
  • Source and target tasks may use the same paper or method, but record construction and downstream use occur in separately initialized episodes.The registry is frozen before target evaluation; only Repair and Guard claims yield reusable records, while Withheld candidates remain unavailable.Source: Problem Setting · p. 2
  • Figure 1 shows verified source claims qualified as Repair, Guard, or Withheld, followed by target-side applicability decisions and outcome audit; applied Guards constrain subsequent planning.At consequential handoffs, identified issues or omissions trigger bounded repair before release.Source: Figure 1 · p. 3
  • Each record contains source context, intervention, measured outcome, supporting evidence, and an applicability boundary.The context identifies the source mechanism, target condition, and triggering observation; the intervention specifies an action and its locus.Source: Problem Setting · p. 3

Per-Step Verification Harness

The harness checks consequential artifacts for unsupported content and missing downstream information before release. Separate verification contexts and bounded repair keep these checks focused on evidence and handoff requirements.

Per-Step Verification Harness

  • Before release, consequential artifacts are checked for conflicts with available evidence or constraints and omissions required by the next step.Consequential outputs can change downstream inputs, determine experiment dispatch, or contribute to an experimental claim.Source: Per-Step Verification Harness · p. 3

Evidence and Constraint Check

  • The evidence-and-constraint verifier raises localized objections grounded in workflow evidence and artifact-specific criteria, rather than rewriting artifacts.Evidence may include the source paper, repository state, tool output, execution logs, or measured results.Source: Evidence and Constraint Check · p. 3

Downstream Requirement Check

  • A separate requirement reviewer reconstructs the next step’s information needs without seeing the candidate artifact or producer rationale, then returns missing requirements.The downstream contract can require implementation details, execution interfaces, attribution evidence, or boundary information.Source: Downstream Requirement Check · pp. 3–4

Repair and Handoff

  • The producer revises artifacts against objections and missing requirements, and both verifiers recheck them for at most K_t attempts before release or branch stoppage, rollback, or escalation.Release requires both checkers to report no unresolved issues or omissions.Source: Repair and Handoff · p. 4

Experimental Claim Qualification

The framework separates whether an experiment was validly executed from whether its result supports an attributable claim. It qualifies evidence across related rounds and preserves supported findings as bounded, auditable records.

Experimental Claim Qualification

  • Passing pre-execution artifact checks establishes readiness for downstream execution, while intervention effectiveness is assessed from the resulting experiment.The implementation, outcome, interpretation, and summary re-enter the harness as experimental evidence.Source: Experimental Claim Qualification · p. 4
  • Execution validity and claim attribution are assessed separately; a valid negative or null result supports a persistent Repair or Guard only when attributable and bounded to tested conditions.Insufficiently attributable runs remain diagnostics.Source: Experimental Claim Qualification · p. 4
  • Execution validity requires all four gates: implementation fidelity, completion of the prescribed run, protocol-following evaluation, and preservation of the paper’s defining mechanism.Failed-gate runs remain operational diagnostics and are excluded from persistent claims.Source: Experimental Claim Qualification · p. 4
  • Rounds addressing the same intervention and locus are grouped, while invalid, confounded, duplicated, or superseded rounds are excluded from qualification.Related valid rounds can provide primary, corroborating, or boundary evidence.Source: Experimental Claim Qualification · p. 4
  • Candidates become an actionable Repair, diagnostic Guard, or Withheld finding; admitted records link implementation, execution, protocol, measurements, attribution checks, and reuse conditions.A Repair requires an implementable intervention, attributable valid evidence, an explicit trigger, and an applicability boundary; a Guard requires an attributable failure mode under observed conditions.Source: Experimental Claim Qualification · p. 4

Downstream Use of Experimental Records

A hybrid controller assesses retrieved experimental records against target-task evidence and assigns reuse decisions. Applied Repairs become method contracts, while Guards constrain planning or diagnosis.

Downstream Use of Experimental Records

  • The controller retrieves potentially compatible records and assesses mechanism, condition, and intervention compatibility using structured checks and an LLM-based assessment.The assessment draws on the target method’s defining mechanism, target conditions, and available observations.Source: Downstream Use of Experimental Records · p. 4
  • It assigns Apply when compatibility, triggers, boundary compliance, and target-mechanism preservation are supported; Defer when required evidence is missing or ambiguous; and Reject when evidence shows incompatibility or a boundary violation.An Apply decision dispatches a target-specific experiment whose scientific outcome is determined after execution.Source: Downstream Use of Experimental Records · p. 4
  • An applied Repair becomes an implementable method contract evaluated by the per-step harness, whereas an applied Guard becomes a planning constraint or diagnostic directive without dispatching an intervention by itself.Source: Downstream Use of Experimental Records · p. 4

Experiments

Across experiments, adaptations often improve after the first round but do not preserve gains monotonically; artifact verification improves selected handoffs while remaining incomplete, and downstream record use exposes controller weaknesses.

Experimental Setup

  • Evaluation reserves intervention-level attribution for equivalent base implementations with memory isolated; other matched executions are interpreted at the task level.The machine-experiment budget counts dispatched experiments, not agent-side verification.Source: Experimental Setup · p. 5

Bounded Self-Verification

  • Three retries raise paper-to-proposal acceptance from 73.3% to 100%; code-loop gains center on completeness and observability, while alignment barely changes.Same-model context isolation may leave correlated blind spots, and source coverage of Method sections remains incomplete.Source: Table 1 · p. 5, Bounded Self-Verification · p. 5

Non-Monotonic Adaptation Trajectories

Automated Reproduction on a Public Benchmark

Experimental Record Audit

  • Eight of 14 candidate claims become Repairs, one becomes a Guard, and five are withheld; the nine retained records have explicit triggers and boundaries.All 34 referenced source artifacts match their archived SHA-256 values, and all five withheld candidates are excluded from reusable entries.Source: Experimental Record Audit · p. 5, Table 4 · p. 6

Memory-Conditioned Adaptation

  • Among five reference-Apply pairs, three are beneficial, one neutral, and one harmful; among three manually dispatched Defer/Reject pairs, two are harmful and one neutral.Reference applicability filters plausible reuse, whereas execution determines the target-specific outcome.Source: Memory-Conditioned Adaptation · p. 6

Controller Error Analysis

  • 25% affirmative precision and 67.2% accuracy trail an always-Reject rule's 79.7% accuracy, but aggregate accuracy rewards inactivity and is not an adequate operational objective.The controller’s false affirmative decisions leave it without a reliable balance between conservative filtering and experimental progress.Source: Controller Error Analysis · pp. 6–7, Table 7 · p. 7

Production Case Studies

Limitations

The production case studies support candidate-level comparisons against frozen baselines, not framework-level attribution. Small, class-imbalanced diagnostics also limit generalization.

Limitations

  • Candidate-level online performance is compared with frozen baselines, not attributed to the framework without a parallel plain-agent or no-memory arm.Stochastic upstream changes also mean attribution is interpreted at the task level, except for code-inspected equivalent implementations.Source: Limitations · p. 7
  • Attribution is interpreted at the task level when stochastic upstream changes alter downstream artifacts, except for code-inspected equivalent implementations.Source: Limitations · p. 7
  • The record-use diagnostic has only eight singlerun pairs, and class imbalance in controller evaluation limits conclusions about generalization.Source: Limitations · p. 7

Conclusion

The paper frames trajectories as sources of candidate evidence that require explicit verification, and presents a framework for turning supported findings into auditable records. Across paper-to-target adaptations, later rounds often improve on the first, but final rounds frequently lose earlier gains; two candidates also achieved positive online lifts over frozen production baselines.

Conclusion

  • Two candidates produced through the complete workflow yielded positive online lifts over frozen production baselines.Source: Conclusion · p. 8
  • Across paper-to-target adaptations, final states often lose earlier gains, and several candidate claims lack sufficient evidence for retention.Source: Conclusion · p. 8
  • Downstream diagnostics identify affirmative applicability judgment by the tested controller as a remaining bottleneck.Source: Conclusion · p. 8
  • The paper presents a framework that verifies research-agent experiments and converts supported findings into auditable, boundary-aware records.Source: Conclusion · p. 8
Loading 2608.05235v1…