Source-linked AI summary
CrossAudit: A Git-Native, Cross-Vendor Audit Loop for Agentic Science
Zhaohe Dong, Yuhao Chen
TL;DR
Autonomous research systems increasingly combine generation and review within the same vendor ecosystem, while their audit histories may remain difficult to replay. CrossAudit proposes cross-vendor, rule-bound supervision recorded in git, and reports a live deployment plus a seeded-defect trial whose corrected results show differing vendor interpretations without establishing superiority.
Problem
Agentic science needs supervision that addresses same-source review and opaque, non-replayable audit histories.
Method
CrossAudit audits each increment with a different vendor against a human-authored versioned rulebook, runs deterministic checks first, and records parsed supervision decisions in git.
Results
The reference implementation and related computational-chemistry deployment demonstrate the protocol’s practical use, while the seeded-defect trial reports differing vendor readings without proving either vendor better.
Takeaways & Limitations
CrossAudit’s practical contribution is a re-inspectable supervision record that a single researcher can adopt with two repositories and two API keys.
Takeaways & Limitations
The protocol does not cover operator tampering, corpus-correlated error, raw model exchanges, or full replay of stochastic model calls.
Abstract
from arXiv · showhide
An AI scientist should not grade its own homework. Yet in the systems we examined, the agent that reviews the work usually comes from the same model family as the agent that produced it, or at least from the same vendor. Model evaluators are known to favour their own generations. Whether models trained alike also share blind spots is a conjecture, not a settled finding, but if they do, the reviewer inherits the author's. The record of what was flagged and what was waved through often sits in platform logs that nobody outside can replay. We present CrossAudit, a protocol for supervising autonomous research pipelines. It rests on three commitments. Each increment of work is audited by an agent from a different vendor against a rulebook a human wrote and versioned. Reports, verdicts, disputes and rulings are git commits, so the supervision history can be re-read and cited; raw model exchanges are not yet part of that record. Scripted checks run before any model does. Advisory judgement never gates the pipeline: a model blocks only by citing a rule, and no model may waive a deterministic failure. Blockers that survive a bounded number of revision rounds go to a person. We state the protocol as eight invariants. We describe a reference implementation built from GitHub Actions and a few hundred lines of Python, and report a live deployment of a closely related variant in a computational-chemistry pipeline. We also ran a seeded-defect trial (30 increments, 43 seeded defects, one run per configuration). A cross-vendor audit of our own repository then voided its blinding. We adopt that audit's findings and report the corrected results. The trial shows that two vendors read the same rulebook differently. It does not show that either is better. The strongest evidence here is the committed, uncontrolled record of cross-vendor audits of this paper itself.
1 Introduction
Agentic research systems make autonomous science increasingly capable while intensifying longstanding reliability concerns. CrossAudit addresses the resulting supervision problem with cross-vendor, rule-bound, inspectable auditing, while explicitly limiting its guarantee to recorded auditability rather than scientific truth.
- Motivation: Agentic pipelines can propose hypotheses, execute experiments, and draft manuscripts with limited human intervention.
- Motivation: Agentic pipelines may systematically log research increments, yet can also mass-produce plausible wrong results faster than human audit cultures can absorb.
- Problem: Same-vendor supervision is structurally exposed to correlated failure because critics and creators commonly share a model family or vendor.
- Contribution: CrossAudit defines eight invariants for vendor-heterogeneous, re-inspectable, third-party-auditable supervision histories.
- Contribution: The reference implementation uses two git repositories, a CI service, and two model API keys, alongside a live computational-chemistry deployment and an explicit threat model.
- Scope: CrossAudit records rule-based evaluation and parsed decisions, but does not certify scientific truth, preserve raw exchanges, or provide full process replay.
2 Related work and the same-source problem
Prior research agents use internal critique, tournaments, or operator-hosted traceability, but generally do not commit to vendor-heterogeneous reviewing. CrossAudit instead treats supervision history as a versioned, citable artefact and moves deterministic checks before publication.
- Existing systems: Research agents automate ideation, experimentation, writeup, review, hypothesis tournaments, literature synthesis, and laboratory protocols.
- Same-source problem: Existing systems do not commit to vendor heterogeneity in the reviewing layer.
- Existing systems: Platform traceability can link conclusions to code or literature, but remains operator-controlled without a second-party auditor or off-platform replayable ledger.
- Same-source problem: LLM evaluators may prefer their own generations and exhibit position, verbosity, and self-enhancement biases; reproducibility therefore does not guarantee unbiased supervision.
- Same-source problem: Later studies qualify rather than overturn self-preference findings, attributing some preference to genuine quality differences and leaving roughly half of significant cases after identity blinding.
- Supervision as an artefact: CrossAudit makes supervision history a first-class, versioned, citable research artefact, while deterministic tools such as statcheck operate post-publication.
3 The CrossAudit protocol
CrossAudit organizes autonomous research supervision around vendor separation, human-authored rules, deterministic checks, and a git-based audit record. Its bounded, fail-closed loop distinguishes objective blockers from advisory judgments and sends unresolved cases to a human.
- Separation of powers: CrossAudit separates Generator and Auditor permissions across Science and Audit repositories, while only the human principal writes the rulebook.The Auditor reads committed artefacts rather than the Generator’s prompts, reasoning, or conversational state.
- Eight invariants: The protocol targets eight invariants covering vendor heterogeneity, ledger completeness, citation validity, deterministic precedence, bounded revision, graded interruption, receipt binding, and fail-closed admission.The reference implementation targets all eight but enforces only a subset.
- Determinism first: Deterministic checks run before model auditing; a hard failure blocks regardless of model opinion, and no model may waive it.The deterministic layer includes schema, unit and dimension, convergence, and provenance checks.
- Rounds, disputes, termination: Revision is bounded at max rounds, disputes return a finding to the Auditor once, and every increment terminates in PASS or human escalation.The bound prevents indefinite disagreement between Generator and Auditor; disputes must be grounded by the principal.
- Graded interruption: Findings are either BLOCKER, which gates an increment, or ADVISORY, which is recorded without gating; computation continues, and humans are interrupted only by ESCALATE.Blockers include deterministic failures, contradictions, missing provenance, and method–declaration mismatches.
- Receipts and admission: Each verdict receipt binds the audited commit, derived artefact manifest, applicable versions, auditor identity, and report commit; absent or invalid receipts deny admission by default.Receipt binding prevents a receipt from referencing an uncommitted report, while fail-closed admission escalates missing, stale, conflicting, or unbound receipts.
4 Reference implementation and deployment
CrossAudit’s reference implementation combines GitHub-based, cross-vendor auditing with deterministic checks and bounded escalation, while a related chemistry deployment demonstrates unattended operation but exposes specification and evaluation limits. Subsequent audits corrected the paper’s blinding claims and show that the seeded-defect trial supports disagreement between configurations, not comparative superiority.
- Reference implementation: The reference implementation uses GitHub and two model API endpoints, deriving audited commits and rounds from repository state rather than caller-supplied payloads.The audit workflow checks out the science repository at the pinned commit, runs deterministic checks, and invokes the auditor with the rulebook and increment context.
- Live deployment: A live computational-chemistry variant runs a Claude-based research agent through committed policies, an epistemic-status ledger, a retraction log, and controller-opened audit cycles.The deployment screens perovskite additive calculations on Alibaba Cloud E-HPC and a GPU worker, with each increment pushed to a private science repository.
- Specification refinements: The deployment diverges from the reference design through a tool-bearing auditor, externalized controller state, and a four-level severity ladder instead of the reference’s two levels.These choices weaken the repositories-alone replay ideal and leave the deployment running against an argument specified for two severity levels.
- Live deployment: The deployment’s first cycle returned BLOCK on manifest-hash and commit-provenance defects, freezing production submission and claim publication until resolution while leaving running jobs uninterrupted.The system gates records and future submissions rather than interrupting E-HPC work already in flight.
- Live deployment: 21 model-audited cycles contained 39 finding occurrences across 23 identifiers; 20 distinct findings were behaviorally confirmed closed and 3 remained open at the second freeze.Among closed findings, median closure lag was one cycle, mean 2.10, and maximum 7; these later cycles were observational telemetry without a control condition.
- Specification refinements: A second implementation exposed three protocol underspecifications, including admission as a tier rather than a binary switch and routing decisions as ledgered trusted-base actions.The specification also distinguishes protocol generality from the reference implementation’s domain-specific deterministic checks.
5 Threat model and limitations
CrossAudit addresses several residual threats rather than claiming to certify scientific truth: it makes audits inspectable, but leaves important risks involving shared corpora, operators, confidentiality, and adversarial inputs.
- Threats: Shared literature can produce confident agreement between vendors on the same wrong or outdated information, so additional LLM auditors cannot be assumed to remove this residual.Mechanisable checks and human reading of advisory trails are the stated mitigations.
- Threats: Prompt injection cannot waive deterministic checks, but it can still shape the LLM-side report, leaving a partially open risk.The reference auditor runs toolless against a fixed schema with increment content fenced as data.
- Threats: Malformed, empty, or rule-free auditor reports escalate rather than pass silently, trading unreliable judgments for human attention.This treatment follows invariant I3.
- Threats: Bounded revision prevents oscillation and deadlock from continuing indefinitely: unresolved cases become human decisions with the parsed exchange preserved on the ledger.The intended floor is escalation with an inspectable record.
- Threats: Operators remain outside CrossAudit’s guarantees because selective publication, history rewriting, and audit re-rolling are not prevented without external anchoring.The resulting third-party-auditability claim is limited to internal consistency as published.
- Scope: Confidentiality obligations for sending unpublished or patentable increments to a second vendor must be settled separately for each deployment.Agreements, retention, and disclosure risk sit outside the protocol.
- Scope: The protocol does not cover miscalibrated instruments, honestly described wrong datasets, upstream fraud, or attribution in multi-agent generation.It is currently specified for single-generator pipelines.
6 Discussion
The discussion frames CrossAudit as continuous integration for scientific claims: a low-cost, inspectable quality floor that shifts human attention toward rules, escalations, and science while creating reusable audit infrastructure.
- Continuous integration for scientific claims: CrossAudit aims to provide a cheap, inspectable floor under scientific quality rather than proof of correctness, analogous to continuous integration in software.Its distinctive stochastic judge is kept from controlling the hard gate by the deterministic check layer.
- Continuous integration for scientific claims: The protocol is intended for unilateral adoption by one researcher using two repositories, model credentials, CI, and branch-protection configuration.The audit loop itself is designed to fit an afternoon, while admission enforcement requires additional configuration.
- Relation to platform-scale systems: CrossAudit complements generation systems such as AI Scientist and AI co-scientist by supervising whatever platform generates rather than replacing generation.The proposal externalizes supervision across vendor lines and onto public ledgers, subject to organizational and confidentiality costs.
- What the ledger buys science socially: A complete supervision history lets reviewers inspect rules, disputes, and differently trained model judgments instead of rebuilding trust from prose and reputation alone.The ledger is public where the operator publishes it.
- The ledger as a data asset: Each increment, finding, dispute, and resolution can become labeled training data and contribute to a field-wide error taxonomy when deployments are openly aggregated.The resulting corpus could provide training signal for generator and auditor agents alike.
- Constitutions as community objects: Stable rule IDs and versioned receipts allow communities, journals, and funders to maintain and require domain-specific Constitutions without ambiguity about governing standards.Constitutions are described as plain, forkable files that can evolve across deployments.
- A console for the human principal: A supervision console is proposed because raw git is demanding for the human whose asynchronous reading remains load-bearing.The console would expose timelines, advisory backlogs, escalation requirements, and standards-ratchet telemetry.
- Management by exception, at research scale: The proposed economic benefit is a hypothesized reduction in human gating effort from O(increments) to O(escalations), conditional on low escalation volume and not yet measured.No deployment recorded the human attention spent, so this remains a measurement plan rather than a result.
7 Conclusion
CrossAudit reframes autonomous-research supervision as a public, versioned artefact built from independent reviewers, explicit rules, deterministic checks, bounded revision, and git. The paper also states that its protocol, studies, and evidential judgments remained author-owned despite heavy language-model assistance in drafting and implementation.
- 7. Conclusion: CrossAudit reframes supervision from a platform feature into a public, versioned artefact produced by structurally independent reviewers.Its core ingredients are a second vendor, stable rule IDs, pre-model scripts, a bounded loop, and git.
- 7. Conclusion: The authors retain responsibility for the protocol, invariants, study designs, and judgments about what the evidence supports despite heavy language-model use.A language model drafted much of the prose and wrote much of the implementation and analysis scripts under author direction.