PaperlayerView exact revision on arXiv

Source-linked AI summary

F(AI)2R: Who Did What, and Who Checked? Verifiable AI Provenance as an Executable Skill

Florian Krebs

arXiv:2607.25637v2cs.DLcs.AIcs.SI

TL;DR

AI-assisted research contributions are rarely recorded in a form humans or machines can audit. This paper generalizes F(AI)2R into aiprov and an executable agent-operated skill, then applies the method to its own production. The experiment records 2,112 typed words steering 12.7 active hours, nearly all spent on scope, semantics, lineage, or people.

  • Problem

    AI can assist research production, but its contributions are rarely recorded in a form that a later human or machine can audit.

    Source: §I · p. 1

  • Method

    The paper generalizes F(AI)2R into aiprov and packages the method as an executable skill that records its own production under human-only verification rungs.

    Source: §IV · p. 3, §VIII · p. 12

  • Results

    2,112 typed words steered 12.7 active hours, nearly all spent on scope, semantics, lineage, or people.

    Source: §VII · p. 12

  • Takeaways & Limitations

    The paper concludes that the repository can serve simultaneously as the paper, its production process, and the method, versioned and validated alongside its outputs.

    Source: §VIII · p. 12

    The originality claim is bounded by the logged search horizon and remains refutable by widening the sweep.

    Source: §VII · p. 11

Abstract

from arXiv · show

F(AI)2R is FAIR research with AI in the loop, twice: an AI-assisted authoring pass and a machine-readable audit pass over every artefact. AI systems now draft, refactor, and verify research artefacts, yet their contributions are rarely recorded in a form a later human or machine can audit. Building on the original F(AI)2R experiment, we generalize its provenance model beyond scholarly writing into aiprov, a PROV-O extension covering any AI-in-the-loop artefact, and we package the method as an executable skill that an AI agent operates itself: setup asks the human operator for their ORCID ID, resolves their identity from the public registry, and scaffolds continuous integration that gates every push on graph conformance and publishes the current build of this very paper. The paper is its own case study. Every activity, claim, and source in its production is recorded in the repository's provenance graph under two invariants: no parentless claim, and verification rungs that only humans may grant.

I. INTRODUCTION

The paper argues that AI-assisted research needs provenance that records who did what and who checked it, rather than relying on disclosure or trust. It extends the two-pass F(AI)2R approach into aiprov, an executable skill, and evaluates the method on the paper’s own production.

I. INTRODUCTION

  • F(AI)2R frames AI’s role as two passes: authoring an artefact and auditing it into a machine-readable record of who did what, when, and from which sources.The paper generalizes that model beyond scholarly writing.Source: §I · p. 1
  • AI contributions are poorly recorded: commits omit whether an author was human or a model, the instructions and sources, and the cost of a change.Policy statements and after-the-fact documents such as model cards do not provide machine-checkable records at single-claim granularity.Source: §I · p. 1
  • The paper treats provenance as the basis for trust and reserves the top verification rungs for humans.Its stated position is neither to ban AI from research nor to trust it blindly.Source: §I · p. 1
  • The original F(AI)2R experiment established the two-pass model and no-parentless-claim rule for a single scholarly paper, while Obscurity-Is-Dead introduced transcript-as-artifact and per-claim verification labels,.The paper argues per-activity accounting matters when several strands advance in parallel under one operator.Source: §I · p. 1
  • The paper contributes aiprov, an executable skill for AI agents, and a self-study that records its own production, including mishaps and method changes.The skill begins with ORCID-based setup and uses continuous integration to gate pushes on graph conformance and publish the current paper build.Source: §I · p. 1

II. BACKGROUND AND RELATED WORK

The paper builds on provenance, attribution, transparency, and FAIR metadata work, while distinguishing aiprov by its claim-level governance and human verification rules. It positions aiprov as compatible with existing provenance and packaging infrastructure.

II. BACKGROUND AND RELATED WORK

  • aiprov extends PROV-O rather than forking it: every class subclasses a prov: term, preserving compatibility with generic PROV tooling.It also builds on the provenance lineage of the Open Provenance Model and PAV.Source: §II · pp. 1–2
  • Unlike CRediT,, which attributes work at manuscript granularity to humans, aiprov attributes per activity and claim and includes AI systems and deterministic tools.Source: §II · p. 2
  • Compared with model cards and datasheets, aiprov records prompts, sessions, and verification for a specific artefact; its graph can travel inside an RO-Crate.Source: §II · p. 2
  • PROV-AGENT captures agent interactions in PROV graphs, while aiprov adds claim-level governance, human-only verification rungs, and a validator that can fail a build.The passage characterizes capture as recording what happened and governance as determining what enters the record and who may vouch for it.Source: §II · p. 2
  • aiprov uses typed, persistently identified entities in a FAIR Digital Object-shaped record, linking people, sources, and artefacts through identifiers and hashes.This is positioned as a route to integration with metadata infrastructure.Source: §II · p. 2
  • The paper separates machine-checkable reference existence from judgments about whether source content supports a claim, and records who made each judgment.Source: §II · p. 2
  • aiprov combines claim-and-evidence modeling, with explicit agents and human-only verification, distinguishing research collaboration records from media or software supply-chain attestations,.Manubot demonstrated CI-built manuscripts with citation-by-identifier.Source: §II · p. 2
  • The paper traces its roots to AI-assisted research,, engineering provenance, research software engineering, and industrial semantic modeling,.It presents aiprov as extending FAIR metadata practice from data description to collaboration description,.Source: §II · p. 2

III. THE DILUTION CRISIS

The paper describes a dilution crisis in which incentives for counted output meet low-cost AI-generated volume, while retrieval systems rely on proxies that generated text can imitate. It proposes bounded novelty claims and claim-level provenance as a way to make sources and support checkable.

III. THE DILUTION CRISIS

  • Language models compress the effort needed to produce plausible papers, while scientific output already exceeds review capacity and paper mills had industrialized fabrication earlier,.GPT-fabricated papers are already indexed and ranked by scholarly search engines.Source: §III · pp. 2–3
  • Publish-or-perish incentives reward counted output, and language models add a near-zero-cost supply channel without creating that pressure.Source: §III · p. 3
  • Retrieval relies on signals such as indexing, citations, and fluency that generated text can imitate; fabricated references can then propagate and degrade models trained on polluted literature,,.The paper connects this problem to metadata quality and graph-based metadata harvesting,.Source: §III · p. 3
  • A novelty claim should state its recorded search horizon—what was searched, where, and when—so readers can rerun or widen the search.Registry sweeps across Crossref, OpenAlex, arXiv, and DataCite are logged with query terms.Source: §III · p. 3
  • aiprov makes claims first-class records with generating activity, attributed agent, and verification state, distinguishing reference existence from content support and human checking.It records refutations explicitly and links artefacts to DOIs, hashes, commits, and registry-resolved identities.Source: §III · p. 3

IV. THE AIPROV METHOD

aiprov extends PROV-O with entities and activities for recording AI-in-the-loop work, including prompts, transcripts, and inference telemetry. Its invariants make claims traceable and reserve specified verification rungs for human grantors, with validation and audit tools supporting enforcement.

IV. THE AIPROV METHOD

  • aiprov models human, AI, and tool agents; authoring, audit, build, and repair activities; and artefacts, claims, sources, transcripts, and prompts.Prompts are hashed and bound to activities as plans, while transcripts preserve the record needed to inspect why an artefact was produced.Source: §IV · p. 3
  • Activity telemetry includes session and request identifiers, token counts, sampling parameters, cost, energy, and tools; absent values are omitted rather than estimated.Generated artefacts and prompt files receive content hashes.Source: §IV · p. 3
  • No claim may lack a generating activity, an attributed agent, or a verification state; the validator treats violations as hard failures.The validator can run as a pre-commit hook or CI gate.Source: §IV · p. 3
  • Verification promotions must climb the ladder and record a grantor; AI cannot grant rungs 5–6, and refusals are logged without changing state.Human verification requires accessible source material; vendoring is an access mechanism, not itself a verification statement.Source: §IV · p. 3
  • Commit binding requires logging an activity against an existing commit, then committing the graph update separately, adding one commit per logged round.This makes backdating require rewriting published history and makes artefact substitution detectable through its recorded hash.Source: §IV · pp. 3–4
  • Every claim must link to its generating activity, attributed agent, and verification state; activities carry telemetry and connect prompts as plans and transcripts.Figure 1 presents these relationships in aiprov’s PROV-O-based vocabulary.Source: §IV · p. 3, Figure 1 · p. 4
  • The validator operationalizes the invariants through seven SPARQL checks, including hard failures for missing claim links and AI-granted human-only rungs.It also checks source access and warns when human-verified sources are link-only; the same exit-code contract serves pre-commit, CI, and audit use.Source: §IV · p. 4
  • Audit tools report tokens, cost, and rung distributions, extract contributors’ backward closure, and render the graph as an offline ledger.Sources enter through registry searches and DOI verification.Source: §IV · p. 4

V. THE EXECUTABLE SKILL

The executable skill turns aiprov into an agent-operated workflow, from ORCID-first setup through continuous integration that validates the graph and publishes the paper. Human review remains explicit in the verification process and preview workflow.

V. THE EXECUTABLE SKILL

  • The agent logs its activities, records claims at ai-confirmed at most, and never fabricates telemetry or grants a human-only rung.Figure 4 distinguishes human operator steps, deterministic CI, and AI-agent steps in the operating loop.Source: §V · p. 4, Figure 4 · p. 6
  • Setup asks for one ORCID iD, resolves the operator’s identity from the public registry, derives the graph base from the git remote, and scaffolds CI and a paper skeleton.If identity resolution fails, only the bare iD is recorded; later commands detect the graph base automatically.Source: §V · p. 4
  • Every push validates the graph, renders the dashboard and disclosure, compiles the paper, and publishes build artefacts; conformance violations block the build.Version-tagged releases include the PDF, source graph, metrics snapshot, packaged skill, and checksums.Source: §V · p. 4
  • The preview republishes the current build after editing rounds and exposes a source-verification queue whose authenticated human grants or refusals are transcribed into the graph.The human retains the judgement; only its transcription is automated.Source: §V · p. 4
  • The agent drafts a reviewable section structure before prose, allowing the operator to contest argument flow, evidence, and figure and citation plans earlier.Both structure and prose passes are logged, and two ladder rungs were redesigned this way.Source: §V · p. 5
  • The prospective experimental-data workflow would hash raw records, log analysis builds, rerun them in CI, and record AI-assisted findings with verification rungs.It is presented as a design, not an implemented deployment.Source: §V · p. 5
  • Auditors can rerun graph validation, re-hash artefacts, re-resolve DOIs, and inspect the activity log and transcripts; violations identify the offending node and exit nonzero.A clean graph exits zero, while the human-rung access gate supplies review material after the source is vendored.Source: §V · p. 5
  • Sandbox tests show the validator refusing an AI’s human-only promotion and a parentless claim, while a human promotion proceeds after the source is vendored.The refusal and access-gate outcomes are captured in Figure 3.Source: Figure 3 · p. 6, §V · p. 6

VI. META-EXPERIMENT: THIS PAPER AS CASE STUDY

This one-operator, one-domain, one-session demonstration records the paper’s production in a provenance graph, making activities, checks, and incidents inspectable. The graph distinguishes machine-checkable records from human judgments and self-reported telemetry that an auditor cannot independently verify.

Setup

  • The demonstration used one operator, one domain, and one session, so its quantities characterize this instance rather than a broader study.The operator’s ORCID identity, an AI system, and the CI service were registered as agents.Source: §VI · p. 6

Results

  • All eight recorded claims remained at ai-confirmed, while the table’s token totals came from provider-reported session usage and its cost was computed from published prices.The cost’s price basis was recorded in the activity label; the dashboard’s aggregate backfills mean Table II is authoritative for token and cost totals.Source: §VI · pp. 6–7, Table II · p. 7, Figure 6 · p. 7
  • Content checks compared each cited source with its specific citing sentence, recorded supporting quotations and checked depth, and repaired three citation overreaches before promotion.A bot-wall deferred one check, operator-owned repositories remained for operator verification, and negative outcomes stayed recorded.Source: §VI · p. 7
  • Four incident types—including an overwrite, malformed registry data, blocked access, and contested rung semantics—were retained as ordinary activities with their repairs or outcomes.The overwrite was reverted, and the registry failure and fix were recorded in commits and a CI build activity.Source: §VI · pp. 7–8
  • The division-of-labour record makes graph-queryable activity counts and human grants distinguishable from judgments reconstructed from the session log.The operator’s direction messages carried scope and contested decisions; the AI performed authoring, audit, and repair, while CI performed deterministic builds.Source: §VI · p. 8

Limitations

  • Auditors can check hashes, commits, DOI resolution, rung grants, and validator verdicts, but cannot verify the honesty of self-reported telemetry.The reported leverage figures state their derivation rules, while the solo-baseline cost is deliberately not estimated.Source: §VI · p. 8

VII. DISCUSSION AND LIMITATIONS

The experiment measured the overhead of making work auditable and recorded an AI overwrite as a recoverable event. The discussion also identifies limits in telemetry, human verification, energy reporting, and copyright scope, while arguing that the method extends beyond papers.

Self-report honesty

  • The overwrite incident was recoverable and recorded, but the graph remains only as truthful as its writer.Mitigations include content hashes, commit bindings, server-side CI validation, and human-only verification rungs.Source: §VII · pp. 8–9

From honesty to evidence

  • Provider-signed logs could reduce agent misreporting, but would not establish which files were touched, how an inference binds to an artefact, or whether a judgement is sound.The proposed dual witness links provider-signed inference records with repository-side commit bindings; signatures shift rather than eliminate trust.Source: §VII · p. 9

Cost and overhead

  • 12.6% of requests, 11.8% of output tokens, and 9.0% of computed session cost were attributed to bookkeeping; 87 of 222 commits contained only graph or transcript updates.The request shares are the method overhead shown in Figure 8; the session classification found no mixed bookkeeping and substantive requests.Source: §VII · p. 9, Figure 8 · p. 9

Cost and environmental questions

  • The record reports 860 million cache-read tokens and USD 1,641 at list prices, but the paper does not settle whether those costs are justified; energy remains unreported.The schema reserves aiprov:energyWh, but it is empty because no provider reported energy.Source: §VII · p. 10

Generality

  • The same invariants and verification ladder apply to code, data pipelines, CAD models, and AAS submodels, while entity subclasses remain domain-specific.The graph is provider-agnostic and the skill follows an open agent-skills format; only three capture shims are client-specific.Source: §VII · p. 10

Human-only verification

  • Human verification records who vouched and on what evidence, but cannot machine-check whether that person read carefully, understood correctly, or judged well.The human-only rungs mark the boundary between structural verification and attributable judgement, not a claim that people are more reliable than machines.Source: §VII · pp. 9, 11

Copyright

  • Because most repository code and prose was AI-generated under human direction, the discussion says much may be uncopyrightable and the license may cover only human contributions and the curated whole.The per-activity graph records which expression a human conceived, selected, arranged, or modified, and which the machine produced.Source: §VII · pp. 11–12

Leverage

  • In the paper’s account, 2,112 typed words steered 12.7 active hours, with nearly all of that time spent on scope, semantics, lineage, or people.Source: §VII · p. 12

VIII. CONCLUSION

The paper generalizes F(AI)2R into aiprov, packages it as an executable skill, and applies it to its own production. It frames provenance infrastructure as technical support while AI-related scientific norms remain under social debate.

VIII. CONCLUSION

  • The paper generalizes F(AI)2R into aiprov, packages it as an executable skill, and applies it to its own production, including mishaps and redesigns.The repository is described as the paper, the process, and the method.Source: §VIII · p. 12
  • The paper connects its approach to AI-assisted research practice, engineering traceability, and formal semantic modeling.It characterizes provenance as the semantics of work where semantics matter.Source: §VIII · p. 12
  • Aiprov could complement shepard by recording who, human or AI, shaped research outputs and how those outputs were verified.The shepard integration remains planned work; the paper identifies further development needs, including multi-agent attribution and energy accounting.Source: §VIII · p. 12
  • The proposed integration path envisions machine-actionable aiprov records being harvested across institutions, with shepard integration as a concrete first step.The paper describes this as an ecosystem-level vision and identifies reviewer-library, backward-closure, and dispute designs as future increments.Source: §VIII · p. 12
  • The paper does not settle how science with AI should develop; it offers technical support while norms around authorship, transparency, credit, originality, cost, and trust are negotiated socially.It calls for debate and offers its repository record, graph, transcript, incidents, and bill as material for that discussion.Source: §VIII · p. 13

Related papers

This paper cites

Loading 2607.25637v2…