Source-linked AI summary
What Git Does Not Record: Provenance Legibility in Two AI-Assisted Research Repositories
Koutian Wu, Zesen Huang, Hejia Geng
TL;DR
AI-use policies ask authors to reconstruct contributions after the work, but Git repositories may not supply a reliable account. This paper audits two repositories by the same author, measuring trace legibility and disclosure trailers. Their records disagree, and neither repository channel establishes an AI-involvement rate.
Problem
AI-use disclosures require authors to describe past contributions, but it is unclear what evidence an ordinary computational repository can supply.
Source: §1 · p. 1Method
Results
85.6% of Project A’s committed trace lines remain visible, versus 21.8% for Project B; repository disclosure rates cannot be read as involvement rates.
Source: §1 · p. 2Takeaways & Limitations
A repository can constrain a provenance claim but cannot establish it; the paper derives requirements for capturing provenance as it happens.
Source: §11 · p. 10The two repositories differ in tool, trace format, subject, duration, and year, so the study cannot attribute contrasts to trace format or claim Section 8 is complete.
Source: §9 · p. 9
Abstract
from arXiv · showhide
Publishers and funders increasingly ask authors to disclose how AI tools contributed to a piece of research. In computational work the repository is where an author or an auditor would look for the answer, and we ask what it can actually supply. We audit two single-investigator repositories by the same author in the same field, each carrying a different class of agent residue: a prescriptive trace of standing rule files and a narrative trace of a running agent scratchpad, in repositories of 466 and 103 commits. The two are legible in opposite ways. 85.6% of the lines ever committed to the prescriptive trace are visible at the current revision, and a standing rule states a conclusion and discards the episode behind it; the narrative trace records episodes and shows 21.8%. Counted by revision the gap is far smaller: 15 of the prescriptive trace's 41 revisions leave no visible line, against 18 of the narrative trace's 41. The Co-Authored-By trailer, the field an automated audit reads, is one of three AI trailer keys Project A carries and covers 66 of the 121 commits carrying any. In Project B it has low sensitivity against an explicit within-repository proxy: 39 commits revise a file maintained to hold an agent's working notes and 1 (2.6%) carries the trailer. Project A's repository-level rate is unstable in time too, stepping from 2.7% to 17.7% across a boundary we did not specify in advance. Repository channels disagree, and neither rate can be read as an involvement rate; we do not isolate the cause, and say so. We give a field-by-field account of which provenance claims such traces support, which they cannot, and which they answer confidently and wrongly, construct an auditable evidence ledger, and derive the requirements this places on any system meant to record research provenance as it happens.
1 Introduction
The paper asks whether Git repositories can support AI-use disclosures and audits two repositories by the same author to assess what their traces and commit logs reveal. The traces offer complementary but incomplete records, and repository disclosure rates cannot be read as AI-involvement rates.
Design
- The audit compares two repositories by one environmental-sciences investigator: Project A has standing rule files, while Project B has a running agent scratchpad.Tool, artifact class, project, and time are confounded, so differences cannot be attributed to the tool alone.Source: §1 · p. 2
Findings
- 85.6% of Project A’s committed trace lines remain visible, but its standing rules preserve conclusions rather than the episodes behind them; Project B records episodes but shows only 21.8%.Project B’s remaining history is recoverable by walking the file’s history, not by reading its current contents.Source: §1 · p. 2
- The Co-Authored-By trailer misses 38 of 39 Project B scratchpad-revision commits, while Project A’s rate rises from 2.7% to 17.7% across a boundary the authors did not specify in advance.The authors cannot isolate the cause; repository evidence does not convert a disclosure rate into an involvement rate.Source: §1 · p. 2
Contributions
- The paper contributes a field-by-field taxonomy, trace-legibility and commit-log measurements, and requirements for systems that capture provenance in situ.Its analysis also produces an auditable ledger behind each reported number.Source: §1 · p. 2
2 Related work
Prior work examines provenance models, repository mining, AI disclosure and verification, and structured disclosure. This paper focuses on what a repository’s ordinary record can establish about AI involvement.
Provenance and the integrity of the record
- Formal provenance models identify entities, activities, and responsible agents; reproducibility research finds that such records must be deliberately produced, while supply-chain attestation presupposes the relevant events were identified.This paper measures which provenance fields survive in an AI-assisted project’s Git repository.Source: §2 · p. 3
Mining repositories
- Repository-mining studies warn that curated histories and metadata can misrepresent development, authorship, and effort; this paper treats those cautions as the object of study.Earlier work reads repositories for reusable agent practice, whereas this paper asks what repositories can attest.Source: §2 · p. 3
AI disclosure and its verification
- AI-disclosure research measures declarations and textual signals, while verification work finds limits to post-hoc detection; this paper asks what repository declarations can establish.The related work also notes that agent evaluation can require expert rubrics and that research is shared between author and agent.Source: §2 · p. 3
Structured disclosure
- Datasheets and model cards use fixed schemas with named fields, including “not recorded”; Section 8 applies that disclosure approach to process provenance.Source: §2 · p. 3
3 Method
The method compares AI-trace artifacts and measures what their Git histories and current contents reveal, alongside the repository’s disclosure channel. It fixes the census point and documents query choices that affect what an audit sees.
3.1 Unit of analysis
- The AI-trace artifact—the file or directory maintained to carry agent state between sessions—is the unit of analysis, making the repositories comparable despite different research content.Project B is evidenced as Cursor-assisted by a versioned scratchpad across 39 commits; the repository does not support a claim about what proportion of its code was model-written.Source: §3.1 · p. 3
- The Project B scratchpad is a project convention, not a file the tool creates and maintains automatically, and its revisions are an imperfect proxy for contemporaneous agent activity.A human may edit it directly, while agent work may be committed later, separately, or alongside unrelated changes.Source: §3.1 · pp. 3–4
- The cases share one author, discipline, and version-control provider, but differ in other respects; Table 1 describes the cases rather than serving as a design matrix.Source: Table 1 · p. 4
3.2 Measurement
- Across three defensible history queries, history-only shares vary by 4.2 points in Project A and 2.1 in Project B, while Project A’s revision count varies 5.9-fold and its apparent start date by 228 days.The paper uses the path-union query because that is what an audit would run.Source: §3.2 · p. 4, Table 2 · p. 4
- Disclosure is counted from commit-message trailer keys rather than AI-related words in message text, which can match vocabulary without indicating a structured claim.The headline rate uses Co-Authored-By; Project A’s trace directory was renamed, so the chosen history query also affects the apparent record.Source: §3.2 · pp. 4–5
- Figure 1 compares each trace artifact’s history-only share with other tracked text files in its own repository; both distributions have a median of 0%.Files are included only if at least 20 lines were ever inserted, limiting small-denominator instability.Source: Figure 1 · p. 5
4 What a repository can and cannot recover
The paper evaluates what a normally kept Git repository supplies for provenance fields, distinguishing trustworthy recovery from partial, absent, and misleading evidence. It highlights fields that can return confident but wrong answers.
4 What a repository can and cannot recover
- The field-by-field verdicts distinguish recoverable, partial, absent, and misleading information; misleading fields return a confident answer that does not mean what it appears to.These verdicts describe default practice, not the limits of what Git could be configured to record.Source: §4 · p. 5
- Git records the state after a human decision, so it cannot distinguish an accepted model suggestion from a hand-written line; the repository can also check artifact-asserted dates against commit dates.The check found one disagreement: a 2025-01-XX header on a line committed on 2026-01-20, an unfilled template placeholder.Source: §4 · p. 5
- Figure 2 compares lines written in each revision with unchanged lines still visible in the census revision, grouping by blame cohort: both panels contain 41 cohorts, rather than path-query counts of 15 and 39.Source: Figure 2 · p. 6
5 Legibility: what the current revision shows
The two traces retain different amounts of committed text, but their surviving lines and revisions are unevenly distributed. Even visible rules preserve conclusions rather than the episodes behind them, and the measurements depend on artifacts being tracked.
5 Legibility: what the current revision shows
- 1,915 of 2,236 prescriptive-trace lines remain visible (14.4% removed), versus 335 of 1,537 narrative-trace lines (78.2% history-only).The measure compares gross committed insertions with lines at the census revision; rewritten lines count each time they are inserted.Source: §5 · p. 6
- The rule directory ranks at the 87th percentile among 174 tracked text files, while the scratchpad ranks at the 99th among 88.The scratchpad is the repository’s second most overwritten text file; each artifact is unusual relative to the files alongside it.Source: §5 · p. 6
- 15 of Project A’s 41 revisions and 18 of Project B’s 41 leave no surviving line; each trace is also top-heavy toward recent revisions.The newest three Project A revisions supply half its visible lines; the newest seven Project B revisions supply half.Source: §5 · p. 6
- Rename-following blame attributes 686 visible lines (35.8%) to 26 commits omitted by a path-limited log, reaching back to 2025-09-01.The path query dates the lines to their move into the directory, making the artifact appear seven months younger.Source: §5 · p. 6
- Visible rules state conclusions drawn from episodes but discard the episodes, so the artifact gives no account of why a rule exists.The authors report that the artifact supplied a rule’s text and commit date, but nothing about the failure that motivated it; they make no quantitative claim about information content.Source: §5 · pp. 6–7
- Both measurements depend on artifacts being tracked; had either artifact been git-ignored before its first commit, every reported number would be zero.Neither artifact was git-ignored, a choice the authors verified; excluding an agent’s working directory requires only a one-line edit.Source: §5 · p. 7
6 Disclosure: what the commit log says
The commit log records disclosures, not AI involvement: its signal varies by trailer key, commit type, and time, and the repositories do not establish the cause of the temporal change.
6 Disclosure: what the commit log says
- 121 of Project A’s 466 commits (26.0%) carry one of three AI trailer keys, while Co-Authored-By alone misses 55 (45.5%) of those disclosures.No commit carries both Made-with and Co-Authored-By; Claude-Session appears only beside Co-Authored-By.Source: §6 · p. 7
- Against the working-notes proxy, Co-Authored-By appears on 1 of Project B’s 39 revisions (2.6%) and 5 of Project A’s 15 (33.3%).The trailer’s sensitivity against this proxy is neither close to the repository-level rate nor consistent across the two cases.Source: §6 · p. 7
- Project A’s Co-Authored-By rate rises from 3 of 111 commits (2.7%) through 2025-12 to 63 of 355 (17.7%) afterwards, but the boundary was selected after viewing the monthly series.The repository is silent through 2026-01, so this is a resumption, not a clean adjacent-month change; the cause remains unresolved because tool settings and commit authorship processes are not recorded.Source: §6 · pp. 7–8
- Project A’s 66 AI trailers contain six identity strings, including three that answer nothing about the model and two that name the same model.The strings identify software that was open, not how much the author relied on it.Source: §6 · p. 8, Figure 4 · p. 8
- 14.2% of Project A’s 466 commits and 1.0% of Project B’s 103 carry Co-Authored-By, but neither rate measures AI involvement.The repositories contain no evidence that converts the disclosure rate into an involvement rate.Source: §6 · pp. 7–8
7 Reconstruction from a prescriptive trace
The audit codes prescriptive rules separately for origin, human review, and evidential strength, but repository evidence cannot establish who originated them or whether they prevented recurrence.
7 Reconstruction from a prescriptive trace
- Every rule has unknown origin and unobserved review state; evidence reaches tool metadata at best, and inference only for why a rule exists.The coding measures what evidence exists, not what caused a rule to change.Source: §7 · p. 8
- The absence of a corrective commit cannot show that a rule prevented recurrence, and the trace supports standards-at-a-date claims rather than claims about who originated the work.The issue could have been fixed silently, not occurred, or gone undetected; authorship remains self-asserted or stated in the commit message.Source: §7 · p. 8
8 Requirements for in-situ capture
The paper derives tool-agnostic capture requirements from measured gaps, emphasizing recording provenance as work happens and making its history accessible.
8 Requirements for in-situ capture
- Capture the accept event in the editor or agent, because repository state records only what happened after a person accepted or modified model output.Whether a human modified the output distinguishes assistance from authorship.Source: §8 · p. 9
- Separate and standardize involvement and tool-identity fields so audits do not conflate contribution with the tool named in a trailer.The involvement field also needs a defined value for “not recorded,” so silence is not read as absence.Source: §8 · p. 9
- Present narrative records as append-only entries or a series, rather than storing them only as a mutable document’s latest state.Git holds the 78.2% absent from the current revision, but readers are not prompted to inspect its history.Source: §8 · p. 9
- Make the tracking decision explicit when an artifact is created, including that excluding it from Git discards the trace.The measured legibility figures depend on whether the artifact was git-ignored.Source: §8 · p. 9
- Bind provenance queries to history rather than paths, following renames and dating content by when it was written rather than when it moved.A path query without rename following returned less than half of one artifact’s revisions without indicating that it was partial.Source: §8 · p. 9
- Check self-asserted artifact dates against repository dates, since the record can expose a disagreement mechanically.Neither tool performed this check.Source: §8 · p. 9
9 Threats to validity
The two-repository design supports an existence claim, not attribution to trace format or completeness. Query choice, line-based measurement, and outcome-aware single-coder judgments further constrain the findings.
Two repositories, one author, and confounded conditions
- The projects differ in tool, trace format, subject, duration, and year, so the comparison cannot attribute outcomes to trace format or establish completeness.The design supports only the existence of repositories that hold a large record but present a small one, and disclosure fields that vary sixfold within one author's history.Source: §9 · p. 9
Single coder, outcome-aware
- A single outcome-aware coder judged evidence availability, not causation, and the authors report no inter-rater agreement.The quantitative results are mechanical and regenerable; the authors argue that the judgments unknown and not observed were least tempting for a motivated coder to assign.Source: §9 · p. 9
The answer depends on the query
- Across three defensible query forms, the history-only share shifts by at most 4.2 points, but revision counts and the apparent start date vary.The headline figures use the path query; blame follows renames and counts 41 revisions and 1,703 lines for Project B, versus 39 revisions and 1,537 lines under the path query.Source: §9 · p. 9
Measurement scope
- The history-only share is line-based, weights rewritten paragraphs and deleted headings equally, and covers only the trace artifact.AI contributions elsewhere are outside the artifact and unidentified in the log, so their size is not claimed.Source: §9 · p. 9
10 Ethics
The paper excludes identifying strings and repository details from its materials, while noting that aggregate measurements may still make the repositories locatable. It also discloses the authors' own AI use in Project A.
Ethics
- The paper omits identifying strings, addresses, repository names, and paths, and keys ledger rows with salted commit-id hashes.It replaces subject lines with conventional commit types but notes that counts, dates, and tool names may let field insiders locate the repositories.Source: §10 · p. 9
- Text, analysis scripts, and figures were produced with an agentic assistant under author review inside Project A.The authors state that their own commits therefore carry the trailers audited in Section 6.Source: §10 · p. 10
11 Conclusion
A well-kept repository supplies less evidence about AI contributions than policy assumes: trace artifacts can be incomplete or misleading, and disclosure fields can miss contributions. Repository evidence constrains provenance claims but cannot establish them.
Conclusion
- A repository can constrain a provenance claim but cannot establish one, because its trace artifacts may show only part of the process or record conclusions instead.The conclusion also reports that the field an audit reads misses 45.5% of one repository's disclosures.Source: §11 · p. 10