PaperlayerView exact revision on arXiv

Source-linked AI summary

Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase

Douglas Leith

arXiv:2609.29744v1cs.SEcs.AI

TL;DR

Assessing AI coding reliability requires visibility into development sessions and errors that commit histories miss. The paper presents a complete AI-authored project dataset, tracing tools, and taxonomies, then finds caught code-generation errors and factual inaccuracies in interactive responses.

  • Problem

    Large-scale AI-code datasets largely omit full session transcripts and process artefacts, limiting visibility into errors caught before commit and the reliability of agents’ claims.

    Source: §I · p. 1

  • Method

    The paper releases the full history of a Claude-built Python tool and applies provenance-tracing tools and taxonomies to its sessions, code, commits, and responses.

    Source: §I · p. 1

  • Results

    14.3% of code-generation events introduced genuine errors caught before commit, and roughly 1 in 4–5 interactive responses contained an inaccuracy.

    Source: §I · p. 1, B. Results · p. 8

  • Takeaways & Limitations

    The analysis shows that the same AI-authored development process includes errors caught by its test suite and inaccuracies in user-facing responses.

    Source: §VIII · p. 10

    The work is confined to one dataset, albeit one documenting a reasonably sized real AI-written tool.

    Source: §VII · p. 10

Abstract

from arXiv · show

We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance tracing tools, (iii) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (ii) code development is mainly proactive, (iii) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AI's interactive responses contains one or more factual errors.

I. INTRODUCTION

The paper addresses the lack of full-process evidence for assessing AI coding reliability by introducing a complete AI-authored project history, provenance tools, and taxonomies. Applying them reveals proactive development alongside caught code errors and factual inaccuracies in interactive responses.

I. INTRODUCTION

  • Assessing AI-driven development requires visibility into the process, including errors in code and claims and how those errors are caught before shipping [1],.Existing datasets largely lack full transcripts and process artefacts, and the needed tracing and error-recovery tools were unavailable.Source: §I · p. 1
  • The Extractor dataset combines full session transcripts, commit history, and AI-authored process artefacts for a 21,000-line tool built entirely by Claude AI.Its history comprises 210 commits, 25 sessions, and 678 user instructions; no code, test, or commit was human-authored.Source: §I · p. 1
  • Two tools trace code changes to instructions or self-corrections and recover self-corrected code-generation errors absent from commit history.The first combines control-flow-graph block decomposition with git log -L history tracing; the second mines session transcripts and tool-use logs.Source: §I · p. 1
  • Three taxonomies cover instruction intent, commit provenance, and response reliability, extending existing schemes or addressing an identified gap.They adapt IDE-chat intent categories to CLI sessions, extend maintenance categories,, and classify factual claims in multi-turn coding-agent conversations.Source: §I · p. 1
  • 14.3% of code-generation events introduced errors caught before commit, while roughly 1 in 4–5 interactive responses contained an inaccuracy.Responses were most accurate when reporting or verifying facts (94.3%) and least accurate when proposing a design fix (79.4%).Source: §I · p. 1

II. RELATED WORK

  • Existing large-scale datasets provide thin session coverage or commit metadata only, while committed-code studies cannot observe errors caught before commit [6],, –.The paper instead traces code blocks to their commit and trigger, asking how code came to be written rather than comparing it with human code.Source: §II · p. 2
  • Related work studies benchmark-task failures rather than individual tests later fixed [11]–[13], while prior taxonomies classify intent, commit rationale, or conversational claims in different settings, –.The paper examines CLI-agent instructions, commit triggers from agent reasoning and tool use, and factual claims about software artefacts.Source: §II · p. 2

III. EXTRACTOR

Extractor automates recovery of Protocol Buffer schemas from compiled Apple binaries. Its development is challenging because private serialization behavior must be inferred from raw ARM64 disassembly across many message classes.

III. EXTRACTOR

  • Extractor automates recovery of complete .proto descriptors for protobuf message classes from Objective-C Mach-O binaries or dyld shared caches.The schemas are generally not shipped with the Apple binaries whose messages need to be read.Source: §III · pp. 2–3
  • Recovering schemas is difficult because PBCodable behavior must be inferred from compiled code, while stripped serialization internals leave little useful decompiler output.Manual recovery requires tracing raw ARM64 instructions across potentially thousands of message classes in a framework.Source: §III · p. 3
  • The Extractor dataset is publicly available with commits, transcripts, AI-generated notes, taxonomies, scoring materials, categorization results, and regeneration scripts.The tool itself is a Claude Sonnet AI-written reverse-engineering tool, not synthetic or toy code.Source: §III · pp. 2–3

IV. ANALYSIS OF CODE DEVELOPMENT PROCESS

This section examines how users direct a CLI coding agent and adapts an IDE-chat intent taxonomy to that setting. Since instructions can bundle distinct steps and intents, the authors decompose them and add categories for behaviors the existing scheme does not capture.

2) Codebook development

  • The authors extended Tang et al.’s IDE-chat taxonomy to CLI instructions, using iterative abductive coding informed by instructions and their immediate context.They sampled instructions by rough thematic clusters, reviewed LLM classifications, and added categories until no new ones were proposed.Source: A. Types of instruction given · p. 3, 2) Codebook development · p. 3

1) Decomposing instructions

2) Codebook development

  • Four subcategories were added: Design Principle Specification, Version Control, Compliance Correction, and Session/Environment Coordination.The latter two capture cross-session coordination and correction of Claude’s conduct rather than its code.Source: 2) Codebook development · pp. 3–4

3) Codebook refinement

The refined taxonomy extends IDE-chat categories to CLI-agent behavior, and the resulting instruction distributions differ across the two settings. In the Extractor corpus, comprehension, planning, consultation, documentation, and validation stand out, while some IDE-chat findings do not carry over.

3) Codebook refinement

  • Two LLMs reached κ = 0.808 on 512 held-out instructions; human annotations of 55 components yielded κ = 0.710 with Claude and 0.718 with Codex.Source: 3) Codebook refinement · p. 4

4) Results

  • Three CLI-specific subcategories capture cross-session coordination, standing design-principle statements, and correction of the AI’s session conduct rather than its code.Source: 4) Results · p. 4
  • Extractor CLI instructions feature more Project Comprehension, Planning & Consultation, Documentation, and Runtime Inspection than IDE-chat instructions; Symptom Description is less frequent and Log Paste is absent.Source: 4) Results · p. 5, Figure 3 · p. 5
  • Iterative Modification is 6.0× more prevalent than New Implementation in Extractor, versus 4.2× in Tang et al.; Alignment Correction is much rarer.Source: 4) Results · p. 5
  • Failure Reporting is 1.77% in Extractor versus 24.0% in Tang et al.; Validation is 18.73% versus 3.99%.The authors attribute the higher Validation share to error detection through code review and runtime testing rather than human inspection and input of failures.Source: 4) Results · p. 5
  • Project Comprehension is 25.7% in Extractor versus 8.2% in Tang et al.; Documentation is 2.6× and Validation 4.7× Tang et al.’s rates.Source: 4) Results · p. 5

B. Triggers for code development: Commit analysis

Commit triggers are classified using session context and a taxonomy organized around proactive and reactive development. Both raters classify most commits as proactive, with the reactive share lower than cited human-maintenance comparisons.

B. Triggers for code development: Commit analysis

  • The taxonomy classifies triggers from the agent’s reasoning and tool use as they occurred, rather than reconstructing rationale afterward from commit messages or issue threads.Source: B. Triggers for code development:… · p. 5
  • The authors link each commit to its session command and surrounding instructions, responses, and actions, adding project-document context when commit messages cite it.Source: B. Triggers for code development:… · p. 5
  • The taxonomy distinguishes proactive from reactive changes and divides them by method or trigger; 14 of 210 commits merged multiple changes and were not decomposed.Reactive triggers include cross-decoder differences, systematic sweeps, external-validator rejection, and unit or regression-test failure.Source: B. Triggers for code development:… · p. 5, Table III · p. 6
  • Reactive commits account for 10.4–13.6% across raters, below the 17% survey estimate and the 20% median across GitHub projects.Table IV reports the commit and originating-code-block breakdown.Source: Table IV · p. 6, B. Triggers for code development:… · p. 6

C. Triggers for code development: Code block analysis

The section measures development at the code-block level and traces blocks to their originating commits and subsequent changes. Most blocks are proactive, and the tracing method bridges file-carve refactors while disclosing unresolved cases.

1) CFG code block decomposition and tracing

  • The tool partitions function ASTs into CFG basic blocks and traces each block’s commit history with git log -L.Partitioning starts blocks at branch points and exception handlers; nested functions and classes are treated as separate problems.Source: 1) CFG code block decomposition and… · p. 6
  • Unlike SZZ, which traces a known bug-fixing commit to a defect-introducing commit at file/diff granularity, this method traces every block’s origin and churn history.The section describes code-block analysis as a finer-grained reference than the coarse commit-level breakdown.Source: 1) CFG code block decomposition and… · p. 6, C. Triggers for code development: Code… · p. 6
  • When git log -L misses history after a partial-file carve-out, the tracer relocates the block by its text, then by its enclosing function name, and requires the carving commit to touch both files.The function-name fallback reduces the unresolved rate from 11.4% to 0.4%; ambiguous matches remain unresolved rather than guessed.Source: 1) CFG code block decomposition and… · p. 7
  • 89.9% of Codex blocks and 89.3% of Claude blocks originate proactively, with similar proactive/reactive proportions across block sizes.The figure breaks down the Codex split by block size; the Claude breakdown is described as almost identical.Source: 1) CFG code block decomposition and… · p. 7, Figure 7 · p. 7
  • Architecture/integration decisions and proactive investigation of the compiled artifact are more common block-for-block than reactive categories; Reactive-Analysis is the largest reactive category.Proactive-Design, Proactive-Analysis and Proactive-Audit together account for 89.6% of Codex blocks and 89.0% of Claude blocks.Source: 1) CFG code block decomposition and… · p. 7

D. Code churn

Code churn is concentrated in three modules, while most blocks undergo no changes. Because errors fixed before commit are absent from commit history, churn alone does not reveal Claude’s coding accuracy.

D. Code churn

  • The great majority of code blocks undergo no changes; among blocks changed once or twice, 270 of 400 are proactive.Reactive changes tend to be one-off fixes, while blocks changed more than twice are rare.Source: D. Code churn · p. 7
  • 79.2% of churn is concentrated in writeto.py, foreign.py and repeated.py.These modules perform the primary reverse-engineering work, and the observed churn is described as consistent with its exploratory nature.Source: D. Code churn · p. 7

V. RELIABILITY OF CLAUDE CODE GENERATION

  • Commit-based churn gives little insight into coding accuracy because obvious errors are caught by tests before they are committed.The paper therefore turns to session histories to analyse Claude’s coding accuracy.Source: D. Code churn · p. 7, §V · p. 7

A. Session-transcript mining

The paper mines raw session transcripts to detect test failures followed by edits and later passing runs, then checks which episodes reflect genuine errors. In the 105 audited events, 14.3% involved errors corrected before commit, but imperfect transcript retention limits the population analyzed.

A. Session-transcript mining

  • The analysis detects candidate self-corrections by matching a failed pytest run to an edit and a later passing run on an overlapping test target.Session logs preserve tool calls and pytest results that commit histories omit; repeated events from session compaction are deduplicated by event UUID.Source: A. Session-transcript mining · p. 8, B. Results · p. 8
  • Because deliberate mutation testing can produce the same fail-edit-pass pattern, an LLM classifies episodes and a human checks every episode categorized as a defect.The classification distinguishes production-logic defects, test/fixture defects, dependent-test update failures, and deliberate mutation-testing cycles.Source: A. Session-transcript mining · p. 8

B. Results

  • 14.3% of 105 code-generation events introduced genuine errors later corrected before any commit.Eight events involved test errors (7.6% of all events), four involved code errors (3.8%), and three involved failures to update tests after correct code changes (2.9%).Source: B. Results · p. 8, Table VI · p. 8
  • The result covers 105 events with full transcripts in the audited population window; imperfect record keeping meant not all events could be recovered.Session transcripts are deleted by default after 30 days, and some were lost before the authors discovered this.Source: B. Results · p. 8
  • The authors conclude that a good test suite is important for protecting against regressions.The measured errors were caught and corrected before commit.Source: B. Results · p. 8

VI. RELIABILITY OF CLAUDE USER INTERACTIONS

The paper assesses factual reliability in Claude’s user-facing responses by decomposing them into checkable components and evaluating them against software artefacts. It describes the analysis as the first fact-checking study of AI coding-session transcripts.

VI. RELIABILITY OF CLAUDE USER INTERACTIONS

  • The fact-checking pipeline decomposes each response into components, identifies checkable facts, gathers evidence, and evaluates accuracy.The paper presents this as the first fact-checking analysis of AI coding-session transcripts.Source: §VI · p. 8

1) Categorise responses

  • The analysis focuses on 6,818 of 9,967 response components: concluding paragraphs, bolded takeaways and summary tables.The components were drawn from responses to 678 user instructions; the taxonomy was developed iteratively and achieved Cohen’s κ = 0.803 between two LLMs.Source: 1) Categorise responses · p. 9
  • The response taxonomy organizes claims by the verification method required, using software artefacts rather than internet search.Checkable facts and mechanism explanations are assessed against the commit current when the response was made; diagnoses and proposed fixes require later evidence.Source: 1) Categorise responses · p. 9
  • Checkable-fact components dominate both LLMs’ category distributions at approximately 64–68%, followed by design-fix proposals at 16–17%.The figure compares the category distributions assigned by the two independent LLMs.Source: 1) Categorise responses · p. 9, Figure 9 · p. 9

2) Evidence collection and scoring

  • Evidence bundles combine scripted codebase checks with LLM-directed additional evidence until no new data is added.About 60% converged immediately, 35% after one extra round, and 5% after two; two LLMs scored all components with repeated human review of disagreements and rubric refinement.Source: 2) Evidence collection and scoring · p. 9

B. Results

Response accuracy varied by category, with design-fix proposals least accurate and checkable facts most accurate. Human checks found high agreement with the AI verdicts across all four categories.

B. Results

  • Because Checkable-fact dominated response components, the authors used stratified sampling to avoid too few samples of other categories and wide confidence intervals.Source: B. Results · p. 9
  • 94.3% accuracy (Claude) and 94.4% (Codex) were reported for Checkable-fact responses.Table VIII reports evaluation verdicts by response category with Wilson 95% confidence intervals.Source: B. Results · p. 10, Table VIII · p. 10
  • 91.1% accuracy (Claude) and 89.5% (Codex) were reported for Mechanism-explanation responses.Table VIII reports evaluation verdicts by response category with Wilson 95% confidence intervals.Source: B. Results · p. 10, Table VIII · p. 10
  • 89.3% accuracy (Claude) and 90.5% (Codex) were reported for Root-cause-diagnosis responses.Table VIII reports evaluation verdicts by response category with Wilson 95% confidence intervals.Source: B. Results · p. 10, Table VIII · p. 10
  • 79.4% accuracy (Claude) and 79.8% (Codex) for Design-fix-proposal were the lowest category scores.Table VIII reports evaluation verdicts by response category with Wilson 95% confidence intervals.Source: B. Results · p. 10, Table VIII · p. 10
  • 98.2% Checkable-fact, 98.0% Mechanism-explanation, 96.6% Root-cause-diagnosis, and 93.3% Design-fix-proposal verdict accuracy were measured against human checks.The checked sample included 200 deficiency verdicts and 200 stratified clean verdicts; applying disagreements would shift the reported figures only slightly, within the confidence intervals.Source: B. Results · p. 10

VII. LIMITATIONS

The paper’s conclusions draw on one AI-built tool, and much of the analysis relies on LLM categorization and scoring. In that setting, the authors report process and response-reliability findings, with model and human checks on the scoring.

VII. LIMITATIONS

  • The analysis is confined to one dataset, albeit a reasonably sized real AI-written tool.The authors note full-project datasets with transcripts and AI-generated notes are otherwise largely absent from the public record.Source: §VII · p. 10
  • Much of the analysis uses LLMs to categorize and score, with two independent models agreeing with each other and human calibration samples.The reported agreement with human calibration samples was κ = 0.70–0.81.Source: §VII · p. 10

VIII. CONCLUSIONS

  • 14.3% of checked AI code-generation events introduced errors, and the AI-authored test suite caught them before commit.Claude’s responses were reported as 94.3% accurate for facts, 91.1% for mechanism explanations, 89.3% for root-cause diagnoses, and 79.4% for design-fix proposals.Source: §VIII · p. 10
  • The conclusions report that CLI instructions emphasize comprehension, planning, and consultation more than IDE-chat instructions, while development was mainly proactive.Source: §VIII · p. 10

A. Examples of response errors

The examples show that response errors ranged from unsupported numerical claims to incorrect mechanisms, diagnoses, and proposed fixes. Later evidence in the development history exposed or tested these claims.

A. Examples of response errors

  • The claim that four functions ran millions of times was contradicted by recorded counts ranging from 40,780 to 296,681 calls.The later report contained the same counts, and a static call-site count supported the structural explanation that each function runs once per higher-level decision.Source: A. Examples of response errors · p. 12
  • The mechanism claim was backwards: later debugging and the fix’s commit message identified the writeTo CFG walk, not readFrom dispatch.The claim’s predicted nine changed fields were correct, but the stated mechanism was not.Source: A. Examples of response errors · p. 12
  • The filesystem diagnosis was wrong: prior file-creation attempts had silently failed, so the file had never existed anywhere.The AI repeated the same file-not-found error in its own environment and did not notice that this disproved its diagnosis.Source: A. Examples of response errors · p. 12
  • Increasing the scan limit to 1,000, 4,000, or 30,000 instructions recovered zero additional values across 752 enums and 14,900 values.The later fix identified an unrelated jump-table decoding gap as the actual bug.Source: A. Examples of response errors · p. 12
Loading 2609.29744v1…