PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent or Human? Detecting Commits by AI Coding Agents in the Absence of Authorship Signatures

Kosei Horikawa, Hao Li, Yutaro Kashiwa, Youssef Esseddiq Ouatiti, Mohammed Sayagh, Bram Adams, Hajimu Iida, Ahmed E. Hassan

arXiv:2610.36229v1cs.SEcs.AI

TL;DR

The paper asks whether Agent commits can be distinguished from Human commits after optional authorship signatures are removed. It evaluates signature-masked commits from many repositories, finding that detectors can rank Agent commits above Human commits but remain limited in precision and transfer. The authors conclude that detection may prioritize review, but cannot replace recorded attribution.

  • Problem

    Because signatures are optional or removable and PR labels do not identify authorship of individual commits, it remains unclear whether unsigned Agent commits can be distinguished from Human commits.

    Source: §1 · p. 2, §4.2.1 · p. 9

  • Method

    The study evaluates masked-commit detectors on repository-disjoint data and examines their signals, transfer to unseen agents, and change over time.

    Source: §4.2.1 · p. 9, §4.2.2 · p. 10

  • Results

    Detectors reach an ROC AUC of up to 0.859, but precision is at most 0.332 at the observed Agent share, with most signal coming from message verbosity and change size.

    Source: §7 · pp. 18–19

  • Takeaways & Limitations

    Signature-free detection can prioritize commits for review but cannot replace recorded attribution, and signature-based Agent labels should be treated as a lower bound on agent activity.

    Source: §7 · p. 19

    Labels do not establish exclusive authorship: undisclosed AI use and squashed Human–Agent contributions may cross the Human and Agent boundaries, with unknown bias direction.

    Source: §6.1 · p. 18

Abstract

from arXiv · show

AI coding agents such as Claude Code, GitHub Copilot coding agent, and Cursor now commit directly to open-source repositories. Empirical studies of these agents identify their commits through authorship signatures, such as bot accounts and Co-authored-by trailers. These signatures are optional, can be removed, and are often attached to pull requests rather than to individual commits. Hence, it is unclear whether Agent commits can be told apart from Human commits once the signatures are gone. We investigate the signals and limits of signature-free detection using 7,837,946 commits from 1,226 repositories, covering five coding agents and Human commits. After signatures are masked, detectors rank Agent commits above Human commits with a ROC AUC of up to 0.859, and most of this signal comes from message verbosity and change size rather than from the code itself. On the shared sample, gpt-6-sol without fine-tuning reaches AUCs similar to XGBoost and TF-IDF. At the observed Agent share, however, the best detector on the full test set reaches a precision of only 0.332. Detection transfers poorly to held-out Claude Code and Copilot, with TF-IDF reaching an AUC of only 0.524 for Claude Code. Temporal changes also depend on the agent, e.g., Copilot AUC drops and later recovers, while more training history helps Claude Code but not Copilot. These results indicate that signature-free detection can prioritize commits for review but cannot replace recorded attribution, and that studies relying on signature-based labels should treat them as a lower bound on agent activity.

1 Introduction

The introduction asks whether individual Agent commits remain distinguishable from Human commits after signatures are removed. It motivates content-based detection and frames the study’s questions around detection performance, generalization, and practical use.

  • Prior studies use signatures such as bot accounts and attribution trailers, but these are optional, removable, and may attach to pull requests rather than individual commits.Human developers may also revise agent-authored code after merging.Source: §1 · p. 2
  • The study examines 7,837,946 default-branch commits from 1,226 repositories, labeling commits from five agents and comparing them with Human commits across three eras.The authors mask signatures and test detectors on repositories absent from training.Source: §1 · p. 2
  • After authorship signatures are removed, detectors tested on unseen repositories at the observed Agent share of 0.059 rank Agent commits above Human commits (ROC AUC up to 0.859), with validation-selected precision at most 0.332.Signal comes mostly from commit message and change size, not code; transfer is weakest for Claude Code and Copilot, with XGBoost AUCs of 0.702 and 0.706, respectively.Source: §1 · p. 2
  • The authors also report that Copilot AUC falls from 0.83 to 0.61 within ten months as fewer commits carry its squash-merge template.More training history helps Claude Code but not Copilot, while TF-IDF signal fades unless the detector is retrained.Source: §1 · p. 3
  • Detection scores can prioritize commits for review, but only disclosure identifies which commits an agent wrote.The authors say signature-based labels should be treated as a lower bound on agent activity.Source: §1 · p. 3

2 Background

The background describes coding agents and the attribution challenges created by optional signatures, mixed human–agent contributions, and commit-level ambiguity. It positions this study as a commit-level detector evaluated under more realistic prevalence and repository-disjoint conditions.

2.1 AI Coding Agents in Software Development

  • Agent-assisted commits are larger than commits labeled Human, and task and change size shape how their contributions compare.The study therefore compares commits within the same repositories and era and tests detection among commits of similar size.Source: §2.1 · p. 3

2.2 Challenges in Attributing Agent Contributions

  • A human account or missing agent identifier does not establish that a change was written without AI assistance.In AIDev, 5.0% of 6,618 PRs labeled human-authored contained explicit agent identifiers; human revision also means a PR label cannot establish who authored each commit.Source: §2.2 · p. 4

2.3 Detecting Agent-Authored Commits

  • The study classifies individual commits rather than PRs, evaluates at the observed Agent share, and uses repository-disjoint validation with chronological testing.It also tests whether masking leaves merge-workflow artifacts that identify agents.Source: §2.3 · p. 4

3 Study Design

The study design selects repositories and default-branch commits across three eras, labels recognized agent signatures, and evaluates detectors using masked commit features. Its protocol uses repository-disjoint splits and reports measures suited to imbalanced classes.

3 Study Design

  • The shared study design covers repository and commit selection, labeling, features, and evaluation across the research questions.The overview is summarized in Fig. 2.Source: §3 · p. 5

3.1 Repository Selection

  • The repository selection starts from AIDev v5 and retains repositories created before 2020-03-29 with at least 400 default-branch commits in each 15-month era.The selection leaves 1,227 repositories for collection; the three eras span pre-LLM, LLM, and Agent periods.Source: §3.1 · p. 5

3.2 Commit Selection

  • The final dataset contains 7,837,946 commits from 1,226 repositories, with agent labels based on recognized accounts or a Claude Code co-author signature.Human commits have no recognized AI-assistance signature after bot filtering; the label does not establish that no AI assistance occurred.Source: §3.2 · p. 5

3.3 Features and Leakage Control

  • The detector uses 39 features, while Code Style and Code Vocabulary are computed only when patches cover every changed file and line and match recorded line counts.Before feature extraction, attribution lines, generation notices, tool URLs, and vendor templates are removed, and remaining tool names are replaced with a placeholder.Source: §3.3 · p. 6

3.4 Evaluation Protocol

  • Training and test repositories are disjoint, and learned-detector thresholds are selected on validation data before testing.The protocol reports ROC AUC, PR-AUC, precision, recall, F1, lift, MCC, and weighted F1; repository-level bootstrap samples provide 95% confidence intervals.Source: §3.4 · p. 7

4 Results

For each research question, the paper presents its motivation, approach, and results.

  • Each research question is presented through its motivation, approach, and results.Source: §4 · p. 7

4.1 RQ1: How do agent-authored commits differ from human commits?

Because commit signatures are optional and content signals may shift across repositories and eras, this section compares Agent commits with Human commits from the same repositories and era, then evaluates signature-free detection on unseen repositories. It examines which commit characteristics differ and whether those differences support reliable classification.

4.1.1 Motivation.

  • Message length, code changes, and tool conventions may distinguish Agent from Human commits, but human practices also shift across eras.The section compares commits within the same repositories and era to separate these differences.Source: §4.1.1 · p. 7

4.1.2 Approach.

  • RQ1 compares repository-level medians and within-repository effects, retaining repositories with minimum counts in three groups and weighting repositories equally.The display includes features with |δ| ≥0.100 for at least one of Claude Code, Copilot, and Cursor; this cutoff is not a significance criterion.Source: §4.1.2 · p. 8, Table 4 · p. 8

4.1.3 Results.

  • Claude Code’s typical message is 73 words versus 10 for Agent-era Human commits; code-change effects are smaller and vary by agent.Claude Code and Copilot have more distinct added tokens and changed lines per file than Human commits, while Cursor intervals span zero; word-count effects are medium to large.Source: §4.1.3 · pp. 8–9, Table 4 · p. 8
  • Across 1,224 repositories, Human messages lengthened and changes touched more lines per file; Agent-era Human commits are the appropriate comparison.Human message features rose from the pre-LLM to LLM era, while code-style and code-diff measures shifted mainly from the LLM to Agent era.Source: §4.1.3 · p. 9, Table 5 · p. 9

4.2.1 Motivation.

  • Commit content can support detection when signatures are absent, but group differences alone do not establish that individual commits can be classified in unseen repositories.The analysis asks which evidence may persist when message conventions change.Source: §4.2.1 · p. 9

4.2.2 Approach.

  • The main binary detector uses masked, correlation-reduced features; analyses compare feature-family ablations, size-matched commits, and classifier and baseline variants.The size-matched comparison pairs commits by repository, language, month, and size range; it does not match exact sizes.Source: §4.2.2 · p. 10

4.2.3 Results.

  • After masking signatures, TF-IDF on masked messages and diffs reaches ROC AUC 0.859; on the full test set, best precision is 0.332 at observed Agent share 0.059.After signatures are masked, detectability comes mostly from message verbosity and change size; among similar-size commits, signal drops substantially, especially for Claude Code.Source: §4.2.3 · pp. 10, 12

4.3 RQ3: How well does detection generalize to agents unseen during training?

The section tests whether a detector trained on known agents can rank commits from an agent absent during training. It finds that rankings remain above chance, but performance and thresholds transfer unevenly across agents.

4.3.1 Motivation.

  • High performance on training agents may reflect their specific habits, risking missed or skewed representation of agents the detector has not seen.The section tests whether the detector ranks commits from an unseen agent above Human commits.Source: §4.3.1 · p. 12

4.3.2 Approach.

  • The evaluation holds out each agent and excludes all commits from its repositories from training, then tests on that agent and Human commits from those repositories.Hyperparameters and thresholds are selected using known agents only.Source: §4.3.2 · pp. 12–13, Table 8 · p. 13

4.3.3 Results.

  • AUCs of 0.893 or higher for Cursor, Devin, and Google Jules rest on few test commits and repositories and may partly reflect Claude Code verbosity in training data.Fixing training counts leaves AUCs almost unchanged, so the reported gap is not explained by training volume.Source: §4.3.3 · p. 13, §4.3.2 · p. 13
  • TF-IDF transfers weakly to unseen Claude Code and Copilot, with AUCs of 0.524 and 0.630, below XGBoost on the same test commits.For these agents, the 32-feature detector, relying mostly on message length and change size, retains more ranking signal.Source: §4.3.3 · p. 13
  • Held-out-agent detection remains above chance, but Claude Code and Copilot XGBoost AUCs are 0.702 and 0.706, and the known-agent threshold recalls only 0.005 of unseen Claude Code commits.The confidence intervals lie above 0.500 for all held-out agents; XGBoost’s threshold-tuned precision is below 0.070 for every held-out agent.Source: §4.3.3 · pp. 13–14, Figure 4 · p. 14, Table 9 · p. 14

4.4 RQ4: How does detection performance change over time?

The study tests how detection changes when models trained on earlier months are applied to later months, and whether longer training histories slow that change. Performance varies by agent and period: Copilot’s AUC falls and recovers without retraining, while more history helps Claude Code but yields no clear AUC gain for Copilot.

4.4.1 Motivation.

  • The analysis measures whether detector performance drifts over time as agents change behavior or message templates and repositories change workflows.It also asks whether longer training histories slow that change.Source: §4.4.1 · p. 15

4.4.2 Approach.

  • Models use expanding monthly training windows, are frozen for later-month tests, and are evaluated on held-out repositories using the same commits across training cutoffs.The threshold and hyperparameters are selected once and then held fixed across later months.Source: §4.4.2 · p. 15

4.4.3 Results.

  • Claude Code AUC rises from 0.705 to 0.752 when training extends through July 2026 rather than June 2025, but Copilot shows no clear AUC gain.On the same August 2026 commits, Claude Code recall more than doubles at similar precision; Copilot precision falls.Source: §4.4.3 · p. 15, Table 10 · p. 16
  • Copilot AUC falls from 0.83 in July 2025 to 0.61 in May 2026, then recovers to 0.78 in August without retraining; Claude Code stays between 0.67 and 0.78.The Copilot dip recurs across training cutoffs and coincides with fewer workflow-template markers in messages.Source: §4.4.3 · p. 15, Figure 5 · p. 16
  • For Claude Code, TF-IDF AUC falls from 0.733 to 0.631 without updated training, but reaches 0.801 when trained through July 2026.With the expanded history, TF-IDF exceeds XGBoost with a paired interval that excludes zero; Copilot TF-IDF remains below XGBoost, without a clear difference.Source: §4.4.3 · p. 16
  • With June 2025 thresholds fixed, Claude Code recall never exceeds 0.091, while precision rises from 0.028 to 0.320 as its test share grows from 0.003 to 0.108.The precision increase alone does not show better ranking; later evaluations need precision and recall alongside AUC.Source: §4.4.3 · pp. 16–17

5 Implications

The implications distinguish what detection can support from what it cannot: it can prioritize review, but it cannot establish authorship or replace recorded attribution. The paper recommends prevalence-aware evaluation and durable attribution, while noting that message style can identify an agent and later change.

5.1 Implications for Researchers

  • Code representations separate agent and human changes only weakly, while message verbosity and change size carry most detection signal; researchers should test unseen agents and later months.At the observed Agent share, precision reaches at most 0.332 despite class-weighted F1 up to 0.919 on the full test set.Source: §5.1 · p. 17

5.2 Implications for Developers

  • Detectors can prioritize review but cannot establish authorship: the highest-scoring 1% yields precision 0.316 at recall 0.053.The paper recommends requiring disclosure, such as a trailer checked in continuous integration, when commit-level authorship matters.Source: §5.2 · p. 17

5.3 Implications for Coding Agent Builders

  • Attribution should survive edits and squash-merges, for example as signed commit metadata rather than an optional message trailer.Users can change or disable agent signatures, and content-based detectors do not close the resulting gap.Source: §5.3 · pp. 17–18
  • Claude Code’s median commit message has 73 words versus 10 for Agent-era Human commits, making message style a visible, changeable fingerprint.Changes to default message style can make earlier detectors obsolete.Source: §5.3 · p. 18

6 Threats to Validity

The paper identifies construct, internal, and external validity boundaries. Labels do not establish exclusive authorship, comparisons are observational, and the data and temporal analyses cover a limited set of repositories and agents.

6.1 Construct Validity

  • Agent and Human labels do not establish exclusive authorship, and features may reflect tasks and repository conventions as well as the labeled groups.Undisclosed AI use and squashed Human–Agent contributions can cross label boundaries, with unknown performance-bias direction.Source: §6.1 · p. 18

6.2 Internal Validity

  • Observational comparisons cannot isolate causal effects because eras, test populations, training histories, preprocessing, and thresholds differ.Feature ablations and SHAP values describe predictions, not causal effects; bootstrap intervals also hold fitted models fixed.Source: §6.2 · p. 18

6.3 External Validity

  • The AIDev sample favors mature, active public repositories and does not represent private or recently created projects.Only Claude Code and Copilot had sufficient commits for every temporal-analysis month, so transfer conclusions are limited to evaluated settings.Source: §6.3 · p. 18

7 Conclusion

The paper evaluates commit-level detection without authorship signatures across 7,837,946 commits from 1,226 repositories. Detection finds some signal, but its limits in precision, transfer, and stability mean it can prioritize review rather than replace recorded attribution.

7 Conclusion

  • The study examines signature-free, commit-level detection using 7,837,946 commits from 1,226 repositories.The analysis asks whether Agent commits can be distinguished from Human commits after signatures are removed.Source: §7 · p. 18
  • After signatures are masked, detectors reach AUC up to 0.859, with most signal from message verbosity and change size; the best detector’s precision is 0.332 at the observed Agent share.Without fine-tuning, gpt-6-sol reaches AUCs similar to XGBoost and TF-IDF on the shared sample.Source: §7 · pp. 18–19
  • Signature-free detection can prioritize commits for review but cannot replace recorded attribution; signature-based labels should be treated as a lower bound on agent activity.The paper also recommends reporting precision at observed prevalence and masking merge-workflow artifacts.Source: §7 · p. 19

Data Availability

  • The replication package includes identifiers, labels, features, predictions, prompts, and responses for all 7,837,946 commits, with a script to regenerate tables and figures.The package contains data with and without masking.Source: Data Availability · p. 19
Loading 2610.36229v1…