Source-linked AI summary
Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activity
Romain Robbes, Théo Matricon, Thomas Degueule, Andre Hora, Stefano Zacchiroli
TL;DR
Research on coding agents needs scalable evidence because they are rapidly adopted and differ from code-completion tools in their autonomy and repository visibility. This paper studies GitHub activity through detection heuristics and distills the promises, perils, and mitigations of mining those traces. It finds that adoption was already visible by October 2025, with 15–19% of GitHub projects showing agent traces, while detection is constrained by changing agents, heterogeneous heuristics, and partial observability.
Problem
Large-scale evidence on how coding agents are used in practice is limited, although their rapid adoption and repository traces make MSR study important.
Method
The paper develops and applies GitHub heuristics for detecting coding-agent activity, then documents associated promises, perils, mitigations, and a shared heuristics repository.
Results
15–19% of GitHub projects showed coding-agent traces by mid-October 2025, and adoption was growing steeply.
Takeaways & Limitations
Repository traces enable MSR studies of coding-agent activity, but researchers need evolving heuristics and shared data because agents and their markers change.
Takeaways & Limitations
Detection is limited by changing agent behaviors and heuristics, partial observability of prompts, and workflow differences across agents.
Abstract
from arXiv · showhide
In 2025, coding agents have seen a very rapid adoption. Coding agents leverage Large Language Models (LLMs) in ways that are markedly different from LLM-based code completion, making their study critical. Moreover, unlike LLM-based completion, coding agents leave visible traces in software repositories, enabling the use of MSR techniques to study their impact on SE practices. This paper documents the promises, perils, and heuristics that we have gathered from studying coding agent activity on GitHub.
1 Introduction
Coding agents differ from code-completion tools through broader autonomy and increasingly visible repository traces, creating an opportunity for large-scale MSR studies. The paper develops heuristics and discusses promises, perils, and a community repository for studying this rapidly changing activity.
- Research gap: Earlier LLM studies largely used controlled experiments, observation studies, or surveys rather than large-scale MSR techniques.This limits the generalizability of their findings.
- Coding agents: Coding agents autonomously perform sequences of repository actions to accomplish developer-delegated tasks and produce reviewable code changes.
- Research opportunity: Coding agents leave substantially more repository traces than code-completion LLMs, making MSR-based study possible.
- Perils: Observed agent data is partial, heterogeneous across agents, and rapidly changing.
- Contributions: The paper contributes heuristics, an analysis of promises and perils with mitigations, and a community-maintained repository of heuristics and sample trace datasets.The repository is intended to help researchers investigate traces and plan studies before analyzing large real-world datasets.
2 Related work
Prior research on LLM-assisted software engineering used experiments, qualitative studies, telemetry, repository mining, and curated datasets, but coding-agent evidence remains sparse and mixed. This motivates MSR studies of agents at repository scale.
- Prior evidence: Research on code assistants found mixed effects on productivity, code quality, task success, and security-related outcomes.Reported studies include faster HTTP-server completion, lower code quality, and more task failures under different comparisons.
- Prior evidence: Qualitative studies identified acceleration and exploration usage modes and found developers spent substantial time evaluating suggestions.One study reported 22% of time evaluating suggestions versus 14% writing code.
- Study approaches: Telemetry studies measured acceptance and generated-code rates in controlled deployments, while repository studies searched for explicit ChatGPT or Copilot mentions.
- Repository evidence: Repository analyses reported contrasting churn findings, while the absence of specific generative-AI markers complicated interpretation.
- Coding-agent gap: Coding-agent studies were still few and had mixed results, leaving MSR as an opportunity to complement the emerging evidence.
- Coding-agent studies: Existing coding-agent studies included a Cursor experiment, an observation study of 19 developers, and 120 benchmark-agent interaction logs.
3 Coding agents
The paper defines coding agents as tool-enabled LLM loops that autonomously interact with their environments, distinguishing them from assistants through dynamic tool use. Their autonomy, oversight, and task scope vary across workflows.
- Definition: A coding agent is an LLM executing in a loop to fulfill a goal while using tools to access its environment.
- Agent loop: The agent loop queries the LLM, parses tool calls, optionally requests permission, executes tools, returns results, and repeats until task completion.
- Agent components: Modern agents combine an LLM with a harness that supplies available tools and detects structured tool calls.Structured outputs support reliable tool calling, while reasoning models support longer tasks.
- Tool use: Tool use enables agents to explore repositories dynamically, edit files, run compilers and tests, and commit changes or author pull requests.
- Tool use: Agents range from shell-based tools to integrations with browsers, databases, ticket repositories, and other complex systems.
- Uses: Coding agents support tasks from dependency updates and bug fixes to documentation, feature implementation, refactoring, and language-porting.
- Autonomy and oversight: Autonomy depends on task sensitivity, developer oversight, environment setup, and workflow, with permissions ranging from unrestricted use to approval or prohibition.
4 Traces and heuristics
The paper develops GitHub heuristics for detecting coding-agent activity across files, commits, issues, and pull requests. These traces reveal agent presence and workflow, but detection is complicated by false positives, heterogeneous conventions, and incomplete prompts.
- Heuristic construction: The authors derive tentative GitHub heuristics through manual investigation of agent documentation and repositories, then validate them with targeted searches.
- Heuristic construction: The heuristics cover multiple GitHub artifact categories and are distilled into broader detection strategies because specific markers may change.
- Detection overview: Table 1 lists agent heuristics and approximate GitHub match counts, including agents with more than 1000 matches and heuristics with more than 250 matches.
- Presence of traces: Coding agents leave visible traces in files, commits, issues, and pull requests that can be detected with specific heuristics or generic strategies.
- Files: Configuration files indicate agent setup and can expose permissions, available tools, and autonomy settings.
- Files: Rules and guidance files provide natural-language instructions or broader project knowledge, and may be generated by agents.
- Files: File naming conventions simplify detection but can produce false positives, such as CONVENTIONS.md files used for ordinary developer documentation.
- Commits: Commits can reveal agent activity through co-author trailers, agent authorship, or generated-by trailers, although workflows may include developer oversight.
5 Promises, perils and mitigations
Coding-agent traces make adoption and agent-based automation observable at scale, creating new opportunities to study human–AI collaboration and software-engineering impact. However, detection is constrained by heterogeneous, incomplete, and rapidly changing traces, requiring continually updated heuristics.
- Promises: 15 to 19% of GitHub projects showed coding-agent traces by mid-October 2025, with adoption growing steeply.The study found use ranging from a handful of co-authored commits to repositories where most commits were co-authored by agents.
- Promises: Coding-agent traces make agent-based automation visible at scale, enabling MSR studies of adoption and software-engineering impact.Unlike earlier studies relying on deliberate developer annotations, repository traces can support more exhaustive detection of LLM-generated code.
- Promises: Public traces across pull requests, issues, code reviews, commits, and files reveal developer delegation, review, correction, and workflow adaptation around agents.These records expose both agent outputs and developers’ responses, including acceptance, rejection, refinement, and guidance practices.
- Perils: More than 40% of projects with coding-agent usage markers had no commit-level markers, while 20% of projects with guidance or configuration files excluded all of them from commits.Different agents and workflows expose different evidence, and configuration may be ignored or stored outside the repository.
- Perils: Repository traces usually show final outputs rather than initial prompts or the extent of developer supervision.Pull requests can be more informative when comments preserve developer–agent interactions, but commits alone make supervision difficult to assess.
- Mitigations: The growing number and diversity of agents, combined with frequent tool changes, can make partial heuristics miss agent use and requires regularly updated detection.The paper motivates a community-maintained heuristic repository because agent-specific artifacts and workflows evolve rapidly.
6 Heuristics and data repository
Because coding-agent heuristics are numerous, varied, and change-prone, the paper proposes sharing them through a community-maintained repository. The repository supports discovery, interactive browsing, automated sampling, and targeted data collection.
- Knowledge sharing: The authors propose distributing the effort of maintaining coding-agent heuristics by sharing knowledge with the MSR community.They identify this as the mitigation for the multiplicity, diversity, and velocity of coding agents.
- Repository contents: The repository provides agent descriptions, time-bounded detection heuristics, GitHub queries, Python scripts, and about 10,000 repositories with agent adoption as of October 2025.These resources support interactive browsing, automated sample collection, and targeted data collection.
- Community maintenance: The authors call on the MSR community to add heuristics and report their effectiveness so the shared knowledge can improve continuously.Community participation is intended to scale maintenance beyond the small number of researchers who may struggle to keep pace with coding-agent changes.
7 Discussion
The paper limits its scope to mining coding-agent usage and emphasizes that its experience-based heuristics are noisy, incomplete, and time-sensitive. Nevertheless, repository traces open a broad research field, while widespread adoption requires MSR studies of human contributions to account for agent activity.
- Limitations: The paper focuses on promises and perils of mining coding-agent usage, excluding environmental, workforce, and intellectual-property issues concerning the agents themselves.Prior limitations of mining Git and GitHub also apply.
- Limitations: The authors’ experience-based account cannot guarantee exhaustive coverage, and its heuristics will require updates as coding agents evolve.They identify the shared heuristic and data repository as a mitigation for keeping the work current.
- Limitations: Heuristics are noisy and may require filtering, such as date ranges, depending on the accuracy a study seeks.The authors excluded Aider’s CONVENTIONS.md heuristic because most such files target developers rather than coding agents.
- Implications: Coding-agent traces open a broad field of studies on agent activity and its impact on Software Engineering.The paper presents examples but expects many more potential studies.
- Implications: If agent autonomy and parallel use continue increasing, the type, amount, and scope of delegated tasks may change rapidly and affect software-engineering practices.This projection assumes the observed trend of doubling agent autonomy on tasks every seven months is sustained for several years.
- Implications: Because agents are already widely used, studies of human contributions may need to identify and exclude agent contributions, as MSR studies exclude bot activity.At very high adoption, defining what counts as a human contribution becomes an open question.
8 Conclusion
Coding agents have rapidly become daily developer tools whose repository traces enable MSR studies at scale. The paper presents detection heuristics and argues that shared community knowledge is needed to study their diverse, fast-changing activity and effects on Software Engineering.
- Conclusion: Coding agents leave explicit traces in files, commits, issues, and pull requests, enabling large-scale MSR study of AI-assisted coding.The paper contrasts this observability with the difficulty of studying LLM-based code completion through MSR techniques.
- Conclusion: The paper’s principal perils are the multiplicity of diverse, fast-changing agents, while its main promise is extensive study of their effects on Software Engineering practices.The authors invite the MSR community to contribute to a shared repository of knowledge and heuristics.