Source-linked AI summary
AgentLogs: A Dataset for Opening the Black Box of GitHub's Cloud Agent
Jonan Richards, Kosei Horikawa, Youmei Fan, Yutaro Kashiwa, Mairieli Wessel
TL;DR
Existing datasets largely describe agent-contribution outcomes, leaving the process of producing them less explored. AgentLogs addresses this gap with a large-scale GitHub dataset of task, session, and step-level activity, enabling study of agent behavior, efficiency, failures, and collaboration.
Problem
Existing datasets capture agent-contribution outcomes, while the process by which agents produce those contributions remains largely unexplored.
Method
AgentLogs collects GitHub agent tasks, sessions, and detailed session logs containing prompts, reasoning traces, tool calls, outputs, and usage metadata.
Results
The dataset exposes step-by-step agent execution rather than only the resulting contributions, supporting analysis of behavior, efficiency, cost, task formulation, failure modes, and collaboration.
Takeaways & Limitations
AgentLogs provides a basis for studying how software-engineering agents work across workflows, human feedback, repository context, models, and adoption.
Abstract
from arXiv · showhide
Generative AI-based software engineering agents are becoming routine contributors to real-world software projects. On GitHub, developers can assign tasks to the Copilot cloud agent, which autonomously explores the repository, edits code, runs commands, and opens or reviews pull requests, producing a detailed log of every step along the way. While existing datasets capture outcomes of agent contributions, such as agent-authored pull requests, the process by which agents produce these contributions remains largely unexplored. To address this gap, we introduce AgentLogs, a large-scale dataset of agent activity on GitHub. AgentLogs comprises 307,416 agent tasks and 549,239 agent sessions in 35,810 of the 1,812,362 popular public repositories that we scanned, together with 64,255,174 session log entries that record each agent run step by step, including prompts, intermediate reasoning, tool calls (e.g., file edits, git operations, and GitHub interactions), and token usage. By exposing not only what agents contribute but also how they work, AgentLogs enables research on agent behavior, efficiency and cost, task formulation, failure modes, and human-agent collaboration in agentic software engineering.
I. DATASET OVERVIEW
AgentLogs records GitHub Copilot agent activity at task, session, and step levels, extending beyond outcome-only datasets to capture execution details. The dataset was collected through GitHub APIs and includes session logs with prompts, model outputs, tool calls, and usage metadata.
- Each agent task can contain one or more sessions, with logs recording model output, tool calls, usage metadata, prompts, and reasoning traces.
- The collection combines a popular-repository seed dataset with GitHub REST and Copilot API endpoints to retrieve task and session metadata.
- AgentLogs captures step-by-step agent execution, including file searches and delegation to sub-agents, in greater detail than GitHub currently displays.
II. INTERNAL STRUCTURE
AgentLogs is organized into structured, sharded Parquet tables designed to support storage and processing of large-scale agent records.
- AgentLogs stores repository, agent task, agent session, log entry, and user records in separate Parquet tables.
- The tables are sharded into downloadable chunks of at most approximately 220MB for log entries, 120MB for sessions, and 80MB for remaining tables.
III. HOW TO ACCESS
The dataset is publicly distributed with documentation, examples, and tooling for both local and streaming analysis. A small repository sample is also provided for initial exploration.
- AgentLogs is available on Hugging Face, with schema documentation, analysis notebooks, sample data, Python type definitions, and stated licenses.
- Example workflows support DuckDB, Polars, incremental Parquet processing, custom Python analysis, and Hugging Face streaming without a local copy.
- The GitHub sample contains 3 repositories, 4 agent tasks, 5 agent sessions, 416 log entries, and 3 users.
IV. POTENTIAL RESEARCH QUESTIONS
AgentLogs frames research questions around agent behavior, human–agent collaboration, and ecosystem adoption. The proposed analyses examine workflows, context use, feedback, developer adaptation, repository adoption, model differences, and growth over time.
- Agent Behavior and Strategies: Researchers can study workflow patterns across sessions and task types, including exploration, editing, and testing.
- Agent Behavior and Strategies: The dataset supports analysis of repository-provided context and its association with agent behavior and outcomes.
- Human–Agent Collaboration: Human–agent collaboration questions include responses to reviewer feedback and changes in developers’ request style, task scope, and follow-up behavior.
- Adoption and Ecosystem: Adoption questions include what characterizes the 1.98% of popular repositories using coding agents and how usage grows over time.
- Adoption and Ecosystem: Because sessions include model metadata, researchers can compare the behavior of underlying models.
APPENDIX A EXAMPLE SESSION LOG
The example session shows Copilot responding to pull-request feedback by editing the changelog and replying to the reviewer. The log records prompts, reasoning, tool calls, repository context, timestamps, and token usage.
- Pull request context: The session begins with a pull request replacing short curl flags with long options and alphabetizing check_dep calls in install.sh.The changes affect lib/core_utils.sh and install.sh.
- Pull request feedback: A reviewer requests that the Unreleased changelog contain only the single self-upgrade change from wget to curl.The request appears in a comment on CHANGELOG.md:10.
- Agent response: The agent reasons about the reviewer’s request and edits CHANGELOG.md to replace the earlier three-item list with one changelog entry.The edit is issued through an edit tool call, followed by confirmation that the file was updated.
- Agent response: The agent replies to comment 3088356364, stating that the changelog was fixed in commit a982039.The reply confirms that the Unreleased section now contains only the requested entry.