PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-chat: Coding Agent Interactions From Real Users in the Wild

Joachim Baumann, Vishakh Padmakumar, Xiang Li, John Yang, Diyi Yang, Sanmi Koyejo

arXiv:2604.20779v1cs.AIcs.CYcs.SE

TL;DR

SWE-chat addresses the lack of empirical evidence about how developers use coding agents and how much of their output survives in practice. It constructs a large-scale dataset of real sessions with interaction traces and authorship attribution, finding bimodal usage, inefficient output, security risks, and frequent user pushback. The authors position these findings as an initial foundation for evaluating agents in real developer workflows.

  • Problem

    Empirical understanding of how developers interact with full coding-agent sessions, use their output, and respond to failures remains limited beyond curated benchmarks.

  • Method

    The paper builds SWE-chat from real coding-agent sessions, combining complete interaction traces, tool-call trajectories, code diffs, authorship attribution, and efficiency and safety metrics.

  • Results

    Coding-agent usage is bimodal and inefficient: less than half of agent-produced code survives into commits, while vibe-coded code introduces more vulnerabilities than human-written or collaboratively authored code.

  • Takeaways & Limitations

    SWE-chat provides an empirical basis for studying human-agent interaction, efficiency gaps, and failure modes beyond controlled evaluations.

  • Takeaways & Limitations

    The findings are an initial, non-definitive starting point for broader in-the-wild agent evaluation and human-agent interaction studies.

Abstract

from arXiv · show

AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes. We find that coding patterns are bimodal: in 41% of sessions, agents author virtually all committed code ("vibe coding"), while in 23%, humans write all code themselves. Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Just 44% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than code authored by humans. Furthermore, users push back against agent outputs -- through corrections, failure reports, and interruptions -- in 44% of all turns. By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat provides an empirical foundation for moving beyond curated benchmarks towards an evidence-based understanding of how AI agents perform in real developer workflows.

1 Introduction

SWE-chat addresses the lack of empirical evidence about real-world human-agent coding interactions by analyzing complete sessions from actual users. It shows diverse, iterative usage alongside substantial inefficiency, security risks, and user pushback.

  • Research gap: Existing evidence on full coding-agent sessions is largely anecdotal, while curated benchmarks omit how developers prompt, steer, override, and commit agent-produced code.SWE-chat targets the gap between benchmark performance and iterative development workflows.
  • Contribution: SWE-chat combines real user interactions, agent trajectories, contextual information, and code authorship attribution in a dataset of actual coding-agent sessions.This supports analysis of both agent outputs and human responses to them.
  • Observed usage: Understanding existing code is the most common user intent, while agents spend a third of their tool calls executing bash commands rather than editing files.This indicates that real workflows extend beyond narrow patch-generation tasks.
  • Observed usage: Coding modes are highly bimodal: agents often write either none or nearly all of the code, but sessions typically span multiple turns and involve detailed user steering.Fully autonomous one-shot problem-solving remains uncommon in the analyzed usage.
  • Failure modes: Less than half of agent-produced code survives into user commits, while vibe-coded code uses roughly 3× more tokens and dollars per committed line than collaborative coding.Vibe-coded code also introduces roughly 9× more vulnerabilities than human-written code and about 5× more than collaboratively authored code.
  • Failure modes: Users compensate for agents’ limited clarification behavior by interrupting, correcting, and pushing back against agent outputs during sessions.The supplied evidence reports interruptions in 5% of turns and pushback in 39% of turns.

2 SWE-chat

SWE-chat is a continuously collected dataset of multi-turn coding-agent sessions from real users and repositories. Its logs, authorship data, annotations, and efficiency and safety metrics support large-scale study of human-agent workflows and failures.

  • Data collection: SWE-chat collects session transcripts from opted-in public GitHub repositories and links checkpoints to commits with line-level human-versus-agent authorship attribution.Logs include prompts, responses, tool calls, edits, shell commands, searches, and token usage.
  • Data structure: Each session alternates user prompts with agent responses containing tool calls and text output.Tool calls include file reads, edits, and shell commands.
  • Dataset statistics: Figure 4 summarizes turns per session, agent tool calls per turn, and the file types most frequently touched by those calls.The dataset contains multi-turn sessions from hundreds of real users interacting with five coding agents.
  • Annotations: The dataset is enriched with annotations intended to study human-agent collaboration and help model developers build more helpful agents.Annotation codebooks were evaluated with inter-annotator agreement, while LLM judges enabled scalable annotation of the full dataset.
  • Analysis methodology: Raw logs and code attribution support metrics for code survival, agent self-rewrites, tokens, cost, time, and user effort per committed line.Semgrep analysis compares security findings introduced across coding modes using pre- and post-commit snapshots.

3 How do humans interact with coding agents in the wild? (RQ1)

SWE-chat shows that real coding-agent use extends beyond patch generation, with diverse tasks, tool-heavy workflows, and sharply bimodal human–agent code authorship. Users also iteratively refine instructions and frequently delegate nearly all committed coding to agents.

  • Task types: 19.0% of prompts concern understanding existing code or behavior, the most common specific request, while creating code, git operations, and debugging are also prevalent.Creating new code and git operations each account for 13.4% of prompts, while debugging accounts for 13.0%.
  • Tool use: One third of agent tool calls are bash commands, predominantly git operations, followed by file reads, edits, and grep searches.Agent trajectories generally begin with reading and searching before moving to file modifications and build commands.
  • Coding modes: 55.8% of committed lines are agent-authored, but sessions are extremely bimodal across human-only, collaborative, and vibe-coding modes.The modes distinguish sessions by the share of committed code authored by agents.
  • Coding modes: 40.8% of sessions are vibe coding, where agents author more than 99% of committed code, and its share doubled from 20% to over 40% during three months.Human-only sessions comprise 22.7%, while collaborative sessions comprise 36.5%.
  • User types: Most users are expert nitpickers who correct agent output while maintaining stable goals, and 47% of vibe-coding sessions fit this persona.Users iteratively refine instructions after seeing agent outputs, unlike benchmarks that provide complete instructions upfront.

4 How do coding agents fail and how do users respond? (RQ2)

Most sessions receive favorable success ratings, but real-world agent use remains inefficient and creates security risks. Users discard substantial agent output, while interruptions and pushbacks provide frequent human oversight of agent autonomy.

  • Session success: 90% of sessions receive success ratings of 50 or higher, although the lowest-rated sessions commonly end through user interruptions or unrelated agent work.The 50 sessions rated 2–15 were manually inspected to identify these failure modes.
  • Coding efficiency: 44.3% of agent-produced code survives into user commits, while vibe-coding sessions retain 59% on average.The higher vibe-coding survival rate may reflect better-targeted output or lower user scrutiny, so the comparison is not causal.
  • Coding efficiency: Vibe-coded sessions consume a median of 204K tokens per 100 committed lines, roughly 3× collaborative sessions and 2× human-only sessions.Their median cost is $0.13 per 100 committed lines, compared with $0.07 for human-only and $0.05 for collaborative sessions.
  • Coding efficiency: Collaborative sessions are fastest at 4.8 minutes per 100 committed lines, compared with 12.6 minutes for vibe coding and 8.6 minutes for human-only sessions.Time and agent runtime omit user coding time before or after a session and are therefore imperfect proxies.
  • Security: Vibe-coded commits introduce 0.76 vulnerabilities per 1,000 committed lines, roughly 9× human-only and 5× collaborative rates.They also fix more vulnerabilities, but every mode introduces more vulnerabilities than it fixes, with the largest gap in vibe coding.
  • Oversight: Users interrupt agents in 3.3%–6.0% of turns and issue correction prompts after 39% of turns, while agents ask for clarification in only 1.1%–2.6%.Interruptions most often occur when agents exit plan mode, perform git operations, or edit files.

5 Discussion

SWE-chat reveals that coding agents are increasingly autonomous but remain inefficient, brittle, and dependent on frequent human oversight. The dataset also supports more realistic benchmarks, adaptive interaction design, and user simulators grounded in real workflows.

  • Autonomy is outpacing oversight: 44% of turns contain user pushback, while agents ask clarifying questions in only 1.4% of turns, leaving users to compensate through manual oversight.Pushback includes corrections, rejections, and failure reports.
  • Agents are powerful but brittle: Less than half of agent-produced code survives into commits, and collaborative human-agent sessions are the most cost-efficient mode observed.The findings indicate that greater autonomy does not necessarily yield more efficient delivery.
  • Agent-written code introduces more security vulnerabilities: Vibe-coded commits introduce Semgrep-detected vulnerabilities at roughly 9× the human-authored rate.
  • Realistic benchmarks grounded in real workflows: SWE-chat enables benchmarks grounded in real session trajectories, including evaluation of whether agents propose appropriate next actions in conversation context.
  • Designing more adaptive human-agent interaction: SWE-chat captures large-scale correction-response cycles that can support research on human oversight and shortcomings in current interaction design.
  • Living evaluation resources: The living dataset can support longitudinal analysis as developer behavior and coding-agent technology change.

A Limitations

The paper’s limitations concern dataset representativeness, incomplete observation of failures and semantic reuse, proxy-based efficiency measures, and unreliable LLM-generated labels. The appendix examples illustrate these interaction patterns but are representative rather than exhaustive.

  • Dataset scope: SWE-chat covers developers using Entire CLI in public repositories who opted into checkpoint logging, so findings may not generalize to proprietary enterprise codebases or other users.A large fraction of the data also comes from Entire.io’s own repository.
  • Measurement coverage: Abandoned sessions are not captured, potentially overestimating success and efficiency, while line-level attribution can underestimate agent usefulness when suggestions are rewritten or refactored.
  • Metric limitations: Committed-line and session-log efficiency metrics are proxies that omit factors such as time spent reading, reviewing, and planning.
  • Annotation reliability: LLM-generated labels are used for scalable filtering but should not support conclusive claims without further validation.
  • Representative examples: The appendix presents representative real-session examples of pushback, clarification, intent categories, and user personas rather than a complete account of session behavior.

C Experimentation details

SWE-chat measures agent efficiency, code survival, authorship, and resource use directly from session logs and commit-linked code attribution. The analyses account for session-to-commit mapping and distinguish total agent effort from net output retained by users.

  • Measurement: Session logs directly measure runtimes, tool-call durations, token usage, and files touched, enabling annotation-free efficiency metrics.The metrics are computed from raw coding-agent data rather than requiring manual annotations.
  • Authorship attribution: Agent-authored code percentages are computed at commit time using checkpoints on shadow branches to attribute committed lines to humans or agents.Temporary checkpoints provide the basis for line-level authorship attribution.
  • Code survival and efficiency: Coding efficiency measures the fraction of total agent-produced effort retained in the commit, including lines later rewritten by the agent.The metric divides agent lines that survived by cumulative agent lines produced.
  • Code survival and efficiency: Code survival rate measures the fraction of the agent’s net output in the final state that humans kept unchanged.It differs from coding efficiency because it excludes agent self-overwrites from the denominator.
  • Analysis scope: 48.6% of sessions are included in commit-level analyses because their session-to-commit line attribution is unambiguous.Sessions may span multiple commits, and multiple sessions may contribute to one commit, so the analysis applies a clean-mapping restriction.

D.1 Dataset statistics

The dataset statistics characterize prompt language and tool-call categories across the collected sessions. English dominates user prompts, while tool calls are summarized using aggregate categories for reporting.

  • Prompt languages: User prompts are predominantly in English.Language detection retains languages with at least 100 prompts and manually verifies 2,000 low-confidence or extremely low-resource classifications.
  • Tool calls: Tool-call types are grouped into aggregate categories to simplify the full distribution presented in Table 5.The table reports the distribution across all agent tool calls.

D.1.3 Agent trajectories

Agent trajectories typically begin with codebase research and progress toward execution and final changes. Interrupted turns often end at the transition from planning to execution, while the dataset mainly represents practical software-engineering repositories.

  • Trajectory composition: Early trajectory positions commonly use read, grep, glob, and git/gh tools to orient the agent in the codebase.These are research-oriented tools used before action tools become more prominent.
  • Trajectory composition: Action tools such as edit, write, and bash:build become more prominent as trajectories progress.The sequential-position analysis shows a shift from research toward implementation and execution.
  • Trajectory endpoints: Natural turns most frequently end with git/gh commands, bash:build, or edit before the agent writes its response.The reverse trajectory analysis identifies the final tool-call composition for uninterrupted turns.
  • Trajectory endpoints: 32% of interrupted turns end with ExitPlanMode, indicating interruption at the transition from planning to execution.In these cases, the agent has finalized its plan and users redirect it before code changes are made.
  • Repository context: Most repositories are user-facing applications or developer tools, so SWE-chat primarily reflects practical software-engineering environments.Repositories are classified by domain and audience using names, descriptions, and README files.

D.1.5 Dataset diversity over time

SWE-chat’s repository mix changes as adoption expands, while prompt analysis characterizes the tasks users bring to coding assistants. Entire.io’s repository contributes a declining minority of sessions after launch.

  • Dataset growth: After Entire.io’s February 10, 2026 public launch, its repository contributed less than 20% of SWE-chat sessions, with its share declining as adoption continued.Figure 22 tracks this cumulative session fraction over time.
  • Prompt diversity: SWE-chat characterizes user tasks through topic analysis of all English user prompts.The analysis is designed to describe the range of tasks users bring to AI coding assistants.

D.2.1 Topic clustering methodology

The methodology cleans and deduplicates English prompts, embeds them, and uses clustering plus LLM-generated labels to characterize prompt topics and session behavior. The resulting analysis identifies 20 clusters covering 57.4% of prompts and links them to pushback, duration, success, and persona patterns.

  • Topic clustering methodology: Prompts are cleaned by removing interruptions, injected messages, skill invocations, images, code blocks, and texts outside the 30–1,500-character range before deduplication.The remaining prompts are embedded with all-mpnet-base-v2.
  • Topic clustering methodology: 20 topic clusters cover 57.4% of all prompts, with each cluster labeled by a GPT-generated topic summary.Cluster analyses include pushback rate, agent turn duration, and session success distributions.
  • Topic clustering methodology: Frontend coding has the highest pushback rate at 75%, while cluster 17 contains unusually long multi-task prompts associated with longer agent turns.Cluster 12 also contains many prompts that appear to have been generated automatically.
  • User personas: Most sessions classify users as expert nitpickers, indicating a dominant persona characterized by precise corrections and high standards.The persona distribution is shown across all sessions.
  • Coding modes: Vibe-coding sessions increased from approximately 20% to over 40% after the Entire CLI launch.The temporal evolution is presented as a 14-day rolling average of coding modes and agent-authored code.

D.5 Code vulnerability analysis with Semgrep

The vulnerability analysis scans repository states before and after commits with Semgrep and retains findings introduced within changed files. It also compares coding-mode efficiency and examines autonomy, interruptions, pushback, and behavior across development activities.

  • Vulnerability analysis: Semgrep scans repository states before and after each commit using its default auto-selected rules, retaining findings in changed files.The ruleset includes Common Weakness Enumeration categories.
  • Vulnerability analysis: Introduced vulnerabilities span many rule and CWE types, including path-joining flaws, externally controlled format strings, missing integrity checks, OS command injection, and SQL injection.One JavaScript path-joining rule accounts for most detected vulnerabilities, but a long tail remains.
  • Vulnerability analysis: The dataset includes an agent-introduced Python OS-command-injection example caused by interpolating user-controlled input into a shell command.The figure identifies CWE-78 and shows the standard list-argument fix.
  • Efficiency: Vibe-coding sessions consume roughly twice as many tokens and require more wall-clock time per 100 committed lines than collaborative sessions.Collaborative coding achieves the best trade-off across the compared efficiency dimensions.
  • Autonomy and interaction: The 99.9th percentile of agent turn duration now exceeds 100 minutes, despite relatively stable median durations.This indicates growth in the tail of autonomous runs over the collection period.
  • Autonomy and interaction: Agent-initiated stops, user interruptions, and user pushback remain relatively stable from January through March 2026.These measures are visualized with a seven-day rolling window.
  • Development activities: Code-writing prompts trigger longer turns than code-reviewing prompts, averaging 4.1 versus 2.4 minutes, alongside more file writes and edits.Writing prompts also elicit more friction than reviewing prompts.

E.1 Validation

Validation develops human-checked annotation codebooks, measures annotator agreement, and evaluates LLMs against 100 human gold labels before selecting models for full-dataset annotation. The procedures cover repository, persona, and prompt-intent classifications.

  • Human validation: Two annotators iteratively refined each codebook until agreeing on 10 data points, then independently labeled 90 additional points.This produced moderate-high inter-annotator agreement across tasks.
  • Human validation: After resolving disagreements, the annotators produced 100 human gold labels for evaluating LLM annotation performance.Cohen’s κ and percentage agreement were reported for categorical tasks, while session success used a continuous score.
  • LLM validation: The study tested 9–11 LLMs with 2–4 prompt paraphrases per task against the human gold labels.The best-performing prompt was reported for each model-task combination before selecting models for full-dataset annotation.
  • Repository type classifier: The repository classifier assigns one domain tag and one target-audience tag from predefined categories using repository metadata and README context.Library and devtools are aggregated because human annotators often disagreed between those categories.
  • User persona classifier: The persona classifier labels users as Expert Nitpicker, Vague Requester, Mind Changer, or Other based on behavioral patterns across the full session timeline.The codebook distinguishes stable-goal execution corrections from changes to the overall goal.
  • Prompt intent classifier: The prompt-intent classifier assigns exactly one primary intent, including create new code, refactor, debug, understand, connect, git, or test.When multiple intents appear, the instructions prioritize the main requested action.
Loading 2604.20779v1…