PaperlayerView exact revision on arXiv

Source-linked AI summary

Engineering Signals of Human-AI Collaboration in the Agentic Coding Era: A Longitudinal Analysis of 33,228 Pull Requests from vLLM and SGLang with Implications for Biomedical AI Agents and Bioinformatics Pipeline Developmen

Jiada Li, Xuesong Ye, Olamide Olowoniyi

arXiv:2608.13884v1cs.SEcs.AIcs.ETcs.HCcs.LG

TL;DR

Team-level evidence on how AI coding tools reshape open-source contribution dynamics and human-AI collaboration remains limited. This study analyzes two AI infrastructure projects longitudinally and finds that throughput, contributor participation, and comment density increased substantially while PR size remained stable.

  • Problem

    Team- and repository-level changes in contribution dynamics, review culture, and human-AI collaboration under AI coding tools remain poorly understood.

  • Method

    The study conducts a descriptive longitudinal case analysis of two high-velocity open-source AI infrastructure projects across four AI-tooling eras without a comparison group.

  • Results

    Throughput, contributor participation, and comment density increased substantially across eras, while PR size remained stable; these trends were overwhelmingly human-driven.

  • Takeaways & Limitations

    As AI-assisted development scales contribution, review, validation, integration capacity, and explicit technical context may become increasingly important.

  • Takeaways & Limitations

    The findings may not generalize beyond two high-velocity, well-funded AI infrastructure repositories and cannot establish causal attribution to AI tooling.

Abstract

from arXiv · show

The rapid adoption of AI coding assistants and autonomous agentic development systems has coincided with major changes in the pace and structure of open-source software engineering. Yet empirical longitudinal evidence of these changes at the team level remains limited. We present a descriptive longitudinal analysis of seven engineering metrics: pull request (PR) throughput, cycle time, contributor diversity, PR comment density, merge rate, new-author participation, and PR size. Metrics were computed from all merged PRs in two high-velocity AI infrastructure repositories, vLLM (February 2023-June 2026; 18,290 PRs) and SGLang (January 2024-June 2026; 14,938 PRs). We segment development into four eras aligned with major changes in AI-assisted software development and examine human- and bot-authored activities. Both projects show substantial increases in development velocity and AI-developer collaboration signals. PR throughput increased 21x in vLLM and 17.9x in SGLang, while bot-authored PRs accounted for less than 0.2% of this growth, indicating that the increase was overwhelmingly human-driven. In the latest era, median cycle time was 1.04 days for vLLM and 0.62 days for SGLang, while P90 cycle times reached 16.8 and 14.3 days, respectively. Monthly unique authors increased steadily in both projects, suggesting broader contributor participation. PR comment density increased 4.2x in vLLM and 3.8x in SGLang, with bot comments contributing an estimated 15-20% of the increase. In contrast, PR size remained relatively stable across eras. Overall, AI-assisted development is associated with higher throughput, broader contributor participation, and increased AI-developer collaboration signals in high-velocity open-source software development.

1. Introduction

AI coding assistants and autonomous agentic systems are described as reshaping software production, with controlled studies reporting individual-developer productivity gains of 26–56%. The study examines this shift longitudinally in vLLM and SGLang through engineering metrics, repository-level collaboration signals, and implications for bioinformatics pipeline development.

  • Motivation: 26–56% productivity gains were reported for individual developers using AI coding tools in controlled experiments.A meta-analysis across 29 studies reported a moderate positive productivity effect (g = 0.33, 95% CI [0.09, ...).
  • Study scope: vLLM and SGLang were selected as descriptive longitudinal cases because they intensively adopt AI coding tools, use transparent continuous-integration PR histories, and span the full arc of adoption.The projects are described as foundational to the AI ecosystem and among the earliest intensive adopters.
  • Research questions: Seven engineering metrics are tracked across four eras: PR throughput, cycle time, unique authors per month, comment density, monthly merge rate, new-author share, and PR size.This research question focuses on how these metrics evolve during AI tooling adoption in high-velocity open-source AI infrastructure projects.
  • Research questions: Repository-level behavioral signals are used to investigate shifts in human-AI collaboration patterns within software engineering teams.The study asks how such collaboration patterns can be detected from repository-level data.
  • Research questions: The study also examines how the engineering metrics and collaboration signals map to bioinformatics pipeline development.This question targets specialized life-science engineering contexts.

2. Background and Related Work

Prior work documents productivity gains, rapidly improving autonomous coding agents, and established frameworks for measuring software-engineering performance and collaboration. Related life-science systems extend agentic automation to bioinformatics analysis and workflow construction.

  • AI coding assistants: 55.8% productivity increase was observed in a randomized experiment of developers implementing a JavaScript HTTP server.The experiment was conducted by Peng et al.
  • Agentic software engineering: 78.4% SWE-bench Verified performance in April 2026, up from 1.96% in August 2024, illustrates rapid progress in autonomous coding agents.These systems can plan, implement, test, and submit changes with minimal human intervention.
  • Repository mining and metrics: 3.3 million pull requests across 11,230 projects established a large empirical basis for repository-mining research.Repository mining supports PR-level analysis of software-engineering processes, including merge decisions and discussion sentiment.
  • Repository mining and metrics: 28% increase in code shipment volume and reduced PR review cycle time were reported for AI-assisted development tools.The findings complement multidimensional productivity frameworks such as SPACE and delivery-focused DORA metrics.
  • Biomedical and bioinformatics agents: 68.6% accuracy on genomics benchmarks was achieved by ToolsGenie 2.0 among recent AI-assisted pipeline-automation systems.The related systems include PromptBio, CARIBOU, and the KBase Research Agent for bioinformatics analysis and workflow construction.

3. Data and Methodology

The study uses public GitHub histories from vLLM and SGLang as descriptive longitudinal case studies, dividing development into four AI-tooling eras. It computes seven project-month metrics from merged PRs and applies regression, confidence intervals, and normalized heatmap visualization.

  • Data collection: 18,290 merged PRs were retrieved through full-population, date-windowed GitHub Search API queries, with high-volume months split into half-month windows to avoid the 1,000-result cap.Opened and merged PRs were grouped by different dates and therefore represent different monthly cohorts.
  • Development eras: Four development eras were defined around major AI coding-tool general-availability milestones, including the 3 Jan 2025–Jun 2026 Agentic Coding Emergence era.This era included Cursor Agent becoming the default, Claude Code launch/GA, and Cursor Background Agent GA.
  • Metric construction: Seven metrics were computed per project-month: PR throughput, cycle time, unique authors, PR comment density, monthly merge rate, new-author share, and PR size.Cycle time used median and 90th-percentile days; PR size used median lines added, lines deleted, and files changed.
  • Metric construction: Monthly merge rate was defined as the ratio of merged PRs to opened PRs, a flow ratio rather than an acceptance probability because its monthly cohorts differ.Values >1 are possible when the merge backlog clears faster than new PRs arrive.
  • Statistical analysis: OLS estimated monthly growth in unique-authors-per-month time series with 95% confidence intervals, while era means and a cross-project z-score heatmap summarized metric patterns.Analysis was performed in Python using pandas, scipy, and matplotlib.

4. Results

The results summarize quantitative findings for vLLM and SGLang across all seven engineering metrics, using the full merged-PR population without sampling.

  • Table 2 summarizes key quantitative results across vLLM and SGLang for all seven metrics.
  • All reported values were computed from the full merged-PR population, with no sampling.

4.1 RQ1: Metric Evolution Across AI Tooling Eras · 4.1.1 Metric 1: PR Throughput · 4.1.2 Metric 1: Bot/Human Activity Decomposition

Across four AI tooling eras, merged PR throughput increased sharply in both vLLM and SGLang, while bot-authored PRs remained negligible. The decomposition therefore indicates that the throughput expansion was overwhelmingly associated with human-authored activity, although repository metadata cannot directly validate AI authorship.

  • 4.1.1 Metric 1: PR Throughput: 21×: vLLM increased average monthly merged PRs from 38 in Era 0 to 797 in Era 3.Its Era 1-to-Era 3 growth was 5.9×.
  • 4.1.1 Metric 1: PR Throughput: 17.9×: SGLang increased average monthly merged PRs from 42 at its Era 1 launch to 743 in Era 3.Figure 1 reports monthly throughput across all four eras using full-population data.
  • 4.1.1 Metric 1: PR Throughput: Order-of-magnitude throughput growth motivated the hypothesis that AI-generated PRs comprised a substantial submission fraction.The passage notes that individual developer throughput cannot explain this growth without a corresponding order-of-magnitude increase in team size.
  • 4.1.1 Metric 1: PR Throughput: Direct AI-authorship validation was unavailable from repository metadata alone.The passage presents this as a limitation of the hypothesis about AI-generated PRs.
  • 4.1.2 Metric 1: Bot/Human Activity Decomposition: 0.17%: vLLM’s bot-authored merged PRs comprised 31 of 18,213 total merged PRs.All 31 were from dependabot[bot] (24 dependency updates) and Copilot (7 PRs).
  • 4.1.2 Metric 1: Bot/Human Activity Decomposition: 1: SGLang had one bot-authored merged PR among 14,830 total merged PRs.The passage reports bot PR prevalence as negligible in both projects.
  • 4.1.2 Metric 1: Bot/Human Activity Decomposition: <0.2%: bot-authored PRs represented a negligible share of merged PRs in both projects across eras.Figure 2 separates human- and bot-authored merged PRs and characterizes the observed throughput growth as human-driven.

4.1.3 Metric 2: PR Cycle Time · 4.1.4 Metric 3: Unique Authors per Month

PR cycle times were fast at the median but substantially more variable in the latest era, with long-tailed review latency in both projects. Monthly unique-author counts increased significantly in vLLM and SGLang.

  • 4.1.3 Metric 2: PR Cycle Time: 1.04 days was vLLM’s Era 3 median PR cycle time, with a 16.8-day P90.These values were computed from the full merged-PR population.
  • 4.1.3 Metric 2: PR Cycle Time: 0.62 days was SGLang’s Era 3 median PR cycle time, with a 14.3-day P90.These values were computed from the full merged-PR population.
  • 4.1.3 Metric 2: PR Cycle Time: 0.85 days was vLLM’s all-time median PR cycle time, compared with 0.37 days for SGLang.Both all-time medians were computed from the full merged-PR population.
  • 4.1.3 Metric 2: PR Cycle Time: The wide median–P90 band in Era 3 reflects increasing heterogeneity in review latency.While half of PRs merge within approximately one day, a significant tail requires weeks of review.
  • 4.1.4 Metric 3: Unique Authors per Month: Both projects show statistically significant upward trends in monthly unique-author counts (p< 0.001).Counts and OLS regression lines were computed from the full merged-PR population with 95% confidence intervals.
  • 4.1.4 Metric 3: Unique Authors per Month: Monthly unique-author counts were analyzed with OLS regression lines and 95% confidence intervals across eras.Figure 4 uses era-colored data points from the full merged-PR population.

4.1.5 Metric 4: Multi-Metric Heatmap · 4.1.6 Metric 5: PR Comment Density

The multi-metric heatmap shows strong Era 3 increases in throughput, unique authors, and comment density, alongside slower cycle times and declining monthly merge rates. PR comment density increased substantially across both projects throughout the timeline.

  • 4.1.5 Metric 4: Multi-Metric Heatmap: Era 3 shows strong positive trends in throughput, unique authors, and comment density across the heatmap.The heatmap uses z-score normalization across all project-months.
  • 4.1.5 Metric 4: Multi-Metric Heatmap: Era 3 cycle-time performance degraded in both projects, although SGLang maintained fast cycle times through Era 2.Cycle time is inverted in the heatmap, so green indicates faster performance.
  • 4.1.6 Metric 5: PR Comment Density: 4.2×: vLLM mean PR comment density increased from 0.87 to 3.63 comments/PR between Era 0 and Era 3.The calculation uses the full merged-PR population.
  • 4.1.6 Metric 5: PR Comment Density: 3.8×: SGLang mean PR comment density increased from 0.74 to 2.81 comments/PR between Era 0 and Era 3.Both projects show upward trends across the full timeline.
  • 4.1.6 Metric 5: PR Comment Density: Rising comment density may reflect greater community participation, increasing PR complexity, or changing review norms.The figure caption notes that PR-size data suggests increasing PR scope is not the primary driver.

4.1.7 Metric 6: New-Author Share per Month · 4.1.8 Metric 7: PR Size

Both projects sustain approximately 40% new-author shares in the latest era while retaining growing returning-contributor bases. PR size remains stable across eras, with similar median additions and unchanged median files changed in vLLM.

  • 4.1.7 Metric 6: New-Author Share per Month: Approximately 40%: Both vLLM and SGLang maintain this Era 3 new-author share.Both projects retain a mix of new and returning contributors across all eras.
  • 4.1.7 Metric 6: New-Author Share per Month: ∼61%: vLLM’s new-author share peaked in Era 0 before gradually declining.The returning contributor base grew in absolute terms as the new-author share declined.
  • 4.1.7 Metric 6: New-Author Share per Month: ∼64%: SGLang’s new-author share peaked in Era 1 before gradually declining.The project nonetheless sustained approximately 40% new-author participation in Era 3.
  • 4.1.8 Metric 7: PR Size: 18 lines to 19 lines: vLLM’s median additions per merged PR changed little from Era 0 to Era 3.The metric was examined to test whether rising comment density reflected increasing PR scope.
  • 4.1.8 Metric 7: PR Size: 2: vLLM’s median files changed per merged PR remained constant across all eras.This stability indicates no broad increase in the number of files touched per PR.
  • 4.1.8 Metric 7: PR Size: 26 median additions to 22: SGLang’s median additions per merged PR remained broadly stable between Era 1 and Era 3.The supplied passage reports a similar stability pattern for SGLang.
  • 4.1.8 Metric 7: PR Size: PR size remained stable across eras despite order-of-magnitude throughput growth.The figure caption characterizes the gains as more PRs of similar scope rather than larger PRs.

4.1.9 Human vs. Bot Metric Comparison

Human activity dominates bot activity across the compared metrics in both projects. Bot contributions are negligible for throughput and author participation, while bot comments account for an estimated 15–20% of the increase in comment density.

  • PR Throughput: Bot-authored PRs are negligible compared with human-authored PRs in both projects.The dashed bot-throughput lines are barely visible above the x-axis.
  • Unique Authors: <1% of unique authors were bots in all eras.Bot activity was also negligible for PRs per author.
  • Comment Density: 15–20% of the comment density increase was attributed to bot comments.The figure distinguishes human activity with solid lines from bot activity with dashed lines.
  • Bot Activity: Bot PR share peaked in Era 2 and declined in Era 3.Bot cycle times were highly variable, but the sample was too small for meaningful conclusions.

5. Discussion

The discussion interprets rising collaboration metrics as potential evidence of healthier coordination while identifying merge-rate declines, longer review tails, and contributor patterns as warning signs. It frames these findings through leadership practices, the Judgment Gap and context collapse, and a conceptual—not yet empirically validated—extension to bioinformatics pipelines.

  • Collaboration signals: Rising comment density with throughput may signal stronger pull-request coordination, while returning contributors may indicate effective onboarding and community integration.Stable PR size alongside rising throughput is also presented as a potential indicator of disciplined scoping.
  • Collaboration signals: 0.685→0.541 and 0.703→0.620 were the Era 3 monthly merge-rate declines for vLLM and SGLang, respectively, suggesting review bottlenecks as opened-PR volume outpaces merge capacity.Cycle-time P90 reached 16.8 days for vLLM and 14.3 days for SGLang, indicating that complex PRs may wait longer for review.
  • Engineering management: Table 3 maps PR throughput, cycle time, unique authors, and PR size to management practices concerning AI-assisted velocity, review speed, community growth, onboarding, and contribution scoping.For PR size, the recommended practice is to track median lines and files per PR.
  • Leadership risks: The Judgment Gap describes the widening space between what AI agents can execute and what engineering leadership requires, while context collapse occurs when technically correct code lacks organizational context.The discussion assigns responsibility for supplying architectural principles, team conventions, and current priorities to leadership.
  • Bioinformatics implications: The bioinformatics extension is conceptual because no bioinformatics teams were analyzed; it emphasizes context loss across wet-lab handoffs and risks such as provenance gaps and biologically invalid but error-free results.The proposed mapping requires domain-specific modifications and empirical validation.
  • Limitations: The study may not generalize beyond high-velocity, well-funded AI infrastructure repositories with strong review cultures and active maintainers.The era segmentation also uses AI-tooling availability dates as proxies for actual adoption.

6. Conclusion

Across 33,228 merged pull requests, throughput, contributor participation, and comment density increased substantially while PR size remained stable, and these changes were overwhelmingly human-driven. The findings motivate recalibrated productivity benchmarks, greater review and integration capacity, and broader causal and comment-level analyses.

  • 33,228 merged PRs across four AI-tooling eras showed increased throughput, contributor participation, and comment density, while PR size remained stable.
  • <0.2% of merged PRs were bot-authored, contributing 0.17% of vLLM and 0.00% of SGLang throughput growth.Bot comments contributed an estimated 15–20% of the comment density increase, with 80–85% attributable to human activity.
  • 4.2× in vLLM and 3.8× in SGLang, PR comment density rose across eras while PR size remained stable.The passage frames this as a hypothesis requiring comment-level analysis to distinguish among community participation, changing review norms, and AI-specific review dynamics.
  • Traditional productivity benchmarks may require recalibration as AI tooling changes the baseline scale of software contribution.Review, validation, and integration capacity may emerge as critical bottlenecks as code generation becomes increasingly automated.
  • Broader repositories, causal identification strategies, and comment-level classification are priorities for future work.The proposed extensions cover different domains, team sizes, AI-tool adoption levels, staggered adoption designs, and comment categories such as clarification, approval, and architectural discussion.
Loading 2608.13884v1…