PaperlayerView exact revision on arXiv

Source-linked AI summary

Chance, Persistent Advantage, and the Generative-AI Era in Open-Source Package Careers

Hazem Ibrahim, Yasir Zaki

arXiv:2609.09687v1cs.SIcs.CY

TL;DR

The paper asks whether the luck-and-persistence pattern found in science and the arts also describes open-source package careers, and whether it changed in the generative-AI era. It analyzes GitHub careers and finds near-lottery hit timing, a persistent factor alongside momentum, and no detectable within-person era shift beyond aging.

  • Problem

    Prior work describes luck and persistent advantage in creative careers, but whether this structure holds in open-source software and changes in the generative-AI era remains untested.

    Source: Introduction · pp. 1–2

  • Method

    The study reconstructs GitHub careers and tests hit timing, decomposes impact persistence, and compares the same contributors across eras against an aging placebo.

    Source: Introduction · p. 2, Experimental Design · p. 4, RQ1: the lottery test · p. 4

  • Results

    The biggest-hit timing is close to a lottery with a slight early lean, impact persistence includes a stable factor and momentum, and the stable-share era contrast is centered on zero (+0.035).

    Source: Discussion · pp. 8–9

  • Takeaways & Limitations

    Open-platform profiles are better read through consistency across many works than through one hit’s size or recency.

    Source: Discussion · p. 9

    Claims cover package contributors, and dependency reach misses impact from applications, documentation, tooling, issue work, and testing.

    Source: Introduction · p. 2, Discussion · p. 9

Abstract

from arXiv · show

Studies of careers in science, film, music, and books report a common pattern. When a person's most successful work arrives is close to a random draw over the works they produce. How large their successes tend to be, in contrast, follows a stable, person-specific factor. We test whether this pattern holds for open-source software careers and whether it changed when generative AI coding tools arrived. From the complete public record of GitHub push events (2015-2025), we reconstruct 102.2M career works by 6.15M contributors, and for the 908k contributors whose repositories publish packages, we measure each work's impact by how many downstream packages come to depend on it. First, we find that the timing of a career's biggest hit is close to a lottery over their works, as in science and the arts, with a small, replicable lean toward early career that grows as careers get longer. Second, some coders reliably produce higher-impact work than others, but this lasting personal factor accounts for only part of why impact persists (about a fifth in our primary specification); the rest behaves like momentum, success feeding on itself for a period of time. Third, within the same contributors, this structure did not change after ChatGPT's release. The stable factor's weight grew by about as much as it grew for an earlier cohort that simply aged, and subtracting the effect of aging from the effect of generative AI puts the shift at +0.03 (95% CI [-0.22, +0.23]), indistinguishable from zero. The success pattern documented in science and the arts therefore describes open-source careers too, and it shows no detectable break across the arrival of generative AI. These results have implications for how track records on open platforms should be read and on what to expect from generative AI for the careers built on them.

Introduction

Earlier research finds that career hits arrive at nearly random points while a persistent person-specific factor helps determine their size; this study asks whether that pattern holds for package-based open-source careers and changed in the generative-AI era. It treats that factor as positional advantage rather than pure ability and preregisters its analyses.

Introduction

  • Open-source careers differ from science and the arts because credit is shared, impact spreads through package dependencies, and platform visibility shapes judgments.The study therefore focuses on contributors who publish packages, the segment where impact can be measured through dependencies.Source: Introduction · pp. 1–2
  • Generative AI motivates opposing predictions: it could compress individual differences or amplify the advantage of people already well positioned.Prior task-level studies do not establish whether the same people still succeed or how much success depends on chance.Source: Introduction · p. 2
  • The study asks whether biggest-hit timing is a lottery and whether impact persistence reflects a stable individual factor or cumulative advantage in open-source package careers.It also tests whether this structure changed within the same contributors in the generative-AI era.Source: Introduction · p. 2
  • The stable factor is interpreted as positional advantage, combining skill, accumulated visibility, and platform position rather than measuring ability alone.This is the more conservative interpretation given that GitHub judgments depend on visible identity and geography.Source: Introduction · p. 2
  • The analysis preregisters its choices, reports the full specification grid with multiple-testing correction, and reserves a held-out half for confirmation.Its contribution is a large-scale test of the random-impact rule and Q-model decomposition, including a placebo-controlled within-person comparison across the AI-era divide.Source: Introduction · p. 2

Related Work

Research on creative careers separates random hit timing from the persistent factor associated with success size, but has not tested that structure in software careers. Related work on GitHub evaluation and generative AI motivates the paper’s measures and its within-person era comparison.

Success in creative careers

  • The random-impact rule and Q-model describe biggest-hit timing as random and success size as work-level luck combined with a stable person-specific factor.This structure has been studied in science and other creative fields, but not software careers.Source: Success in creative careers · p. 2

Open-source careers and platform evaluation

  • GitHub activity is read as a public résumé, while evaluations vary with author identity and geography; prior work also traces impact through package dependencies.Existing studies do not measure how much an individual career reflects luck versus a stable personal factor.Source: Open-source careers and platform… · p. 3

Generative AI and work

  • Prior generative-AI studies measure task outcomes over weeks or months, leaving open whether the division of career success across people changed.The paper addresses this gap by comparing the same people before and after a fixed calendar date.Source: Generative AI and work · p. 3

Data and Measurement

The study reconstructs GitHub careers, resolves contributor identities, and measures package impact through dependency data. Its preregistered, held-out design tests a fixed primary condition alongside a broader specification grid.

Data sources and coverage

  • GitHub Archive provides push records through September 2025, while package-dependency snapshots leave a 2020–2022 gap bridged by interpolation and checked in sensitivity analyses.The GitHub record’s public commit-level author information was removed in October 2025, limiting later reconstruction.Source: Data sources and coverage · p. 3

Github identity resolution

  • A hash dictionary and an 80%-commit backup rule assign commit authors to accounts, followed by bot filtering and validation against known identities.Assignment precision is 1.000 for 2015–19 and at least 0.98 in every later era and region; unresolved identities reduce coverage rather than correctness.Source: Github identity resolution · p. 3

Measuring works, credit and impact

  • A work is one person’s contribution to one repository; impact counts downstream package dependencies, log-scaled and centered against same-year, same-ecosystem works.R-share is the primary credit rule, with R-lead and R-core as alternatives; impact is measured at 24, 36, or 60 months.Source: Measuring works, credit and impact · pp. 3–4

Experimental Design

  • The reconstructed record contains 102.2M works by 6.15M contributors, while dependency-based estimates cover 2.44M works by 908,417 package contributors.The primary condition is R-share, W=60, at least five works, and reach2; contributors are split into discovery and held-out halves, with confirmatory numbers drawn from the latter.Source: Measuring works, credit and impact · p. 4, Experimental Design · p. 4

Methods

The methods test whether biggest hits occur at lottery-predicted career positions, separate persistent impact correlation from fading momentum, and compare career changes across the generative-AI divide against an aging placebo.

RQ1: the lottery test

  • For each career, the lottery test compares the normalized position of its highest-impact work with a uniform draw over that career’s works.It summarizes departures using mean position, total variation distance, and permutation tests; careers whose works all tie at zero impact are excluded.Source: RQ1: the lottery test · p. 4

RQ2: stable-factor decomposition

  • The decomposition fits impact correlations across works at different career distances as a stable plateau plus a decaying momentum term.A plateau counts as genuine only if it clears size, persistence-ratio, and within-career spread checks; validation also uses synthetic careers, split halves, and agreement across impact measures.Source: RQ2: stable-factor decomposition · pp. 4–5

RQ3: era-comparison design

  • The era comparison follows incumbents across calendar periods, uses equal impact horizons, and compares the real divide with an earlier aging-only placebo.Works too recent to be fully measured are dropped; a matched-censoring placebo and direct real-minus-placebo contrast test whether the stable-share shift differs from aging.Source: RQ3: era-comparison design · pp. 5–6

Results

Biggest-hit timing is close to a lottery, with a small early-career lean; impact persistence reflects both a stable person-specific factor and momentum. Within contributors, the stable factor showed no detectable AI-era shift beyond career aging.

RQ1: Hit timing is close to a lottery

  • Biggest-hit timing is close to a lottery: the primary-condition mean position is 0.548 versus 0.562 expected, with a slight early lean and small departures across all 54 conditions.The primary-condition decile view shows points slightly above the lottery prediction early and at or below it late; median TVD is 0.028 in discovery and 0.029 held-out.Source: RQ1: Hit timing is close to a lottery · p. 6, Figure 2 · p. 6, Figure 3 · p. 7
  • The early lean grows with career length: median TVD rises from 0.019 at a ≥3-work floor to 0.041 at ≥10 works, while the median early shift roughly doubles.The pattern appears under every credit rule and both impact measures; the authors suggest accumulated early attention or a less crowded early ecosystem as possible explanations.Source: RQ1: Hit timing is close to a lottery · p. 6

RQ2: A stable factor, but a minority share

  • At the primary condition, the stable factor accounts for 0.189 of impact persistence, while the stricter three-criterion verdict falls short in both halves.The stable factor is above zero in all 54 discovery and held-out conditions, but its share varies with data and calendar handling; zero-impact works may pull estimates down, while Φ also includes platform position.Source: RQ2: A stable factor, but a minority… · pp. 6–8
  • Excluding interpolated measurements raises the persistence ratio to 1.00 in discovery and 0.346 held-out, indicating that interpolation inflates estimated momentum.The held-out re-run still misses the verdict at the variance-growth limit, 1.53 against the 1.5 limit; the lottery result does not move.Source: RQ2: A stable factor, but a minority… · p. 7
  • The estimated stable share is larger for deeper dependency reach and proportional credit, while per-person effects under reach2 and in-degree correlate at Spearman ρ=0.979.Longer careers measure the stable factor more reliably: split-half consistency rises from r=0.29 at ≥3 works to r=0.53 at ≥10, while median A barely moves.Source: RQ2: A stable factor, but a minority… · p. 7

RQ3: No detectable break in the generative-AI era

  • The stable factor’s share rose by +0.035 after subtracting the placebo cohort’s aging-related rise, with a 95% CI of [−0.218, +0.230] that covers zero.All 72 real-minus-placebo confidence intervals cover zero; the design cannot rule out smaller era shifts.Source: RQ3: No detectable break in the… · p. 8
  • Biggest-hit timing remains lottery-like across eras, and per-coder effect rankings carry across the real divide about as well as across the placebo divide (held-out rank ρ=0.30 vs. 0.40).Matching post-era works to the same career stage as late pre-era works removes the stable-share rise, ∆A=+0.000 [−0.247, +0.271]; recent works are also censored before they can accumulate impact.Source: RQ3: No detectable break in the… · p. 8

Discussion

Open-source package careers largely follow the success pattern found in science and the arts: hit timing is near-random, while impact reflects both a stable personal factor and momentum. The discussion also finds no detectable generative-AI-era break beyond aging, while noting boundaries on what the data can establish.

Discussion

  • Biggest hits are close to lottery-timed, with a small early-career lean that grows in longer careers.The authors suggest extra attention to early works through repository lifecycles and ecosystem entry timing as a mechanism for future work to test.Source: Discussion · p. 8
  • A real individual signal explains only about a fifth of impact persistence; the rest is fading momentum.The signal is strongest for dependency spread and shared credit, and split-half consistency roughly doubles from short to long careers even though the stable share does not change.Source: Discussion · p. 8
  • Profile readers should weigh consistency across many works rather than the size or recency of one hit.The authors caution that inferring ability from hit timing is close to reading a lottery ticket.Source: Discussion · p. 9
  • The stable share’s rise after the AI-era divide is comparable to aging alone, and the direct contrast includes zero.The apparent rise disappears with equalized censoring or comparisons at the same career position; Figure 5 reports D=+0.035 [−0.218, +0.230].Source: Discussion · p. 9, Figure 5 · p. 10
  • The evidence covers package contributors, and dependency reach omits several forms of software work; AI-era comparisons use calendar periods rather than measured adoption.Package contributors represent about 2% of reconstructed works; the AI comparison is limited to incumbents active into the post-era, with a short, right-censored window.Source: Discussion · p. 9

Ethics Statement

The study analyzes public platform data without intervention or private-data access, and reports aggregate results while protecting individual identities.

Ethics Statement

  • The study is observational secondary analysis of public platform data, with no intervention or private data accessed.Under the institution’s policy, it was not human-subjects research and required no IRB review.Source: Ethics Statement · p. 11
  • Identity resolution uses public platform identities, with uncertain matches left unresolved rather than misattributed.Author emails are hashed, and the resolution process is validated for precision.Source: Ethics Statement · p. 11
  • Outputs are aggregate and example careers anonymized; no individual rankings, fault tables, or personal-attribute classifier are released.The paper also states that platform terms and data licenses are respected.Source: Ethics Statement · p. 11

Appendix Overview

The appendix provides the full specification and sensitivity materials, with analysis code and aggregate result files available publicly.

Appendix Overview

  • The appendix reports the full pre-registered specification grid in both halves, without omitting conditions.It also includes the full pre/post grid and the calendar-gap sensitivity analysis.Source: Appendix Overview · pp. 11–12
  • Validation and provenance details summarized in the main text are reported in the appendix.The calendar-gap sensitivity is also reported in full.Source: Appendix Overview · p. 12
  • Analysis code and aggregate result files behind every figure and table are publicly available.Tables are generated programmatically from sealed result files rather than transcribed by hand.Source: Appendix Overview · p. 12

Identity-Resolution Validation Detail

Identity resolution combines exact decoding of GitHub no-reply hashes with a filtered fallback assignment, and validation shows high precision across eras and regions.

Identity-Resolution Validation Detail

Estimator Validation Suite

The estimator suite tested whether its methods could distinguish genuine stable effects from processes that can mimic them. Synthetic careers also exposed a spurious early-lean artifact, motivating randomized tie-breaking and exclusion of all-tie careers.

Estimator validation

  • Both estimators were tested on nine synthetic career regimes with twenty pass/fail checks before real data analysis.The adversarial suite required rejecting misleading apparent stable factors while affirming a genuine one.Source: Estimator Validation Suite · p. 12
  • The tests included cumulative advantage, near-unit-root autocorrelation, drifting person effects, measurement error, known stable factors, and zero-inflated impacts.The estimators were required to return A≈0 for pure cumulative advantage and recover a genuine stable factor within its bootstrap CI.Source: Estimator Validation Suite · p. 12
  • Synthetic lottery tests showed that deterministic tie-breaking on zero-heavy proxies can manufacture an early lean, so ties are randomized and all-tie careers excluded.That artifact dictated the tie-handling rule frozen in the registration.Source: Estimator Validation Suite · p. 12

Full Specification-Grid Results

Across the live pre-registered grid, the exact lottery null is rejected in every condition, in both discovery and held-out halves. The tables also summarize stable-factor verdicts and condition-level moderators.

Full specification grid

  • Every live condition rejects the exact lottery null at BH-adjusted permutation p = 6.7 × 10−4.The result covers all reported conditions in the discovery and held-out halves.Source: Full Specification-Grid Results · p. 12
  • The 54-condition grid crosses three credit rules, three horizons, three career-length floors, and two impact measures.W=120 conditions are empty at the calendar close because every horizon is right-censored.Source: Full Specification-Grid Results · p. 12
  • Tables report sample size, lottery departure, stable share and bootstrap CI, persistence ratio, stationarity, and the full pre-registered verdict for each condition.Moderator medians are computed over these rows.Source: Full Specification-Grid Results · p. 12

Pipeline-Replacement Robustness

Replacing the measurement pipeline leaves the principal findings essentially unchanged, while pre/post comparisons do not establish a generative-AI-era shift beyond aging. Calendar sensitivities strengthen the stable-share estimates, but their magnitude is less securely identified; several analyses are explicitly post-unsealing.

Pipeline-Replacement Robustness

  • Pipeline replacement changes median stable share only from 0.170 under v1 to 0.172 under v2, while the verdict count shifts from 37/54 to 36/54.Median TVD falls from 0.043 to 0.028, with median per-condition ΔTVD −0.010; median per-condition ΔA is +0.008.Source: Pipeline-Replacement Robustness · p. 12

Full Pre/Post Grid

  • At W=36, 92–93% censoring accompanies sign-unstable real-divide ΔA, whereas W=24 and placebo conditions behave stably.The figure attributes the W=36 instability to censoring rather than an era effect.Source: Full Pre/Post Grid · p. 13, Figure A5 · p. 16
  • +0.035 [−0.218, +0.230] is the held-out primary-condition difference-in-differences estimate for stable share, and all 72 condition-level CIs cover zero.The ±0.10 equivalence test passes in 0 of 72 conditions, so the data rule out large era shifts but cannot certify that small ones are absent; matched censoring leaves no residual era effect.Source: Full Pre/Post Grid · p. 13
  • Career-age stratification leaves all 288 era-condition TVDs unchanged within |Δ| < 10−4 and changes no FDR conclusion.Hit position correlates only weakly with career age at era entry; primary-condition ρ ranges from −0.16 to +0.10 across eras and halves.Source: Full Pre/Post Grid · p. 14
  • Per-coder effects from reach2 and in-degree correlate at Spearman ρ=0.979 in both halves, above the registered 0.7 threshold for treating the proxies as one factor.The sample sizes are 14,091 in discovery and 13,957 held-out.Source: Full Pre/Post Grid · p. 15

Calendar-Gap Sensitivity in Full

  • In discovery, calendar-gap sensitivities yield A=0.310 with B=0 and ratio 1.00 in the primary or primary-adjacent condition, reversing the main-calendar verdict failure.In held-out runs, both variants again raise the ratio above the 0.20 threshold, but the full discovery reversal does not replicate and the composite verdict still fails the stationarity limit.Source: Calendar-Gap Sensitivity in Full · p. 15

Preregistration Timeline and Deviations

  • The consolidation dropped several registered secondary specifications, and the primary normalization stratum was changed because language recovery was unavailable at scale.The primary proxy, transitive downstream reach, was operationalized as its two-hop truncation, reach2; two registered channels were deferred in full.Source: Preregistration Timeline and Deviations · p. 16
  • The career-clock, career-age, and cross-proxy analyses were specified in a dated post-unsealing note and are labeled post-unsealing rather than sealed confirmation.The direct era-contrast and matched-censoring tests were also specified after the unseal, before their numbers were computed.Source: Preregistration Timeline and Deviations · pp. 16–17
Loading 2609.09687v1…