PaperlayerView exact revision on arXiv

Source-linked AI summary

The Software Supply Chain as a Market for Lemons: A Multivocal Review of Trust Signal Collapse

Ranindya Paramitha, Christian Kästner, Laurie Williams

arXiv:2608.20678v1cs.CRcs.SE

TL;DR

Practitioners rely on cheap signals to assess open-source dependencies, but the landscape of signal collapse and the ecosystem’s response had not been examined across signals. This paper reviews Google Search and Reddit sources, finding that cheap signals collapse under multiple forces and documented responses are more often advice than action.

  • Problem

    The landscape of collapse across dependency-adoption trust signals and the ecosystem’s response remained unexplored, despite practitioners’ reliance on such signals to assess trustworthiness.

    Source: §1 · p. 1

  • Method

    A multivocal review coded 252 Google Search sources and 870 Reddit threads covering eight dependency-adoption trust signals.

    Source: §1 · p. 1

  • Results

    Cheap trust signals collapse through adversarial, ambiguous, and non-adversarial forces; 54.6% of Google Search sources offer recommendations but describe no practitioner action.

    Source: §1 · p. 2, §4 · p. 5

  • Takeaways & Limitations

    The authors conclude that signal substitution and manipulation detection cannot fully break the cycle, while structurally sound alternatives are known but not adopted at scale.

    Source: §7 · p. 9

    The Google Search corpus is bounded by its retrieval strategy, underrepresenting non-English practitioner communities and private organizational grey literature.

    Source: §6 · p. 8

Abstract

from arXiv · show

Practitioners evaluating open-source dependencies rely on cheap trust signals, e.g., stars, download counts, and contributor activity, as substitutes for direct code inspection, assuming those signals reflect genuine trustworthiness. Prior work has documented individual signal gaming, but the landscape of collapses across all dependency-adoption signals, as well as the ecosystem's response, remains unexplored. The goal of this study is to aid software practitioners in understanding the reliability of dependency adoption trust signals, such as download counts and contributor activity, by conducting a multivocal review of 252 Google Search sources and 870 Reddit threads. After coding the corpora, we find that cheap trust signals collapse under three simultaneous forces: adversarial manipulation, gaming techniques indistinguishable from legitimate behavior, and non-adversarial AI-driven inflation. The documented responses are more advice than actual action: 54.6% of Google Search sources contain advice on what practitioners should do, with no actual action taken. Responses proposed substituting one cheap signal for another or aggregating multiple signals, which are now also gameable. Non-adversarial inflation, i.e., degradation caused by the emergence of legitimate AI tooling, lacks documented actual behavior change in either corpus. The gap between known remedy and actual practice points toward a market for lemons: when faking signals costs less than earning them, good and bad dependencies become indistinguishable. Relying on individual practitioners to verify the cheap signals is not sustainable. Costlier signals, such as cryptographic attestation, should be made mandatory so that they become the default for all, not a voluntary choice for the few.

1 INTRODUCTION

This study reviews how dependency-adoption trust signals collapse and how the ecosystem responds, finding that signal production has become cheap enough to make trustworthy and untrustworthy dependencies difficult to distinguish. It combines Google Search and Reddit sources to examine manipulation mechanisms, response patterns, and changes in reliance on signals.

Introduction

  • AI makes convincing stars, commit histories, documentation, and endorsements cheap to produce, threatening a market for lemons where developers cannot distinguish trustworthy from untrustworthy packages.Prior work documented individual manipulations, including fake stars, inflated downloads, injected documentation, and maintainer social engineering, but not a systematic review across signals and responses.Source: §1 · p. 1
  • The multivocal review analyzes 252 Google Search sources and 870 Reddit threads across eight dependency-adoption signals, using coding to examine collapse mechanisms and ecosystem responses.The signals include stars, downloads, documentation, contributor activity, badges, issue responses, and community endorsement.Source: §1 · pp. 1–2
  • Faking now costs less than earning, while legitimate AI tools, attackers, and honest actors can produce identical signal outputs, making cheap-signal collapse structural rather than merely a detection problem.Sophisticated attacks can reconstruct the entire trust envelope at once, rendering composite evaluation ineffective.Source: §1 · p. 2
  • 54.6% of Google Search sources offer recommendations without describing practitioner action; proposed substitutes are mostly gameable cheap signals, and legitimate-AI inflation lacks documented behavior-changing responses.The documented response is therefore predominantly advice without action.Source: §1 · p. 2
  • The study contributes ten manipulation-mechanism categories across eight signals, distinguishes adversarial, ambiguous, and non-adversarial degradation, and provides a codebook, annotated dataset, and role-based recommendations.Manual coding reliability exceeded 70% agreement across RQ1 fields on a 40-source sample; LLM-assisted thematic coding separately analyzed both corpora.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORKS

This background frames dependency trust signals as proxies whose informativeness depends on cost separation, then positions the study as a cross-signal synthesis of manipulation and ecosystem responses using grey-literature-inclusive review methods.

2.1 Trust and Signals in the Software Supply Chain

  • Because applications contain hundreds of transitive dependencies, developers use observable adoption and health signals—including stars, downloads, contributor activity, maintenance cadence, documentation, and community recommendations—as trust proxies [14].A signal informs users when its production cost separates high- from low-quality dependencies; when that condition fails, the signal pools, Goodhart’s Law makes it worth gaming, and adverse selection can produce a market for lemons.Source: §2.1 · p. 2

2.2 Signal Gaming in the Software Supply Chain

  • Prior studies isolate manipulation of stars, downloads [40], README legitimacy, maintainer status, or AI retrieval via planted UGC [46]; this review synthesizes mechanisms, intentions, and ecosystem responses.Each prior study examined a single signal, mechanism, or incident.Source: §2.2 · pp. 2–3

2.3 Ecosystem Responses of Signal Manipulation

  • Prior adoption studies do not examine responses after signal gaming is disclosed, artifact-attack taxonomies omit perception-targeting manipulation, and platform responses remain isolated case studies [14] [26].Whether responses collectively address signal collapse, and how widely they are adopted or advised, remains open.Source: §2.3 · p. 3

2.4 Multivocal Reviews in Software Engineering

  • Because trust-signal incidents and responses appear predominantly in grey literature, this MLR combines 252 Google Search sources and 870 Reddit threads across eight signal categories.Prior software-engineering MLRs establish grey literature as a valid and necessary source for practitioner knowledge.Source: §2.4 · p. 3

3 METHODOLOGY

The study uses a multivocal literature review to capture practitioner responses across formal and grey literature, then analyzes signal manipulation and ecosystem responses through structured and thematic coding.

3 METHODOLOGY

  • The MLR included academic and practitioner sources because practitioner responses are documented predominantly outside academic venues.Grey literature can document fake-star service pricing, practitioner accounts of signal gaming, and platform responses.Source: §3 · p. 3

3.1 Corpus Construction

  • The review scoped eight dependency-adoption trust signals [14] and treated AI-driven activity inflation separately from deliberate activity spoofing.Search vocabulary was developed through an LLM-assisted exploratory pre-search; anchor queries targeted known incidents or papers, while seed queries broadly searched signal concepts.Source: §3.1 · p. 3
  • 321 candidate sources were retrieved through Google Search, supplemented by practitioner-platform browsing, snowballing, and vendor and institutional reports.Searches ran July 1–19, 2026; all first-page links from up to three anchor and seed queries per signal were collected.Source: §3.1 · p. 3
  • Sources had to discuss a scoped signal and be publicly accessible in English; 252 Google sources and 870 Reddit threads formed the final corpora.The Reddit retrieval contained 95,190 threads from 2023–2026; 870 relevant threads were selected with high LLM confidence and a post score above zero. The 252 Google sources spanned eight signals, with 54 on S3b and 14 on S2; 152 (60.3%) were directly supply-chain-related, and 100 (39.7%) covered adjacent contexts. Source tiers were 49 Tier 1, 63 Tier 2, and 137 Tier 3; S2’s small count was noted as a corpus limitation.Source: §3.1 · pp. 3–4

3.2 Analysis

  • The two-pass Google coding scheme captured manipulation mechanisms and intent for RQ1, and response type, trust-direction changes, and proposed alternatives for RQ2.Two coders assessed a balanced 40-source sample manually and with an LLM; after calibration, the LLM coded the remaining RQ1 sources, while the first author coded the remaining RQ2 sources.Source: §3.2 · p. 4
  • All RQ1 closed-code fields exceeded 70% agreement after three calibration rounds, while RQ2 trust-direction coding remained manual because LLM agreement was below 70%.The study separately applied an LLM-assisted thematic coding tool to both corpora, using coder, aggregator, and reviewer agents to develop codebooks with supporting quotes.Source: §3.2 · p. 4

4 RQ1 RESULTS: SIGNAL COLLAPSE MECHANISMS

The sources document trust-signal collapse through cheap adversarial manipulation, behavior indistinguishable from legitimate activity, and inflation caused by legitimate AI tooling. Sophisticated attacks also reconstruct a trustworthy appearance across multiple signals.

Mechanism landscape

  • Of 252 sources, 167 (66.3%) documented specific mechanisms: stars were predominantly bot-gamed, while activity, documentation, and community endorsement faced varied forms of manipulation or AI use.MT-BOT appeared in 31 of 32 Google Search sources documenting star manipulation; activity-legitimate, badges, and issue-response were documented mostly outside software supply chain-specific literature, despite prior evidence that developers use them when choosing dependencies [14].Source: §4 · pp. 4–5, Table 3 · p. 5

Finding #1

  • Adversarial manipulation has become cheap: stars cost $0.03–$0.85, downloads are inflated through stolen credentials and install loops, and a 2-month campaign used 270+ commits to mimic development.One source documented 63,872 suspected fake accounts in a single campaign; AI can also generate documentation in seconds.Source: §4 · pp. 5–6
  • 38.3% of documented mechanisms used behavior structurally indistinguishable from legitimate practice, including soliciting stars, optimizing for AI retrieval, and generating documentation with AI.Because these actions are also performed by honest project owners, the signal itself does not reveal motivation.Source: §4 · p. 6
  • 14.4% of Google Search sources documented non-adversarial inflation; AI coding agents generate 275 million commits per week, with commits rising 180% while releases rose 30%.This legitimate tooling inflates activity signals, and the sources also describe badges and issue-response signals collapsing without an attacker.Source: §4 · p. 6
  • 52.7% of documented Google Search manipulation was not meaningfully detectable because cheap attacks, legitimate behavior, and non-adversarial AI inflation can produce the same observable signals.The sources attribute this detection gap to signals leaving no trace of the actor’s motivation.Source: §4 · pp. 5–6

Finding #2

  • Sophisticated attacks reconstruct a trustworthy project across multiple signals, making composite checks ineffective rather than manipulating only one signal.GitVenom combined fabricated README files, spoofed commit histories, and package injection; the ESLint compromise used stolen credentials, phishing-harvested identity, and malicious publication; xz built genuine commit activity and community participation before its payload. The May 2026 Fedora incident used compromised credentials and an AI agent across activity, issue response, and documentation; mobile SDKs likewise backdate commits and fake stars and forks to gain recommendations.Source: §4 · p. 6

5 RQ2 RESULTS: RESPONSES TO SIGNAL COLLAPSES

Documented responses to trust-signal collapse are mostly advice, with practitioners more often downweighting signals than abandoning them. Proposed substitutes and aggregation remain vulnerable, while responses to legitimate AI-driven inflation rarely document behavior change.

Finding #3

  • 54.6% of Google Search sources contain advice without enacted action, while only 18.4% document actions actually carried out.Advice spans composite evaluation, tooling, process controls, and platform escalation; 27.0% of Google Search sources are analyst-inferred responses.Source: §5 · pp. 6–7

Finding #4

  • 55% of Google Search sources with a documented trust direction record signal downweighting, versus 23.3% replacement and 1.2% abandonment.Responses generally retain signals but add verification, increasing evaluation costs; direct inspection, maintainer-history audits, and sandboxing remain minority practices.Source: §5 · p. 7

Finding #5

  • 49 of 163 Google Search sources propose alternative signals, and 65.3% of those alternatives are cheap, observable metrics vulnerable to similar gaming.Aggregation may raise manipulation costs only temporarily because coordinated campaigns can reproduce multiple signals; 34.7% propose cryptographic alternatives, while commit signing appears in 8 sources but is adopted by only 10% of GitHub repositories.Source: §5 · pp. 7–8

Finding #6

  • Among 24 Google Search sources classified as non-adversarial (MA-NON), 33% had no documented response; of those with responses, only 3 documented actual behavior change.Source: §5 · p. 8

6 LIMITATIONS

The findings are bounded by corpus retrieval and coverage, coding and source subjectivity, and the distinction between documented responses and observed ecosystem-wide behavior. Some signal-specific results are less generalizable because those signals are thinly represented.

Corpus completeness and subjectivity

  • The Google Search corpus excludes sources outside queried or indexed platforms, underrepresents non-English communities and private organizational grey literature, and ends July 19th, 2026.Source: §6 · p. 8
  • Reddit coverage is limited to 2023–2026 tool-indexed communities; LLM relevance filtering excluded borderline threads, while deleted, private, and non-English threads are absent.The corpus consists predominantly of experiential and self-reported accounts shaped by source framing and incentives; coding records reported mechanisms and responses rather than independently verified ground truth.Source: §6 · p. 8

Signal coverage

  • Google Search signal-specific findings for S2:documentation (n=14) and S4:platform-awarded-achievements (n=11) are less generalizable than findings for better-evidenced signals.Source: §6 · p. 8

Coding reliability

  • All RQ1 closed-code fields exceeded 70% agreement, but Reddit codes lacked manual validation; only included Reddit IDs and quotes were manually checked for relevance.Agreement was assessed against LLM codings on a 40-source sample after three prompt-calibration rounds; the single RQ2 trust_direction judgment was subjective, though its decision space was narrow.Source: §6 · p. 8

Construct validity and Generalisability

  • Findings describe documented responses across eight selected signals, not observed behavior or population distributions; Google percentages and counts characterize that corpus, not ecosystem-wide prevalence.Other observable signals, including license type, organizational affiliation, and vulnerability scan results, were outside scope; tool names, prices, and adoption rates may become outdated as the landscape evolves.Source: §6 · p. 8

7 DISCUSSIONS, IMPLICATIONS, AND CONCLUSIONS

The discussion argues that cheap trust signals are collapsing into a market for lemons: advice and substitutes have not restored trust, while costly alternatives remain underadopted. It concludes that restoring safe, low-cost reuse requires institutional change rather than relying on individual verification.

7.1 Why the Current Trajectory Is Unsustainable

  • 55.2% of documented Google Search trust directions downweight signals, while 12.9% reaffirm them, reflecting different awareness stages amid rising verification costs.The discussion describes practitioners spending more time evaluating dependencies than writing code, while some organizations maintain costly internally audited mirrors. PyPI’s removal of download-count displays is the only documented platform retirement of a gameable signal; switching costs also limit individual exits.Source: §7.1 · p. 9

7.2 Why Signal Substitution and Aggregation Cannot Break the Cycle

  • Substitution and aggregation cannot break the cycle because cheap alternatives are gameable, and patient attackers can manufacture multiple signals simultaneously.Shared interests and genuinely costly signals can resist manipulation, but anonymous, transitive dependency graphs weaken shared incentives; Goodhart’s Law helps explain why targeted substitutes become worth faking. The discussion identifies cryptographic signatures and reproducible builds as alternatives causally bound to what they certify, and recommends commit signing, SBOM, provenance attestation, or accountable maintainer relationships.Source: §7.2 · pp. 9–10

7.3 Why Detection Cannot Break the Cycle Either

  • 38.3% of documented mechanisms are indistinguishable from legitimate behavior (MA-AMB), while 14.4% involve no adversarial actor (MA-NON), limiting manipulation-focused detection.The discussion proposes measuring signal reliability regardless of intent and transparent platform labeling of AI activity; it reports 97.2% F1 attribution across five major agents, but no registry or code-hosting platform systematically discloses AI contribution rates. MA-NON appears in 40 Google Search sources and has only 6 described behavior changes, all individual adaptations; future research should examine practitioner responses to AI-inflated platform-awarded achievements and issue-response signals.Source: §7.3 · p. 10

7.4 Why the Known Remedies are Not Being Adopted

  • 34.7% of proposed responses mention structurally sound alternatives, but adoption is only 10% of repositories, leaving a two-tier market rather than restoring shared trust.Attestation’s value depends on its absence becoming a red flag, so early adopters bear costs before ecosystem-wide benefits accrue; the discussion argues recommendations alone cannot change incentives. It proposes making signed provenance a platform baseline, such as a publication or inclusion requirement for packages above a usage threshold, rather than voluntary opt-in.Source: §7.4 · pp. 10–11

7.5 To Conclude: Where the Ecosystem Stands

  • The conclusion calls for shifting trust from cheap signals toward cryptographic indices, institutional attestation, and accountable identity, backed by institutional enforcement.Without that shift, verification overhead will continue to accumulate around gameable signals, threatening the low-cost trust advantage of open-source reuse.Source: §7.5 · p. 11

Related papers

Loading 2608.20678v1…