PaperlayerView exact revision on arXiv

Source-linked AI summary

Do AI Coding Assistants Check Before They Install? A Pre-Registered Demand-Side Audit of Trust Signals in the Research Software Supply Chain

Pengyin Shan

arXiv:2609.07754v1cs.CRcs.AIcs.SE

TL;DR

Whether coding assistants read positive supply-chain trust signals before installing research software has not been measured. The authors preregistered a controlled study across six projects and found that assistants rarely opened signals and never verified them. They conclude that publishing signals alone is insufficient and that verification needs to be built into the assistant-running program.

  • Problem

    Whether assistants read or act on positive supply-chain signals at install time, particularly for research software, had not been measured.

    Source: §1 · p. 2

  • Method

    A preregistered study tested three models across six research-software projects and recorded signal access and verification from container logs.

    Source: §1 · pp. 1–2, §3.5 · p. 6

  • Results

    9 of 1,920 registered trials opened a provenance signal before installation, and none ran a verification command.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Publishing signals is necessary but not sufficient; the authors conclude that verification must be an explicit step in the program running the assistant.

    Source: §5 · p. 14

    The six-project controlled study describes behavior on research software, not the wider ecosystem, and most installation commands failed during dependency resolution.

    Source: §6 · p. 16

Abstract

from arXiv · show

AI coding assistants now select, install, and configure software, and attackers have exploited that position through invented package names, compromised maintainer accounts, and manipulated repository text. In response, the supply-chain community publishes machine-checkable trust signals: software bills of materials, signed releases, build provenance attestations, and declared official channels. Whether coding assistants read or act on those signals has not been measured for any of these classes on research software. We pre-registered and ran a controlled study on six open-source research software projects (three HPC, three quantum computing) drawn from an 87-project corpus, with protocol, seed, panel, and analysis plan deposited with a DOI before any trial. W created nine modified copies for each project: no signal, one per signal class, two with a signature or attestation from the wrong issuer, one with all four signals, and one reproducing documented conflicts in the project's own metadata. Three models under two ways of operating an assistant, with and without an approval step, gave 1,920 registered trials, plus a supplement on three frontier models. We scored behavior from container logs rather than from what the assistant said, and recorded the cost of every trial. Verification was rare under every condition: in 9 of 1,920 registered trials (0.5%), the assistant opened any provenance signal before installing in 0 of 384 control trials, and no trial ran a verification command, so signal presence had no measurable effect. We drew three conclusions: publishing signals is necessary but not sufficient; price did not buy verification (the model that verified most often costs $0.10 per trial; the most capable, at $1.00, verified nothing); verification must be built into the program that runs the assistant. We release the per-trial cost ledger, the protocol, and every log.

1 Introduction

The paper asks whether assistants consume trust signals at install time, a demand-side gap distinct from whether projects publish them. It measures this behavior in a preregistered controlled audit and releases trial and cost artifacts.

1 Introduction

  • Across 1,920 pre-registered trials, assistants opened a signature, attestation, software bill of materials, or channel declaration before installation in 9 trials, but never checked or verified any signal.Source: §1 · pp. 1–2
  • Attackers have exploited assistants’ position in software installation through plausible nonexistent package names and compromised maintainer accounts.The introduction also situates these risks alongside policy and standards responses.Source: §1 · p. 2
  • The supply-side signals include signed releases, build-provenance attestations, SBOMs, and official-channel declarations, but prior audits found few such signals on research software.In a hand-verified 30-project set, 2 projects published signed releases, 1 had an attestation, and none had an SBOM; metadata disagreed in 83.9% of cross-checkable cases.Source: §1 · p. 2
  • Whether assistants read positive, defense-side signals at install time was unmeasured, despite prior work on attack cues and an observational pilot pointing to limited signal use.The paper frames this as the demand-side companion to research-software audits of published declarations.Source: §1 · p. 2
  • The study froze its protocol, screening rules, condition matrix, outcomes, and analysis plan and deposited them with a DOI before any trial.The controlled measurement covered six HPC and quantum-computing projects, nine conditions, two assistant-running approaches, and three models.Source: §1 · p. 2
  • Every trial has token counts, cost, and wall-clock time, and the released artifacts include the protocol, tooling, project copies or generators, transcripts, coding sheet, and cost ledger.The authors report cost per verification-positive trial for each model.Source: §1 · pp. 2–3

2 Related Work

Prior work examines install-time attack detection, defense-signal infrastructure and supply-side declarations, package choice, and evaluation methodology. This study targets the unmeasured response to positive supply-chain signals, especially for research software.

Install-time behavior of coding assistants

  • Studies of package hallucination and install-time attacks examine attack indicators, while work on denial signals and generic provenance cues addresses different settings; positive defense-side signals on research software remained unmeasured.The paper’s outcome categories build on [12], and its single-signal conditions follow the ablation approach of.Source: §2 · p. 3

Defense-signal infrastructure and its supply side

  • Supply-side tools define the injected signals, while research-software audits and package-selection studies address published practices or package choice rather than verification of a selected package.The paper cites the research-software supply-chain taxonomy [19] and the author’s corpus measurements [11] and observational pilot.Source: §2 · p. 3

Evaluation methodology

  • Because assistant explanations can misrepresent process, the study scores end states and instrumented file-access and command logs, releasing instance-level outcomes and per-trial costs.Rationale text is coded separately as a secondary outcome.Source: §2 · p. 3

3 Methods

The methods pair preregistered procedures with a seeded, screened selection of six research-software projects from an 87-project corpus. The panel and local-source installation design define the scope of the controlled study.

3.1 Pre-registration

  • The protocol fixed the hypothesis, analyses, exclusion and stopping rules, and cost scheme before trials, with configuration and corpus checks tied to a deposited DOI.The originally registered local model was replaced before execution after unusable outputs in all 4 pilot trials and 13 diagnostic calls.Source: §3.1 · p. 3

3.2 Candidate selection

  • The panel was drawn from 87 projects—44 HPC and 43 quantum-computing—using a seeded ranking and five screening rules rather than hand selection.Rules covered repository resolution, source-checkout installability, project independence, exclusion of prior named examples, and human confirmation.Source: §3.2 · pp. 3–4
  • Source-checkout installation was allowed because local-copy installs were used and requiring registry listings would have left only four eligible HPC candidates for three places.This differs from the prior study of mainstream packages installed by name from a registry [12].Source: §3.2 · p. 4
  • The selected panel comprised faasm, mpi4py, envpool, qutip, covalent, and qrisp, with some metadata-conflict conditions inapplicable to projects lacking the relevant metadata.faasm had no Python package metadata, while covalent cited no article and faasm had no author list.Source: §3.2 · p. 4

3.3 Conditions

Each project received modified repository copies representing control, individual trust signals, issuer mismatches, combined signals, and documented metadata conflicts. The conditions test both signal presence and whether the material comes from a declared identity.

3.3 Conditions

  • Each project received nine modified repository copies: eight add a signal or signal combination, and one reproduces the project’s own metadata inconsistencies.The signals were implemented with release files, signatures, attestations, an SBOM, and a SECURITY.md channel declaration.Source: §3.3 · pp. 4–5
  • The condition matrix includes a shared unmodified control with double the trials, plus valid SBOM, signed-release, attestation, channel-declaration, and all-signals conditions.The SBOM is referenced from the README, and the signed release verifies against the identity declared in the copy’s metadata.Source: §3.3 · pp. 4–5
  • Signature and attestation issuer-mismatch conditions pair present cryptographic material with an identity other than the one declared in metadata.The injected material necessarily used the researchers’ own identity, allowing the design to measure whether an assistant distinguishes it from the project’s identity.Source: §3.3 · p. 5
  • The inconsistent-surface condition reproduces documented conflicts such as a cited article in place of software, a stale archive record, or an ambiguous author identity.These patterns came from the supply-side audit’s verification log [11], rather than being invented for the study.Source: §3.3 · p. 5

3.4 Harnesses, models, and trials

The study compared gated and autonomous assistant harnesses across three model arms selected for reproducibility, deployment realism, and cost accessibility. A separate frontier-model supplement tested control and all-signal conditions.

3.4 Harnesses, models, and trials

  • The gated harness requests approval before each command, whereas the autonomous harness runs planned commands directly.The comparison tests whether an approval step changes assistant behavior; approvals in the gated harness are automatic.Source: §3.4 · p. 5
  • The three main-study arms used qwen2.5-coder:14b locally, claude-sonnet-5, and gemini-3.5-flash-lite for reproducibility, deployment realism, and cost accessibility, respectively.The registered local model was replaced after gpt-oss:20b produced no usable output; the change was logged.Source: §3.4 · p. 5
  • The main study comprised 1,920 trials across six projects, nine conditions, two harnesses, and three models; the frontier supplement planned 216 trials across three models and two conditions.The supplement completed 194 trials before its registered spending cap stopped one model's execution.Source: §3.4 · p. 5

3.5 Environment

Each trial used an isolated project copy and instrumented container records to score assistant behavior at the decision point. The outcomes distinguish signal retrieval from verification and record practical limits on the environment and coding of assistant explanations.

3.5 Environment

  • The containers had no network access or package index, so installations needing unavailable dependencies failed during resolution; the study recorded whether an install was attempted, not whether it succeeded.Command exit status was reported separately.Source: §3.5 · pp. 5–6
  • Each trial used one modified project copy in a container, and Figure 2 traces the assistant loop, decision-point scoring, container records, and a cost-ledger row.The instrument records file opens, read actions, and commands rather than relying on assistant text.Source: Figure 2 · p. 6, §3.5 · p. 6
  • Trials were capped at 30 turns, 900 seconds per command, and 2,048 output tokens per turn; gated approvals were automatic, so the study did not test human refusals.Unreadable output after two consecutive turns was coded as malformed and retained.Source: §3.5 · p. 6

3.6 Outcomes

  • Verification required a logged verification command and result, while rationale coding was secondary and its agreement statistic was deferred to the next preprint version.The container records, not assistant claims, were used to check verification claims.Source: §3.5 · p. 6, §3.6 · pp. 6–7
  • The five end states describe whether assistants installed silently, remarked afterward, declined, retrieved a signal before installing, or ended incomplete; retrieval was recorded separately from the end state.The primary measure counted retrieval before the install-or-decline decision, while post-install comments and refusals without retrieval scored negative.Source: §3.6 · p. 6
  • Table 1 shows one cost-ledger row for a claude-sonnet-5 trial, with its cost derived from token counts at the frozen rate card.Cache-write tokens use the five-minute cache rate.Source: Table 1 · p. 7

3.7 Cost accounting

The study logged a fixed set of per-trial usage and timing fields, calculated costs from a pre-frozen provider rate card, and released the ledger for inspection.

3.7 Cost accounting

  • Each cost record includes model, condition, harness, project, trial number, timing, API calls, token categories, and computed US-dollar cost.The field set was fixed in the pre-registered configuration.Source: §3.7 · p. 7
  • Costs were computed from token counts using provider prices copied on 2026-08-22 and frozen with the protocol, then compared with provider billing-console totals without adjusting the ledger.The local arm had no per-call cost; its energy use was not measured.Source: §3.7 · p. 7
  • The pre-specified summaries were cost per trial, total spending against the hosted-spending cap, and cost per verification-positive trial; the complete ledger was released as cost_ledger.csv.The cost-per-verification-positive measure divides an arm's total spending by its trials containing a verification action.Source: §3.7 · p. 7

3.8 Analysis plan

The registered confirmatory test compared verification behavior when signals were present versus absent, using a project-adjusted model and a specified fallback. Other analyses were descriptive, and the plan set rules for incomplete execution and reporting outcomes.

3.8 Analysis plan

  • The confirmatory test compared verification-action likelihood with signals present versus the shared control, using project-adjusted logistic regression and a likelihood-ratio test.The present pool excluded control and inconsistent-surface conditions.Source: §3.8 · p. 7
  • If regression fitting failed or a comparison side lacked positive or negative trials, the plan used an exact within-project test combined across projects by 10,000 label reshuffles.Rates were reported with Wilson 95% intervals, with no other p-values in this version.Source: §3.8 · p. 7
  • Signal-class, issuer, inconsistency, harness, model-arm, retrieval, cost, and frontier-supplement comparisons were pre-specified as descriptive, without significance claims.Any exploratory p-values in a later version would be labeled and adjusted within their multiple-comparison family.Source: §3.8 · p. 7
  • Table 2 summarizes execution, and Table 3 presents installation-command outcomes by arm, distinguishing exit status, reported installation status, and machine end state.Table 2's caption gives the execution dates and deposit-assembly date.Source: Table 3 · p. 8, Table 2 · p. 8
  • Infrastructure failures were rerun once, malformed outputs were retained, and execution stopped after registered trials finished or hosted spending reached its cap.A second infrastructure failure left a cell incomplete; the plan required listing incomplete cells.Source: §3.8 · p. 8

4 Results

Across the registered study, assistants rarely opened provenance signals and never ran a verification command; signal presence and harness did not measurably change that behavior. Results also show that verification-positive trials were costly relative to ordinary trials, while the frontier supplement was exploratory and incomplete.

4.1 What ran, and what it cost

  • All 1,920 registered trials completed, while the supplement completed 194 of 216 before its $100.00 cap was exceeded at $100.59; hosted spending totaled $100.59.The local arm cost $0.00 over 19.9 hours of wall-clock time.Source: §4.1 · p. 8, Figure 3 · p. 9

4.2 Did a present signal change verification behavior?

  • 8 of 1,344 signal-present trials (0.6%) versus 0 of 384 controls (0.0%) were verification-positive; pooled tests found no measurable signal effect (p = 0.50).Every interval included zero, and no condition's point estimate exceeded 2.1%.Source: §4.2 · p. 9

4.3 Each signal class on its own

  • Only the attestation class had 4 of 192 positive trials (2.1%, 95% interval 0.8 to 5.2%); no trial verified, so logs could not distinguish valid from wrong-issuer material.Combining all four signals produced no positive trial and the lowest retrieval rate among the five signal classes.Source: §4.3 · p. 9, §4.4 · p. 10, Table 4 · p. 10

4.5 When the project’s own metadata disagrees with itself

  • 1 of 192 inconsistent-surface trials was positive, and assistants proceeded through reproduced metadata conflicts at a rate indistinguishable from consistent metadata, despite 83.9% supply-side prevalence [11].The inconsistent-surface end states were 153 proceeded, 36 declined, 2 incomplete, and 1 verified then proceeded.Source: §4.5 · p. 10

4.6 Harness and model

  • The gated harness produced 3 positives in 960 trials (0.3%) and the autonomous harness 6 in 960 (0.6%); all 9 main-study positives came from Sonnet.Approval requests were granted in 951 of 960 gated trials, and the study found no harness effect; Sonnet ended 156 of 360 trials at the turn limit or with malformed output.Source: §4.6 · p. 11, Table 5 · p. 14

4.7 Opening without checking

  • 16 of 2,114 trials (0.8%) had any retrieval event; 6 opened a signal before installing, while 4 opened one only after installation and counted as negative.Opening without checking was the only observed behavior; no trial ran a verification command.Source: §4.7 · p. 12, Table 6 · p. 14

4.8 What verification cost

  • Sonnet cost $0.103 per trial and $4.11 per verification-positive trial (9 positives); Fable cost $1.001 per trial and verified nothing.Kimi-K2.6 was the cheapest arm that verified at all, at $0.048 per trial, with one positive; the other arms' cost per positive was undefined.Source: §4.8 · pp. 12–13, Figure 10 · p. 13

4.9 The frontier supplement

  • In the frontier supplement, Kimi-K2.6 had 1 positive in 72 trials (1.4%), while Fable and gpt-5.6 had none; no frontier arm ran a verification command.The supplement was exploratory and ran only the two bookend conditions; its spending cap stopped Fable at 50 of 72 trials.Source: §4.9 · pp. 13–14, Figure 11 · p. 13

5 Discussion

The measured assistants rarely opened published trust signals and never verified them, so publishing alone was insufficient. The discussion argues that verification belongs in the assistant’s harness, while model rankings and cost-based procurement remain unsupported by the near-zero results.

5 Discussion

  • Signals were opened in 16 of 2,114 trials and checked in none, supporting the conclusion that publishing SBOMs, signatures, attestations, and channel declarations is necessary but insufficient.The result held across valid, forged, combined, and metadata-conflict conditions.Source: §5 · p. 14
  • The cost ledger frames verification as a per-installation budget line rather than a premium model tier, and recommends reporting cost per behavior of interest.The repeatedly opening arm cost $4.11 per positive trial; the most expensive arm opened nothing in 50 trials.Source: §5 · p. 14, Table 7 · p. 15
  • The study recommends procurement focus on whether a harness checks signatures and attestations before installation and records the check in its logs.A harness built to verify remains unmeasured.Source: §5 · pp. 14–15
  • The study cannot rank the six models on verification because five produced one positive trial or none and their intervals overlap; Fable opened nothing in 50 trials.Sonnet opened a signal file in 9 of 360 trials, at 4.11 US dollars per positive trial. Kimi-K2.6 showed the behavior once in 1 of 72 trials, at 0.048 per trial.Source: §5 · p. 15
  • The authors propose remeasuring the panel as systems change, broadening model and vendor coverage, and evaluating a harness that performs verification.These extensions follow from the study’s stated limitations.Source: §5 · p. 15
  • The paired studies measure both sides of the same mechanism: research software’s trust declarations are scarce and inconsistent, while assistants rarely open or check them.The authors release the instrument as a library for repeat measurement as models, harnesses, and publishing practice change.Source: §5 · p. 15

6 Limitations

The study covers six projects under controlled conditions, so its findings describe measured behavior on research software rather than the broader ecosystem. Near-zero positives and several design and model-selection choices further constrain interpretation and generalization.

6 Limitations

  • Six projects were studied, limiting the findings to controlled behavior on research software rather than the ecosystem.Trials used isolated containers and local copies of modified projects, not live development environments or mainstream registry installs.Source: §6 · p. 16
  • Hosted models and prices can change, the fixed prompt leaves prompt variation untested, and the local arm’s zero-dollar cost omits energy use.The hosted arms are reproducible in procedure but not indefinitely; the local arm is pinned and can be rerun.Source: §6 · p. 16
  • With 9 positive trials, the study cannot rank signal classes, harnesses, or models, and the approval condition measures being asked rather than a person refusing.Most installation commands failed at dependency resolution; behavior after a successful installation was not measured.Source: §6 · p. 16
  • Two projects account for every positive trial, including faasm, which lacks Python package metadata and may change how assistants explore it.The study releases the per-project table.Source: §6 · p. 16
  • All hosted confirmatory arms came from US vendors, leaving broader vendor and jurisdiction coverage for future work.The local-arm selection included one European candidate, Devstral from Mistral.Source: §6 · p. 16

7 Conclusion

In 1,920 registered trials, assistants opened a provenance signal before installing only nine times and verified none; no condition, harness, model, or price changed that. The paper releases the trial transcripts, coded outcomes, and costs to support repeat measurement as tooling changes.

7 Conclusion

  • In 9 of 1,920 registered trials, assistants opened a provenance signal before installing, but none checked or verified one.No condition, harness, model, or price changed that outcome.Source: §7 · p. 16
  • The study measures the demand side of provenance, which the authors say no prior measurement had covered.The observed failure was whether assistants read signals before installation.Source: §7 · p. 16
  • The authors release every trial’s transcript, coded outcome, and cost so others can repeat the measurement as tooling changes.They aim to support decisions about where verification should be built and what it is worth paying for.Source: §7 · p. 16
Loading 2609.07754v1…