PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeHealth Predicts the Token Cost of Agentic Code Review: A Pre-Registered Study of 465 Agent-Authored Pull Requests

Lo Gullstrand Heander, Markus Borg, Emma Söderberg, Christofer Rydenfält

arXiv:2610.21914v1cs.SEcs.AI

TL;DR

Automated code review is increasingly routed through multi-agent LLM pipelines, but its metered cost had not been related to code quality. In a pre-registered observational study of agent-authored PRs, the authors find lower review-and-fix token use with higher CodeHealth across three LLMs, independent of PR footprint.

  • Problem

    The cost of automated code review, and its relationship to code quality, had not been reported, despite the shift toward agentic review.

    Source: §1 · p. 2, §2.2 · p. 4

  • Method

    A fixed five-agent pipeline processed 465 agent-authored PRs from 246 repositories, with CodeHealth and footprint modeled separately for each of three LLMs.

    Source: §3.1 · p. 5, §7 · p. 28

  • Results

    One CodeHealth point was associated with 3.5% to 6.1% fewer tokens for all three LLMs, and adjusting for scored-file size absorbed the association.

    Source: §7 · pp. 28–29

  • Takeaways & Limitations

    The refactoring case extends to agent-reviewed code when refactoring changes the structure agents read through, rather than only raising the CodeHealth score.

    Source: §7 · p. 29

    The study counted review comments but did not assess whether they were correct or identified problems worth fixing.

    Source: §5.6 · p. 25

Abstract

from arXiv · show

Context: Organizations increasingly delegate code review to multi-agent large language model (LLM) pipelines billed per token. Low code quality raises human maintenance effort, but its effect on agent review cost has not been measured. Objective: We test whether baseline CodeHealth predicts the token cost of agentic review and fix beyond pull request (PR) footprint. Method: In a pre-registered observational study, we sampled 465 agent-authored PRs from 246 repositories in AIDev, jointly stratified by CodeHealth and footprint under repository caps. A frozen five-agent review-and-fix pipeline processed every PR with three different LLMs. Per LLM, we fitted a mixed model of log tokens on CodeHealth and log PR footprint, with a repository random intercept. Results: For all three LLMs the CodeHealth coefficient was negative, with 95% CI excluding zero; each CodeHealth point was associated with 3.5% to 6.1% lower token cost. CodeHealth was unrelated to PR footprint, so PR size does not explain the association. Two of three registered smells predicted fix cost; batch fixing left median CodeHealth unchanged. Exploratory analyses located the association in the review stage, where agents iterated more on unhealthy code without writing more comments. Adjusting for the size of the scored files absorbed the association, which we interpret as mediation by file size. Conclusions: A quality metric calibrated against human maintainability also predicts the inference cost of multi-agent review and fix. The effect is modest per point and not carried by PR size, so the refactoring case extends to agent-reviewed code when it changes structure and file size, not only the score. Data Availability: https://doi.org/10.5281/zenodo.22127920

1 Introduction

As code review shifts to multi-agent LLMs, its cost is metered in tokens, but the relationship between code quality and agent review cost had not been tested. This study asks whether CodeHealth predicts review-and-fix tokens beyond PR footprint.

1 Introduction

  • Organizations routing code review through multi-agent LLM pipelines face metered token costs, a cost of automated review not yet related to code quality.Prior technical-debt research measured human attention, working time, and effort instead.Source: §1 · pp. 1–2
  • The study tests whether baseline CodeHealth predicts agentic review-and-fix cost beyond PR footprint, because token use is mechanically coupled to diff size.It treats footprint as a possible mediator rather than simply adjusting it away.Source: §1 · p. 2
  • A CodeHealth point was associated with 3.5% to 6.1% lower total review-and-fix tokens across all three LLMs, controlling for footprint, language, and repository.The association’s 95% confidence interval excluded zero for each LLM; CodeHealth was essentially unrelated to PR footprint, so larger PRs did not account for the cost difference.Source: §1 · p. 2, Figure 1 · p. 3
  • The study contributes a pre-registered observational analysis across three LLMs and a frozen, reproducible five-agent review-and-fix harness.The harness uses four review agents and one fix agent, with prompts hashed and registered.Source: §1 · p. 2
  • The sample jointly stratifies CodeHealth and footprint across 465 agent-authored PRs from 246 AIDev repositories, with an audit of achieved balance.The study also documents two AIDev sampling failure modes and a pooling artifact in multi-LLM PR studies.Source: §1 · pp. 2–3

2 Background

Prior code-review research primarily measures human costs, while CodeHealth has been studied as a file-level maintainability measure and in AI-assisted development. The paper identifies inference cost as an unmeasured outcome in agentic review and asks whether CodeHealth predicts it beyond PR footprint.

2.1 CodeHealth as a Measurable Construct

  • CodeHealth is a file-level static-analysis score from 1 to 10, aggregating source-level smells such as complex methods, nested conditionals, and duplicated logic.The paper uses unhealthy, problematic, and healthy tiers for scores below 4, 4–8, and 9 or above.Source: §2.1 · pp. 3–4

2 Background

  • Prior code-review studies measured costs in human attention, time, or effort, while automated-review work predominantly reported quality and adoption rather than inference cost.The reported outcomes included comment precision, F1, developer-rated correctness and importance, adoption, and PR cycle time.Source: §2 · p. 3, §2.2 · p. 4

2.2 Agentic Code Review

  • Prior agentic-review studies reported quality and adoption outcomes, but not inference cost as a function of code properties.A proposed CodeHealth routing signal concerned task solving rather than review and presupposed a measured cost association on real PRs.Source: §2.2 · p. 4

2.3 CodeHealth and Agentic Review Cost

  • No study had related CodeHealth to automated-review cost; this study estimates the relationship while separating CodeHealth from PR footprint.The paper treats footprint as a mediator candidate because adjusting it away could erase part of the effect of interest.Source: §2.3 · p. 4

3 Method

The study applies a fixed review-and-fix pipeline to a stratified sample of agent-authored PRs in a pre-registered observational design. It estimates CodeHealth associations separately by LLM while accounting for footprint, language, and repository.

3 Method

  • The pre-registered observational study applies an unchanged pipeline to PRs; code varies between observations, and no condition is manipulated or randomized.The protocol’s registration fell during collection, after review agents started and before the fix agent ran.Source: §3.1 · p. 5, §3 · p. 5, Figure 2 · p. 6

3.1 Design and Hypotheses

  • Baseline CodeHealth is measured on the PR head before review, aggregated over touched files and weighted by changed lines; its estimate is associational, not causal.The model uses log total tokens as its outcome and estimates the percentage change associated with one CodeHealth point, holding footprint, language, and repository fixed.Source: §3.1 · p. 5
  • The registered hypothesis requires negative CodeHealth coefficients for all three LLMs and 95% confidence intervals excluding zero for at least two.The sample is drawn from AIDev’s curated tier of agent-authored PRs, restricted to merged PRs in five primary languages and repositories capped during sampling.Source: §3.1 · p. 5, §3.2 · p. 6

3.3 Pilot Studies and Design Evolution

Two pilot sampling failures drove changes to the study design; pilot 3 tested the resulting setup at smaller scale before the main collection.

3.3 Pilot Studies and Design Evolution

  • Pilot 1 confounded CodeHealth tier with PR size, while pilot 2 concentrated its unhealthy tier in one repository, so neither result generalized.Pilot 1’s healthy PRs had a median of about 40 changed lines; pilot 2 drew 12 of 13 unhealthy PRs from one repository.Source: §3.3 · p. 7
  • The pilots prompted joint CodeHealth-by-footprint stratification, repository caps, a repository random intercept, and a decomposition checklist.After batch fixing left CodeHealth unchanged for most pilot-2 PRs, the study made cost the primary outcome and retained change in CodeHealth as descriptive.Source: §3.3 · p. 7
  • Pilot 3 used the final 3 × 3 sampling grid and all three LLMs, but its 45 PRs supplied hypothesis-generating rather than confirmatory evidence.The main collection repeated its setup at larger scale with relaxed repository caps; character caps on tool output were added to limit spiraling token costs.Source: §3.3 · pp. 7–8, Table 1 · p. 8

3.4 Sampling

Sampling crossed CodeHealth tiers with PR-footprint terciles under repository caps, yielding 465 PRs from 246 repositories. Only the unhealthy medium- and large-footprint cells fell short of their targets.

3.4 Sampling

  • The 3 × 3 sampling grid crossed three CodeHealth tiers with three PR-footprint terciles, defined by changed-line thresholds.Small PRs had 59 or fewer changed lines, medium PRs 60–177, and large PRs 178 or more.Source: §3.4 · p. 8
  • Repository caps limited each repository to two PRs per cell and nine overall, and excluded all 103 PRs used in pilots.The sampler streamed the candidate pool to exhaustion under a fixed seed and per-PR checkpoints.Source: §3.4 · p. 8

3.5 Achieved Sample

  • The achieved sample contained 465 PRs from 246 repositories, with a mean of 1.89 PRs per repository.Fifty-five repositories spanned at least two CodeHealth tiers and supplied the within-repository contrasts for the random-intercept model.Source: §3.5 · p. 8
  • Only unhealthy-medium (33 of 56) and unhealthy-large (40 of 56) cells missed their targets, because low-CodeHealth PRs were scarce in the candidate pool.The sampler streamed all 19,403 candidates; the unhealthy tier, rather than total PR count, constrained sample size.Source: §3.5 · p. 8
  • The sample had a median CodeHealth of 8.14 and median 93 changed lines, with changed-line counts reaching 91,128.The authors report medians because the changed-line distribution was heavy-tailed; sampling was not stratified by language or originating coding agent.Source: §3.5 · p. 8

3.6 The Measurement Instrument

A fixed review-and-fix instrument processed each PR in an isolated worktree, with four review agents followed by one batch fix agent. The study used three LLMs with differing serving and cache regimes, so token totals were not compared inferentially across models.

3.6 The Measurement Instrument

  • The instrument ran four review agents in a single pass each, then one fix agent applied all reviewers’ comments in a single batch pass.The PR worktree was created at the PR head commit with a detached HEAD, isolating concurrent runs.Source: §3.6 · pp. 8–9
  • Review agents were restricted to PR-introduced or exposed issues, while dedicated test and intent-alignment reviewers were excluded.The fix agent received the PR intent as context, and only it could edit files.Source: §3.6 · p. 9
  • The prompts, orchestration logic, and LLM identities were frozen and filed in the pre-registration, with post-filing changes requiring an amendment.One tool-layer defect during collection was reported through an OSF amendment.Source: §3.6 · p. 9

3.7 LLMs and Infrastructure

  • The three LLMs represented inexpensive hosted, locally served open-weight, and current-generation hosted commercial deployment regimes.The hosted models were provider-versioned, so weight-level stability across the collection window could not be verified.Source: §3.7 · p. 9
  • Different prefix-cache regimes made total token counts incomparable across LLMs, so the analysis used separate within-LLM fits.Review iterations and uncached input plus output tokens were registered as cache-neutral secondary outcomes.Source: §3.7 · pp. 9–10
  • Figure 4 traces three median-cost PRs with similar footprints through the instrument, but the examples are illustrative rather than evidence for the estimates.Each PR came from a different CodeHealth tier and was processed with sonnet-5.Source: Figure 4 · p. 10

3.8 Measures

The measures define baseline CodeHealth over changed files, token cost across reviewers and the fix agent, and secondary outcomes for iterations, comments, and score change. Footprint is treated as a mediator candidate, while repository language and identity enter the model as fixed and random effects.

3.8 Measures

  • Baseline CodeHealth is a line-change-weighted mean of per-file scores across the PR’s changed files, measured before the pipeline runs.The analysis uses the continuous score; tiers serve only to stratify sampling.Source: §3.8 · p. 10
  • Total tokens per PR sum input and output tokens across the four review agents and the fix agent; cached tokens are a subset of input tokens.Cached tokens are recorded separately for cost weighting.Source: §3.8 · p. 10
  • Secondary outcomes include review and fix iterations, reviewer-role comment counts, and post-fix minus baseline CodeHealth.A missing post-fix score is treated as missing rather than zero when no file scores are returned.Source: §3.8 · p. 10
  • PR footprint is a mediator candidate, repository language is a fixed factor, and repository identity is a random intercept.The four review agents’ roles are partitioned approximately along the ISO/IEC 25010 taxonomy; DesignReviewer spans two characteristics.Source: §3.8 · p. 10, Table 3 · p. 11
  • The three LLM instruments differed in serving stacks, versions, and cached-input shares, which motivated separate rather than inferentially compared token analyses.The cached-input share pooled review and fix agents across the 465-PR collection.Source: Table 4 · p. 11

3.9 Data Analysis

The analysis used separate repository-random-intercept models for each LLM and a registered criterion requiring consistent CodeHealth direction across models. It also specified footprint decomposition, smell tests, descriptive CodeHealth change, and corrected exploratory analyses.

3.9 Data Analysis

  • The primary linear mixed model regressed log total tokens on baseline CodeHealth, log footprint, and language, with a repository random intercept.The model used restricted maximum likelihood; no random slopes were fitted because repositories contributed few PRs on average.Source: §3.9 · p. 11
  • Each LLM was fitted separately, and registered success required all three CodeHealth estimates to share a sign and at least two confidence intervals to exclude zero.The three fits reused the same PRs, so naive pooling would overstate their independent information.Source: §3.9 · p. 11
  • The footprint decomposition estimated CodeHealth-to-footprint, footprint-to-cost, and direct CodeHealth-to-cost associations, but could not establish that unhealthy code causes larger PRs.Tasks were not matched across CodeHealth levels.Source: §3.9 · p. 11
  • Registered smell tests used partial Spearman correlations between each smell and fix-agent tokens, controlling for changed lines and Holm-correcting the three hypotheses.The tested smells were Complex Method, Code Duplication, and Complex Conditional.Source: §3.9 · pp. 11–12
  • CodeHealth change was reported descriptively by LLM, while analyses not registered in advance were exploratory and Benjamini-Hochberg corrected.Token cost and footprint were log-transformed, using log1p when values could be zero.Source: §3.9 · p. 12

3.10 Attrition and Exclusion

The study specifies how incomplete sessions and exclusions are handled, then uses a limited replication subsample to assess run-to-run variation. Registration details disclose that pilot evidence was underpowered for the achieved design and document an instrument defect corrected before final estimates.

3.10 Attrition and Exclusion

  • 28 of 1,395 runs were excluded, all because a loop guard stopped fix sessions before completion; no run was excluded for infrastructure failure.Excluded PRs were not replaced, and exclusions were reported by LLM cell.Source: §3.10 · p. 12
  • Guard-stopped review sessions were retained as real review cost, while a PR without a completed fix record was excluded for that LLM.The sensitivity analysis excluding PRs with incomplete sessions affected 103 qwen3-6 PRs; the other two LLMs had none.Source: §3.10 · p. 12, Table 6 · p. 13

3.11 Replication Subsample for Instrument Stability

  • The main collection used one run per PR per LLM, so measured cost combines stable diff properties with run-to-run agent variation.Temperature 0 does not make a served LLM deterministic, and tool-loop divergence can redirect the token-consuming trajectory.Source: §3.11 · p. 12
  • The stability check reran 27 PRs from distinct repositories twice more per LLM, selected without conditioning on observed outcomes.The exploratory mixed-model analysis separates between-PR from run-to-run variation; replicates were not averaged into confirmatory estimates.Source: §3.11 · p. 12

3.12 Pre-Registration and Prior Evidence

  • Review sessions began before registration was frozen, but their stored token, iteration, and comment counts were not read; fix sessions began afterward.A tool-output defect affecting 67 PRs was corrected, and those PRs were recollected from scratch; all Section 4 estimates use corrected-tool data.Source: §3.12 · pp. 12–14
  • All three pilot slopes were negative, but every LLM's required sample size exceeded the 465-PR design; H1 therefore used a conjunctive direction-consistency criterion.At the achieved design, the minimum detectable reduction was 4.7% to 6.7% per CodeHealth point, enough for only two of three pilot point estimates.Source: §3.12 · p. 13, Table 7 · p. 14

3.13 Replication Package

The replication package makes the study materials and data available, with session-level records and transcripts supporting inspection of agent runs.

3.13 Replication Package

  • The package includes the pipeline, prompts, sampler, analysis scripts, documentation, and frozen dataset, alongside session records and raw agent transcripts.Review-agent tool results can be reconstructed by replaying logged calls; the fix agent's intermediate edit states cannot be reproduced byte for byte.Source: §3.13 · p. 14

3.14 Use of LLMs and Reporting Guidelines

The authors describe LLM use and reporting practices for the study instrument and its software stack. They also identify partial compliance with recommendations to repeat experiments and validate outputs against human judgment.

3.14 Use of LLMs and Reporting Guidelines

  • The pipeline, sampler, harness, and analysis scripts were developed with Claude Code; Opus 5.5 reviewed the draft, and the authors take responsibility for the final text.The four review agents and fix agent constitute the study instrument, built from LLMs under the empirical-software-engineering guidelines.Source: §3.14 · pp. 14–15
  • The study repeated experiments only on a 27-PR subsample and did not assess whether review comments were correct or fixes were good.The authors identify these as gaps concerning non-determinism and human validation of generated artifacts.Source: §3.14 · p. 15

4 Results

Across the three LLMs, higher baseline CodeHealth was associated with lower review-and-fix token cost, even after controlling for PR footprint. Exploratory results locate the association mainly in review work, while adjustment for scored-file size absorbs it.

4.2 RQ1: Baseline CodeHealth and Cost

  • One CodeHealth point was associated with 6.1%, 4.8%, and 3.5% lower total tokens for deepseek-v4-flash, qwen3-6, and sonnet-5, respectively; all three 95% CIs excluded zero.The estimates held footprint, language, and repository constant.Source: §4.2 · p. 15, Figure 5 · p. 16

4.3 RQ2: Direct or Footprint-Mediated Effect

  • CodeHealth was not associated with PR footprint, while footprint was the strongest cost driver; the CodeHealth–cost association persisted when footprint was held constant.The authors therefore describe the effect as direct with respect to PR footprint.Source: §4.3 · p. 16, Table 9 · p. 17

4.4 RQ3: Specific Code Smells and Fix Cost

  • Complex Method and Complex Conditional predicted fix-stage token cost in all three LLMs, but Code Duplication did not reach significance for any of them.The confirmed partial correlations for the two supported smells ranged from .145 to .212.Source: §4.4 · p. 17, Table 10 · p. 17

4.5 RQ4: CodeHealth After Batch Fix

  • Median ∆CodeHealth was exactly zero for every LLM and baseline tier, although individual PR scores moved in both directions.The description does not establish that batch fixing leaves CodeHealth unaffected.Source: §4.5 · pp. 17–18, Figure 6 · p. 18

4.6.1–4.6.2 Review-stage analyses

  • The CodeHealth association strengthened when analysis was limited to review tokens; unhealthy code was associated with more iterations and tokens per iteration, but not more review comments.The review-stage coefficient was more negative than the total-token coefficient for all three LLMs; the input-side pattern was more reliable than the output-side pattern.Source: §4.6.1 · p. 19, §4.6.2 · p. 19, Table 12 · p. 20

4.6.2 Review-Stage Decomposition: Iterations, Input, Output, and Comments

  • One CodeHealth point was associated with 7.6%, 7.2%, and 5.0% lower review cost per finding for deepseek-v4-flash, qwen3-6, and sonnet-5, respectively.All three intervals excluded zero and survived correction; the ratio is undefined for PRs with no review comments.Source: §4.6.2 · p. 19

4.6.3 Code Smell Count and Review Comments

  • Smell count predicted review-comment counts for all three LLMs while CodeHealth remained nonsignificant in the same fits; smell count and CodeHealth were related but not interchangeable.Within the unhealthy tier, smell counts ranged from 4 to 70 despite the score being near its floor.Source: §4.6.3 · p. 20

4.6.4 Size of the Scored Files

  • Adding scored-file size absorbed the CodeHealth association for all three LLMs, but the study could not separate association with CodeHealth from association with file size.CodeHealth and log file size correlated at Spearman ρ = −.71 to −.73; the adjusted intervals did not exclude zero.Source: §4.6.4 · pp. 20–21

5 Discussion

The discussion locates the cost association in agents’ code-reading work and interprets file size as a pathway, while cautioning that it is an association rather than a forecast. It argues that refactoring may benefit both human and agent readers when it changes code structure, and identifies unanswered questions about review quality, smell effects, and file size.

5 Discussion

  • Across a 6.9-point median CodeHealth contrast, healthy-tier PRs cost 91,000 tokens for sonnet-5, 134,000 for qwen3-6, and 360,000 for deepseek-v4-flash less per review-and-fix pass.The authors call the effect modest and describe the estimate as an association, not a forecast.Source: §5 · pp. 21–22, §5.2 · p. 23

5.1 Cost Associated with Code Reading

  • Removing the fix stage steepens the CodeHealth association for all three LLMs, locating the association in review, though review and fix cannot be separated for sonnet-5.The fix stage runs only after comments and adds variance the authors believe CodeHealth does not explain.Source: §5.1 · p. 22

5.2 File Size as a Pathway

  • Adjusting for scored-file lines absorbs the CodeHealth coefficient for all three LLMs; the authors interpret this as mediation by file size, not a spurious association.The sample cannot distinguish long, low-cohesion files from files long for unrelated reasons, such as generated code or data tables.Source: §5.2 · p. 23

5.3 Cognitive Load and Agentic Inference Cost

  • Lower CodeHealth is associated with more review turns, retrieved context, and generated tokens, but not more findings; the authors avoid inferring agent comprehension or cognitive load.They propose that interleaved logic, long methods, and low cohesion can require more context and path tracing before a finding is stated.Source: §5.3 · pp. 23–24

5.4 Smell Count Versus CodeHealth Score

  • Finding counts track smell counts rather than CodeHealth scores, while two of three registered smells predicted fix cost and the Code Duplication prediction failed.The authors interpret these as different predictors of different outcomes: smell count relates to comment volume, while CodeHealth relates to cost per finding.Source: §5.4 · p. 24

5.5 Roads to Code Improvement

  • The authors favor metric-guided refactoring with human decisions and review, and describe a CodeHealth quality gate as one way to hold a line while refactoring improves it.They rejected giving the fix agent a CodeHealth target because unrelated refactoring could complicate review and make later reverts harder; they argue refactoring should serve human and agent readers.Source: §5.5 · pp. 24–25

5.6 Future Work

  • Future work should grade comment correctness, isolate smell effects, and stratify by file size; the registered smell analysis tested fix cost, not review or total cost.The authors also suggest testing different tool and parameter choices and separating larger-file effects from lower quality within files.Source: §5.6 · p. 25

6 Threats to Validity

The authors organize validity threats around the observational design, sample and pipeline boundaries, measurement choices, and statistical uncertainty. Their safeguards address several threats, but some interpretation and generalization limits remain.

6 Threats to Validity

  • Because the study is observational, CodeHealth tier plays the role of treatment in the validity framework, and treatment-related labels should be read in that sense.The authors organize threats using internal, external, construct, and conclusion validity categories.Source: §6 · p. 26

6.1 Internal Validity

  • Repository caps and a random intercept address repository confounding, but within-repository effects are estimable only in the 55 repositories spanning multiple tiers.Tasks are not matched across CodeHealth levels, so the CodeHealth-to-footprint path does not support a causal interpretation.Source: §6.1 · p. 26
  • The authors pinned CodeHealth at version 1.4.0 and recollected affected PRs after fixing an output defect, while acknowledging residual risk from an undetected defect.They also report that baseline CodeHealth has more room to improve in unhealthy PRs and do not correct for regression to the mean because RQ4 is descriptive.Source: §6.1 · p. 26

6.2 External Validity

  • Results concern merged, agent-authored PRs in popular open-source projects and this study’s own pipeline; transfer to other authors, codebases, languages, or pipelines is not established.The three LLMs are a snapshot of current-generation systems, and successors may differ as capability, pricing, and serving efficiency change.Source: §6.2 · p. 26

6.3 Construct Validity

  • CodeHealth is one proprietary composite with CodeScene-affiliated authors among its validation literature; independent replication could strengthen the construct, and file-size adjustment cannot distinguish design-related length from other long files.Baseline scores cover whole touched files at the PR head commit, including the PR’s own changes; the head and base scores correlate at r = .995 for scorable PRs.Source: §6.3 · pp. 26–27
  • Total tokens do not capture every cost, and the study did not assess comment correctness, fix quality, or human agreement; prompts were fixed, and agents never received CodeHealth.Prior exposure to sampled code remains possible, and the authors say a membership probe would be needed to settle contamination concerns.Source: §6.3 · pp. 27–28

6.4 Conclusion Validity

  • The achieved sample was below each LLM’s pilot-extrapolated requirement, and the authors identify low power as the principal conclusion threat; heavy tails and multiple analyses also shaped their methods.They avoided pooling LLM rows after pilot results changed under clustered standard errors.Source: §6.4 · p. 28
  • Run-to-run noise makes qwen3-6’s reported uncertainty plausibly too small, and its interval excludes zero only when guard-stopped sessions are retained as cost.The replication subsample contains only 27 PRs, so its variance estimate is itself imprecise.Source: §6.4 · p. 28

7 Conclusions

The study finds that CodeHealth predicts review-and-fix token cost beyond PR footprint, with exploratory analyses locating the association in review and interpreting it as mediated by scored-file size. The authors support refactoring that changes code structure, while bounding their conclusions to the studied metric, cost construct, pipeline, LLMs, and PRs.

7 Conclusions

  • Across all three LLMs, one CodeHealth point was associated with 3.5% to 6.1% fewer review-and-fix tokens at constant footprint, language, and repository.The 95% confidence interval excluded zero for each LLM; unhealthy code did not arrive in larger PRs.Source: §7 · p. 28
  • Exploratory analyses located the association in review: agents took more turns and retrieved more context on unhealthy code without raising more comments.Adjusting for scored-file size absorbed the association for all three LLMs; the authors interpret this as mediation by file size, which they could not separate from CodeHealth in this sample.Source: §7 · pp. 28–29
  • The authors support refactoring that changes the structure agents read, such as splitting long methods and low-cohesion modules, rather than raising the score alone.The contrast between typical unhealthy- and healthy-tier PRs corresponded to about 91,000 to 360,000 tokens per review-and-fix pass, depending on the LLM.Source: §7 · p. 29
  • The conclusions are bounded to one quality metric, one cost construct, one pipeline, three LLMs at one development point, and merged agent-authored PRs in popular open-source repositories.The published pipeline and replication package allow retesting with later LLMs and other quality or maintainability metrics.Source: §7 · p. 29

Data Availability

The authors provide source code and data through a DOI and report their conflict-of-interest and ethics disclosures.

Data Availability

Conflict of Interest

  • A study author is employed by CodeScene, which provides the CodeHealth metric used in the study; the authors state that no commercial interests compromised scientific rigor.Source: Conflict of Interest · p. 29

Ethics Approval and Consent

Clinical Trial Number

Related papers

Loading 2610.21914v1…