On the Relation between Code Quality and Machine Learning Performance: A Large-scale Empirical Study

• Notebook flexibility supports rapid experimentation but can encourage departures from software-engineering practices, creating technical debt that impedes reliability, reproducibility, and production deployment.Moving experimental notebooks into production often requires extensive refactoring or rewriting, increasing costs and delaying deployment.Source: §1 · p. 2
• The study addresses whether popularity correlates with quality, how best-practice violations relate to performance, and whether expert and non-expert users differ in violations.The motivation is to test whether poor coding practices threaten model outcomes or whether high performance is compatible with high-quality code.Source: §1 · pp. 2–3
• Neither notebook popularity nor code expertise reliably signals code quality or ML performance, challenging their use as reuse-selection proxies.This matters because practitioners commonly reuse existing notebooks and may propagate poor practices when highly upvoted or expert-authored examples violate coding standards.Source: §1 · pp. 2–3
• The study analyzes 265,363 Python notebooks submitted to Kaggle competitions using Pylint for general quality and SonarQube with 34 ML-specific rules.Kaggle supplies competition scores, upvotes, and user expertise tiers for examining performance and social signals.Source: §1 · p. 3
• ML-specific violations consistently associate negatively with performance, whereas general Python code quality is effectively decoupled from performance across competition types and expertise levels.The findings show that the performance–quality relationship is neither uniformly present nor absent; it depends on which quality notion is measured.Source: §1 · p. 3

2.1 Code Quality of Python and ML Notebooks

• Prior studies measured prevalent quality violations in notebooks and ML code, including Pylint findings, notebook best-practice violations, and ML-specific smells, but did not link them to model outcomes.For example, Siddik and Bezemer analyzed 246,599 Kaggle notebooks, while Quaranta et al. found 74.49% violated at least one convention rule in 1,380 collaboratively authored notebooks.Source: §2.1 · p. 4
• General-purpose linters miss ML workflow practices, motivating notebook-aware tools such as Julynter and NBLizer, ML-aware tools such as dslinter and MLScent, and SonarQube’s data-science rules.Pyra checked 16 rules on 2,286 Kaggle notebooks and identified 4,214 violations.Source: §2.1 · p. 5
• Whether notebook code quality relates to resulting model performance remained unstudied at scale; this study addresses the gap using 265,363 Kaggle notebooks with Pylint and SonarQube.Pylint measures general Python quality, while SonarQube measures ML-specific quality.Source: §2.1 · p. 5

2.2 Social Signals and Kaggle as a Study Platform

• Evidence that social signals indicate quality is mixed: Quaranta et al. found slightly better best-practice compliance among highly upvoted notebooks, while reputation showed no correlation with coding-practice compliance in related studies.Whether Kaggle upvotes and expertise tiers signal quality at platform scale remained open.Source: §2.2 · pp. 5–6
• Existing Kaggle datasets provide metadata and code resources, but none combined notebook-level static-analysis violations, competition performance scores, and social metadata.The study builds on Meta Kaggle and Meta Kaggle Code.Source: §2.2 · p. 6

3.1 Data and Tool Choice

• Kaggle was selected because its domain-specific notebooks are linked to competition tasks and scores, avoiding the broader-purpose notebook filtering required for repositories such as GitHub.The study used a January 30, 2026 snapshot of Kaggle’s open users’ data rather than scraping the website.Source: §3.1 · p. 6, §3.2 · p. 7
• Private Kaggle competition scores served as the performance proxy because they are computed on hidden test data and better reflect generalization than public scores.Because competitions use heterogeneous metrics and score directions, scores were polarity-aligned and converted to within-competition z-scores expressing each notebook’s relative standing.Source: §3.1 · p. 6, §3.3.4 · p. 12

3.4 Quality Rules Selection

• Quality analysis used Pylint for general Python issues and SonarQube configured with 34 data-science and ML-specific rules, while notebook-specific false positives and unavailable dependency checks were excluded.Notebooks were converted to Python scripts with JupyText; the scripts were analyzed without executing their original environments.Source: §3.1 · p. 7, §3.4 · p. 13

3.3 Dataset Enrichment

• The dataset was enriched with user Code and Competition expertise tiers, violation densities per 100 executable lines, and severity-specific violation counts for both analysis tools.Density used SLOC to exclude comments and conversion-added metadata, while team notebooks inherited the highest member expertise level; 85% were single-authored.Source: §3.3 · p. 10, §3.3.1 · pp. 10–11, §3.3.2 · p. 11

3.3.3 Quality score

• Separate Python and ML quality scores were computed from severity-weighted Pylint and SonarQube violations, respectively, with the highest-severity category receiving weight 5.The Python score weights Errors at 5, whereas the ML score gives SonarQube Blocker issues equivalent weight and leaves other severities unweighted.Source: §3.3.3 · pp. 11–12

3.2 Notebook Selection

• From an 8-million-notebook Kaggle snapshot, successive selection produced 414,129 competition-submission notebooks, 325,823 Python notebooks, and ultimately 265,363 notebooks after 32,054 lacked calculable z-scores.Notebooks with Python versions below 3.6 or unspecified versions were also excluded, leaving 324,286 notebooks before the z-score exclusions.Source: §3.2 · p. 7, §3.2.1 · p. 7, §3.2.2 · p. 8

4.1 Statistical framework

• Spearman’s rank correlation and Mann-Whitney-Wilcoxon U tests provided a non-parametric framework robust to non-normality and outliers.Effect sizes used revised Cohen’s thresholds, with CLES additionally reported for Mann-Whitney-Wilcoxon comparisons.Source: §4.1 · pp. 13–14

4.2.1 Statistical Methodology

• Rrb = 0.135 showed a small selection effect, while post-deadline differences were negligible across performance, Pylint, and ML violations.Selected notebooks had a 57% chance of outperforming non-selected notebooks, but did not differ in code violations; submission timing did not significantly affect quality or performance.Source: Table 2 · p. 16, §4.2.1 · p. 16, Table 3 · p. 17

4.2.2 Final Sampling

• The analysis used the Full Dataset (N = 265, 363) for RQ1 and the Best-Per-Team Dataset (N = 45, 443) for RQ2–RQ3.Keeping the highest-performing notebook per team and competition mitigated intra-team dependency and pseudo-replication, while the full dataset preserved notebook-specific popularity.Source: §4.2.2 · p. 17

4.3 RQ1; 4.4 RQ2

• RQ1 correlated upvotes with Pylint and ML quality scores using two-sided Spearman tests, whereas RQ2 correlated performance with four score- and density-based quality variables.RQ2 examined overall, competition-type, author-expertise, and combined strata, focusing on Playground, Featured, and Research competitions.Source: §4.3 · pp. 17–18, §4.4 · p. 18

4.5 RQ3. Do expert and non-expert users differ in their violation of best practices?

• RQ3 compared expert and non-expert distributions of normalized Pylint and ML violations by severity, alongside performance, using two-sided Wilcoxon-Mann-Whitney U tests.Experts comprised Expert, Master, and Grand Master users; non-experts comprised Novice and Contributor users.Source: §4.4 · p. 18, §4.5 · p. 19

4.6 Consolidated Power Analysis

• α = 0.005 and power = 0.95 for research questions and 0.99 for confounder assessments governed all tests, with sensitivity analyses converting Cohen’s d to r.Minimum detectable effects ranged from ρ = 0.021 to ρ = 0.135 for RQ2 configurations, all interpreted as negligible to small.Source: §4.1 · p. 14, §4.6 · p. 19, §4.4 · p. 19

5.1 RQ1. Is there a correlation between notebook popularity and code quality?

• Popularity has negligible correlations with both Pylint and ML quality scores, so upvotes do not indicate that a notebook contains quality code.The tests used the full dataset (n = 265, 363), and both correlations were statistically significant but below the negligible-effect threshold |ρ| < 0.1.Source: §5.1 · pp. 20–21, Table 5 · p. 21

5.2 RQ2. How do violations of best practices relate to machine learning performance?

• ML-specific violations correlate negatively with performance, while Pylint quality is negligible or unrelated across datasets, competition types, and expertise groups.On the Best-Per-Team Dataset, ML quality correlated positively with performance (ρ = 0.120), whereas ML violations correlated negatively (ρ = −0.132); Pylint quality showed a negligible negative correlation (ρ = −0.0795).Source: §5.2.1 · p. 21, §5.2.2 · p. 22

5.2.2 Analysis by Contextual Factors

• The negative association between ML violations and performance persists for code and competition experts and non-experts, with correlations from ρ = −0.117 to ρ = −0.157.For experts, correlations were ρ = −0.148 for code expertise and ρ = −0.157 for competition expertise; non-expert correlations were ρ = −0.129 and ρ = −0.117.Source: §5.2.2 · p. 22
• Competition expertise strengthens the ML-performance association, reaching ρ = −0.19 for ML violations in Research competitions, but becomes non-significant in Playground competitions (p = 0.404).The Playground result likely reflects the smaller sample (N = 541) and the more casual nature of those competitions.Source: §5.2.2 · p. 23

5.3.1 Competition expert and non-expert

• Competition experts achieve higher performance, with fewer ML-specific violations but more Pylint errors and refactoring issues than competition non-experts.Performance differences had medium effect size (rrb = 0.224, CLES = 61%); experts violated fewer ML-specific rules (rrb = −0.278), primarily through fewer Major violations (rrb = −0.261).Source: §5.3.1 · p. 25, Table 16 · p. 25

5.3.2 Code expert and non-expert

• Code experts do not differ meaningfully from non-experts in performance or code quality, with rrb = 0.028 and negligible effects for variables showing significance.Several measures, including Pylint warnings and critical ML violations, were not statistically significant (p > 0.005).Source: §5.3.2 · p. 26, Table 17 · p. 26

6 Discussion

• General Python quality is decoupled from ML performance, whereas ML-specific violations show a consistent negative association that survives expertise and competition-type controls.The authors interpret this as a robust association rather than a demonstrated causal mechanism.Source: §6 · p. 26, §6.1 · p. 26, §6.2 · p. 27

6.1 General quality and ML-specific quality are different constructs

• General Python and ML-specific quality are distinct constructs because Pylint targets software-engineering concerns, while SonarQube targets practices whose misuse can affect pipeline behavior.General Python best practices neither improve nor harm ML performance, whereas ML-specific practices carry performance-relevant information.Source: §6.1 · p. 27

6.3 Two expert populations and the limits of social signals

• Competition experts perform better and violate fewer ML-specific rules, but show negligible overall Pylint differences and more Error and Refactor violations.Code experts are neither better performers nor cleaner coders, with negligible effects across comparisons.Source: §6.3 · pp. 27–28
• Popularity and code expertise do not reliably indicate notebook quality or performance, while competition expertise provides only a small indication of ML quality and performance.The discussion warns that highly voted, expert-authored, or high-performing notebooks may propagate low-quality code as pedagogical templates.Source: §6.3 · p. 28, §6.5.1 · p. 33

6.4 Most common violations: our dataset violations against the literature

• 87.25% of studied notebooks contain the dominant ML-specific violation, omitting dtype in pandas.read_csv or pandas.read_table, whose mean density is an order of magnitude above the second-ranked rule.Three of the five most violated rules target the pandas API, reflecting widespread delegation of type inference to pandas despite associated risks.Source: §6.4 · p. 28, Table 18 · p. 29
• ML-rule prevalence does not establish performance impact: the study measures aggregate association, not each rule’s individual contribution.Low counts for framework-specific rules are also confounded by library prevalence and their narrower API-misuse scope.Source: §6.4 · p. 30

6.5.1 For ML newcomers

• For newcomers, the authors recommend learning software engineering alongside ML and treating cloned notebooks as reviewable starting points rather than validated examples.Upvotes and code-expert badges offer no guarantee of sound software-engineering practices; competition expertise gives only a small ML-practice signal.Source: §6.5.1 · p. 33

6.5.2 For ML practitioners

• For practitioners, adopting ML-specific practices is plausibly beneficial, while general Python practices have zero measured performance cost and retain benefits for reproducibility, reuse, and comprehension.Future research should expand the 34-rule ML-specific set, estimate individual rule effects, study clones jointly with violations, and examine pipeline-step distributions.Source: §6.5.2 · pp. 33–34, §6.5.3 · p. 34

7.1 Construct Validity

• Construct validity is limited by dated notebooks, proxy performance scores, tool-dependent false positives, severity weighting, expertise aggregation, and within-competition z-scores.Older code may trigger inaccurate SonarQube ML rules; the performance proxy may include non-competitive submissions; 334 Blocker violations make the SonarQube weighting unlikely to affect conclusions; mixed-expertise teams comprise at most 2.5% of the dataset, and z-scores do not support absolute-performance conclusions.Source: §7.1 · pp. 34–35

7.2 Internal Validity

• Internal validity is strengthened by retaining one notebook per team and competition, while selection and post-deadline biases are small or negligible; residual bias remains when analysed code did not produce the reported score.A 30-line minimum excludes trivial or precomputed-result notebooks, but some remaining code may still be unrelated to performance output.Source: §7.2 · p. 36

7.3 Conclusion Validity

• Conclusion validity is supported by non-parametric tests, a priori sensitivity analyses, and α = 0.005, but no multiple-comparison correction is applied.Competition types with insufficient sample sizes were excluded, and conclusions rely on agreement across many tests with p-values far below alpha.Source: §7.3 · p. 36

7.4 External Validity

• External validity is limited because the study examines Python code from Kaggle competition notebooks, which may not represent other notebook media, communities, languages, or later practices.The data come from a single January 2026 snapshot, and the findings may not generalise even to Kaggle users outside competition participation.Source: §7.4 · p. 37

• The study analyzed 265,363 Kaggle machine-learning notebooks using Pylint for general Python quality and SonarQube with 34 ML-specific rules.Violations were related to standardized private competition scores, upvotes, and Kaggle progression tiers.Source: §8 · p. 37
• General Python violations showed negligible or non-significant correlations with performance, whereas ML-specific violations had a consistent small negative association across competition types and author expertise.The findings suggest software engineering practices do not cost performance, while ML-specific practices may support better outcomes.Source: §8 · p. 37
• Popularity and code expertise provided no indication of notebook quality or performance; competition expertise related to better performance and fewer ML-specific violations, without guaranteeing general Python quality.Source: §8 · p. 37
• Software engineering best practices can be integrated from early experimentation, while highly voted or expert-authored notebooks provide no guarantee of quality.The authors argue that sampling strategies based on these social signals should be reconsidered.Source: §8 · p. 37
• Future work will identify which ML-specific practices matter most, examine how violations propagate through clone-and-own practices, and locate violations across ML-pipeline steps.Source: §8 · p. 38

• All authors contributed to study conception, design, writing, and interpretation; the first author led data collection and most data analysis.The other authors reviewed and edited the manuscript, and all authors discussed uncertainties and contributed to interpreting results.Source: Declarations · p. 38
• The French National Research Agency supported the work through project ANR-AAPG2024, PROFIL.Source: Declarations · p. 38
• The study datasets and R/Python analysis scripts are available at the stated Zenodo repository.Repository: https://doi.org/10.5281/zenodo.21464700.Source: Declarations · p. 38
• Ethical approval and informed consent were not applicable.Source: Declarations · p. 38
• The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.Source: Declarations · p. 38

A SonarQube Selected Rules

• The SonarQube analysis profile comprised 34 Python rules.The selected rules are displayed in Table 19.Source: A SonarQube Selected Rules · p. 42, Table 19 · p. 42
• Table 19 presents the SonarQube Python rules profile used in the analysis.The supplied passage identifies the table as the profile display but does not provide individual rule names or values.Source: Table 19 · p. 42

B Cohen’s effect size conversion formula adapted for groups of different size

• For user-selected notebooks (n = 11, 918) versus non-selected notebooks (n = 253, 445), the adapted conversion yielded d = 0.049 and r = 0.01.The reported effect size was converted using the adapted Cohen’s formula for groups of different size.Source: B Cohen’s effect size conversion… · pp. 48–49
• The user-selected and non-selected notebook groups had an effect of d = 0.049 before conversion.The groups contained 11,918 and 253,445 notebooks, respectively.Source: B Cohen’s effect size conversion… · p. 48
• For notebooks submitted before (n = 214, 294) versus after (n = 51, 069) the competition deadline, the adapted conversion yielded d = 0.026 and r = 0.01.The reported effect size was converted using the adapted Cohen’s formula for groups of different size.Source: B Cohen’s effect size conversion… · p. 49
• The before- versus after-deadline notebook groups had an effect of d = 0.026 before conversion.The groups contained 214,294 and 51,069 notebooks, respectively.Source: B Cohen’s effect size conversion… · p. 49

RQ3. Competition expertise power analysis

• After conversion, the analysis reported an effect of r = 0.021367.The passage also reports 0.07893142 + 13.64.Source: RQ3. Competition expertise power… · p. 49

RQ3. Code expertise power analysis

• r = 0.021364 was the effect found after conversion.The reported converted value was 0.060930642 + 8.13.Source: RQ3. Code expertise power analysis · p. 49

C Power Analysis Parameters and Results Tables

• The sensitivity-analysis tables report G*Power parameters and results for selected versus non-selected notebooks.Source: Table 20 · p. 50
• Additional G*Power analyses compare notebooks submitted before and after competition deadlines.Source: Table 21 · p. 50

D Most violated Pylint rules in our dataset

• Table 30 ranks the top 20 Pylint rules violated in the full dataset by mean density, with the mean expressed in ×10^-3.Source: Table 30 · p. 50
• The top-20 Pylint-rule table continues across two subsequent pages.Source: Table 30 · p. 50, D Most violated Pylint rules in our… · pp. 51–52

C Power Analysis Parameters and Results Tables

• The reported sensitivity analyses include an overall comparison and a separate analysis for group 2.Source: Table 22 · p. 53, Table 23 · p. 53, Table 28 · p. 55
• Group 2 sensitivity analyses separately examine competition experts and non-experts.Source: Table 24 · p. 53, Table 25 · p. 54
• Group 2 sensitivity analyses also separately examine code experts and non-experts.Source: Table 26 · p. 54, Table 27 · p. 54