Source-linked AI summary
On the Relation between Code Quality and Machine Learning Performance: A Large-scale Empirical Study
Marius Mignard, Steven Costiou, Anne Etien
TL;DR
The paper examines whether notebook code quality relates to machine-learning performance and whether social signals indicate either. Across its study, general Python quality is decoupled from performance, while ML-specific violations show a consistent small negative association; popularity and code expertise provide no indication.
Problem
The study addresses limited evidence about whether code quality and ML performance are related and whether popularity or expertise reliably signal notebook quality.
Method
The paper investigates correlations between notebook code violations, ML performance, popularity, and user expertise across three research questions.
Results
General Python quality is decoupled from performance, whereas ML-specific violations show a consistent small negative association; popularity and code expertise indicate neither quality nor performance.
Takeaways & Limitations
General software-engineering practices are not associated with worse performance, while ML-specific practices are associated with better outcomes.
Takeaways & Limitations
Some analysed notebooks may contain code that did not produce the measured performance score, introducing potential bias.
Abstract
from arXiv · showhide
Context: Computational notebooks are the standard environment for machine learning (ML) development. Within the ML community, model performance is often the primary considered metric, and code quality is treated as a secondary concern. This prioritization relies on a largely untested assumption that code quality and ML performance are unrelated. Practitioners also reuse existing code that may come from notebooks selected through social signals (popularity, author expertise) whose reliability as quality proxies has never been assessed. Objective: We empirically investigated the relationship between code quality and ML performance in notebooks, and evaluated whether popularity and author expertise give indication on code quality or performance. Method: We conducted a large-scale empirical study of 265,363 Python notebooks submitted to Kaggle competitions. We assessed code quality with two static analysis tools: Pylint, capturing general Python code quality, and SonarQube, configured with a profile of 34 rules targeting data-science and ML-specific practices. Results: The relationship between code quality and performance depends on the notion of quality considered. General Python code quality is decoupled from ML performance, showing negligible or non-significant correlations across all observations. In contrast, ML-specific violations exhibit a consistent, small negative association with performance that persists across all observations. The popularity of a notebook does not give information on the code quality or performance. Code expertise provides no information on quality or performance, but competition expertise correlates with better performance, fewer ML-specific violations, and slightly more Python errors and refactoring violations.
1 Introduction
This introduction frames a large-scale test of whether notebook code quality relates to machine-learning performance and whether popularity or expertise reliably signal quality. It argues that the relationship depends on the quality notion: general Python quality is decoupled from performance, whereas ML-specific violations are negatively associated with it.
- Computational notebooks support rapid, iterative experimentation but may encourage departures from established software-engineering best practices.Their cell-based, interactive design favors trial-and-error workflows, while flexibility can create software-quality concerns.
- Poor notebook quality can create technical debt that hinders production transitions, requiring refactoring or rewriting and increasing costs and deployment delays.The introduction links code quality to industrial workflow agility.
- The paper examines whether popularity and author expertise are reliable proxies for notebook quality or performance amid widespread code reuse and “clone and own” practices.Highly upvoted or expert-authored notebooks may become pedagogical references for novices, making these social signals consequential for reuse and education.
- The study analyzes 265,363 Kaggle competition notebooks using Pylint and ML-specific SonarQube static analysis.Kaggle provides a setting where notebooks are associated with competitions and can be compared at scale.
- General Python code quality is effectively decoupled from ML performance, whereas ML-specific violations show a consistent negative association across competition types and expertise levels.The study distinguishes Pylint-based general quality from ML-specific practices assessed with SonarQube.
2 State of the Art
Prior work has measured code-quality violations in notebooks and ML code and developed specialized analysis tools, but has not tested their relationship with model performance at scale. Whether popularity and expertise reliably signal notebook quality also remains unresolved.
- Prior quality studies: Static-analysis research has documented quality violations across notebooks and ML code, including differences between notebooks and scripts and between ML and non-ML notebooks.Findings vary across datasets: notebooks were reported as having more violations than scripts in one study, but the opposite conclusion in a Kaggle replication.
- Specialized quality tools: General-purpose linters cannot detect workflow-specific ML practices, motivating notebook-aware and ML-aware tools such as Julynter and SonarQube’s data-science rules.Examples include non-reproducible randomness, silent type inference during data loading, and API misuses that alter pipeline behavior.
- Research gap: No prior study tested at scale whether notebook code quality relates to resulting model performance; this study addresses the gap using 265,363 Kaggle notebooks.It uses Pylint for general Python quality and SonarQube for ML-specific quality.
- Social signals: Practitioners may use popularity and expertise as trust signals, but evidence is limited and conflicting, leaving their reliability as quality indicators unresolved at platform scale.Prior Kaggle evidence found only a slight association between upvotes and best-practice compliance in a small, pre-filtered sample.
- Research resources: Existing datasets provide notebook metadata, ML-stage annotations, or static-analysis information, but none combines notebook-level violations, competition performance, and social metadata.The study builds on Meta Kaggle and Meta Kaggle Code.
3 Notebook Selection Methodology and Dataset Building
The study built a Python Kaggle-notebook dataset by filtering competition submissions and excluding unsuitable competitions, languages, versions, and short notebooks. It used Kaggle competition scores as the ML-performance proxy and Pylint plus SonarQube to measure general and ML-specific code-quality violations.
- Kaggle as data source: Kaggle competition scores served as the proxy for ML performance, linking each analyzed notebook to a competition task and score.Competition submissions were required because the study jointly examined coding practices and ML performance.
- Code quality tools: Pylint measured general coding flaws, convention issues, and Python errors, while SonarQube supplied rules targeting data-science libraries and ML workflow practices.Notebooks were converted to Python scripts with JupyText before static analysis.
- Notebook selection: The selection began with 414,129 competition-submitted notebooks, excluded Getting Started competitions, and retained 355,760 notebooks.Getting Started competitions were removed because tutorial-derived notebooks could bias the analysis.
- Notebook selection: After excluding non-Python notebooks and scripts, the dataset contained 325,823 notebooks, reflecting the study’s focus on Python code violations.Python represented 98% of Kaggle code according to the supplied passage.
- Notebook selection: Removing notebooks with fewer than 30 source lines of code left 309,291 notebooks, excluding minimal code and pre-calculated-result submissions.Source lines of code excluded blank lines, comments, and Markdown text.
4 Study Design
The study uses non-parametric tests, effect-size thresholds, and sensitivity analyses to examine 265,363 Kaggle notebooks while addressing selection, timing, intra-team, and skill-related confounding. Selection bias was small and post-deadline bias negligible, supporting the final sampling strategy.
- 4.1 Statistical framework: The analysis used Spearman’s rank correlation and Mann-Whitney-Wilcoxon U tests, avoiding distributional assumptions and providing robustness to non-normal data and outliers.Common Language Effect Size was additionally reported for Mann-Whitney-Wilcoxon comparisons.
- 4.2 Confounding variables and sampling: The design addressed selection, post-deadline, intra-team dependency, and skill confounding before defining final datasets for the research questions.Skill was incorporated by stratifying relevant analyses by expertise and competition type, while intra-team dependency was removed through sampling.
- 4.2.2 Final datasets: The study analyzed 265,363 notebooks, using the full dataset for RQ1 because popularity is notebook-specific and iterative team submissions can have different upvote counts.The complete dataset retained the granularity needed to analyze popularity.
- 4.2 Confounding variables and sampling: Post-deadline submission bias was negligible for performance, Pylint violations, and ML violations, with Rrb values of −0.037, −0.01, and −0.05, respectively.The findings indicate that submission timing did not significantly affect code quality or notebook performance.
- 4.2 Confounding variables and sampling: Selection bias was small (Rrb = 0.135), with selected notebooks outperforming non-selected notebooks in approximately 57% of pairwise comparisons.The comparison also found negligible differences in Pylint and ML violations.
5 Results
ML-specific violations show a small, consistent negative association with performance, whereas general Python quality is largely unrelated to performance. Popularity and code expertise provide little indication of notebook quality or performance, while competition expertise is associated with better performance and fewer ML-specific violations.
- RQ1: Popularity has negligible correlations with both Pylint quality and ML quality scores, so upvotes do not indicate notebook code quality.Both correlations were statistically significant, but their absolute Spearman correlations were below 0.1, the negligible-effect threshold.
- RQ2: ML-specific violations consistently correlate negatively with performance, while Pylint violations show negligible or non-significant correlations across competition types and expertise groups.The negative association is significant across competition types and expertise groups, whereas general Python quality remains decoupled from performance.
- RQ2: ρ = −0.193 is the strongest reported competition-type association for ML-specific violations in Research competitions, compared with ρ = −0.114 in Playground and ρ = −0.141 in Featured competitions.The Research effect is small; the Playground and Featured effects are significant but smaller.
- RQ2: Among competition experts, the strongest ML-violation association occurs in Research competitions (ρ = −0.19), while the Playground association is non-significant (p = 0.404).The Playground result may reflect its smaller sample size (N = 541) and more casual competition context.
- RQ3: Competition experts achieve better performance and fewer ML violations, but show slightly more Python errors and refactoring violations; code experts do not differ meaningfully from non-experts.Competition experts show a medium performance effect (rrb = 0.224, CLES = 61%), whereas code expertise yields negligible performance differences.
6 Discussion
The discussion shows that code quality is not a single construct: general Python quality is unrelated to ML performance, whereas fewer ML-specific violations consistently associate with better performance. Popularity and code expertise provide no reliable quality signal, while competition expertise offers limited information about ML quality and performance.
- General Python quality is decoupled from ML performance, whereas ML-specific violations show a consistent negative association with performance.The asymmetry reflects that Pylint and the ML-specific SonarQube profile capture different facets of notebook quality.
- The negative association between ML-specific violations and performance persists across author-expertise strata, suggesting that expertise alone does not explain the effect.The effect remains observable for experts and non-experts in both code and competition expertise analyses.
- Competition experts perform better and violate fewer ML-specific rules, but they show negligible overall Pylint differences and more Error and Refactor violations.Code experts are neither better performers nor cleaner coders, with comparisons showing negligible effects.
- Popularity and code expertise do not reliably indicate notebook quality or performance, so trusted notebooks may still propagate poor software-engineering practices.Only competition expertise offers a small or medium indication of performance and ML quality, not general Python quality.
- The most prevalent ML-specific violation concerns missing dtype parameters in pandas read operations, occurring in 87.25% of notebooks, while prevalence does not establish performance impact.Pandas API rules dominate the most frequent violations, and individual rule contributions to performance remain future work.
- Adopting ML-specific practices appears plausibly beneficial, while applying general Python conventions is performance-neutral and retains maintainability benefits.The study recommends learning software-engineering practices alongside ML and expanding the rule set to identify individual practice effects.
7 Limitations and Threats to Validity
The study addresses construct, internal, conclusion, and external validity threats through design choices and empirical checks. Key limitations concern measurement choices, selection and dependency biases, statistical assumptions, and generalisability beyond the analysed data.
- Internal validity: Notebook selection may retain outdated practices, code unrelated to performance, or submissions that do not genuinely aim to maximise competition scores.The dataset spans notebooks authored from February 2017 to January 2026, and notebooks with fewer than 30 source lines were excluded but some irrelevant code may remain.
- Construct validity: The study’s validity is threatened by its use of selected static-analysis tools, severity weights, competition scores, expertise groupings, and within-competition z-scores as proxies or measures.Different tools or rule sets may yield different effects, false positives may add noise, and the z-score reflects relative rather than absolute performance.
- Internal validity: Keeping one notebook per team and competition neutralises intra-team pseudo-replication, while assessed selection and post-deadline biases were small or negligible.Selection bias was small for performance and negligible for violation counts; post-deadline bias was negligible across performance, Pylint violations, and ML violations.
- Conclusion validity: The statistical analysis uses non-parametric tests and sensitivity analyses, but reports many uncorrected p-values, relying on agreement across tests and a conservative α = 0.005.Sensitivity analyses used power 0.95 for research questions, 0.99 for confounder assessments, and did not interpret effects below minimal detectable thresholds.
- External validity: Findings may not generalise beyond Python code in Kaggle competition notebooks, because the notebook medium, language, community, and competition context may influence results.The study used a single Kaggle data snapshot from January 2026, and relationships may shift as practices, libraries, and platform tiers evolve.
8 Conclusion
The study finds that the relationship between code quality and machine-learning performance depends on the quality notion: general Python quality is decoupled from performance, whereas ML-specific violations show a small negative association. Popularity and code expertise provide no indication of quality or performance, while competition expertise relates to better performance and fewer ML-specific violations.
- Study design: The study analyzes 265,363 Python ML notebooks from Kaggle using Pylint and SonarQube, including 34 ML-specific rules.Pylint captures general Python quality, while SonarQube targets data-science and ML-specific practices.
- Results: General Python code quality is decoupled from ML performance, while ML-specific violations show a consistent small negative correlation with competition scores.The study reports negligible or non-significant correlations for Pylint violations across all tests, contrasting with the negative association for ML-specific violations.
- Social signals: Neither notebook popularity nor code expertise indicates code quality or performance, whereas competition expertise relates to better performance and fewer ML-specific violations.Competition expertise does not guarantee general Python quality.
- Implications: High-quality code is compatible with high performance, supporting integration of software-engineering best practices from the earliest stages of ML experimentation.The paper presents ML-specific practices as possibly conducive to better performance rather than as a costly afterthought.
- Implications: Highly voted or expert-authored notebooks offer no guarantee of quality, so sampling strategies based on these signals should be reconsidered.This implication follows from the finding that popularity and code expertise provide no indication of quality or performance.
Declarations
All authors contributed to the study’s conception, design, writing, and interpretation, while the first author led data collection and most analysis. The work was funded by ANR, with data and analysis scripts publicly available; ethical approval, informed consent, and conflicts were addressed as stated.
- Declarations: All authors contributed to the study’s conception, design, writing, and interpretation, while the first author conducted data collection and most data analysis.The first author prepared the original draft, and the other authors reviewed and edited it.
- Declarations: The study was supported by the French National Research Agency under project ANR-AAPG2024, PROFIL.
- Declarations: The generated and analysed datasets and the R/Python analysis scripts are available in a Zenodo repository.Repository: https://doi.org/10.5281/zenodo.21464700.
- Declarations: Ethical approval and informed consent were not applicable, and the authors declared no known competing financial interests or personal relationships.
A SonarQube Selected Rules · B Cohen’s effect size conversion formula adapted for groups of different size
The SonarQube analysis used a 34-rule Python profile, while adapted Cohen’s conversion yielded r = 0.01 for both unequal-group comparisons. The comparisons involved user-selected versus non-selected notebooks and pre- versus post-deadline submissions.
- A SonarQube Selected Rules: SonarQube analysis comprised 34 Python rules in the selected profile.The rules were displayed in Table 19.
- A SonarQube Selected Rules: The profile contained 34 selected Python rules for SonarQube analysis.
- A SonarQube Selected Rules: Table 19 presents the SonarQube Python Rules Profile used for the analysis.
- B Cohen’s effect size conversion formula adapted for groups of different size: For 11,918 user-selected versus 253,445 non-selected notebooks, the adapted conversion changed d = 0.049 to r = 0.01.
- B Cohen’s effect size conversion formula adapted for groups of different size: The user-selected and non-selected notebook groups had an initial effect size of d = 0.049 before conversion.
- B Cohen’s effect size conversion formula adapted for groups of different size: For 214,294 pre-deadline versus 51,069 post-deadline notebooks, the adapted conversion changed d = 0.026 to r = 0.01.
- B Cohen’s effect size conversion formula adapted for groups of different size: The pre- versus post-deadline notebook comparison had an initial effect size of d = 0.026 before conversion.
RQ3. Competition expertise power analysis
The competition expertise power analysis found a small effect after conversion, with r = 0.021367.
- RQ3. Competition expertise power analysis: After conversion, the analysis found an effect of r = 0.021367.The passage also reports 0.07893142 + 13.64, without specifying their roles.
RQ3. Code expertise power analysis
The code expertise power analysis found a very small effect after conversion, with r = 0.021364.
- RQ3. Code expertise power analysis: r = 0.021364 was the effect found after conversion in the code expertise power analysis.The passage also reports 0.060930642 + 8.13, without specifying what these values represent.
C Power Analysis Parameters and Results Tables · D Most violated Pylint rules in our dataset
The section reports G*Power sensitivity-analysis parameters and results for multiple notebook and expertise groups, then identifies the 20 most frequently violated Pylint rules in the full dataset.
- C Power Analysis Parameters and Results Tables: G*Power sensitivity-analysis parameters and results are reported for users’ selected versus non-selected notebooks.The supplied table identifies the comparison but does not provide cell values.
- C Power Analysis Parameters and Results Tables: A separate G*Power sensitivity-analysis table compares notebooks submitted before and after competition deadlines.The supplied table identifies the comparison but does not provide cell values.
- D Most violated Pylint rules in our dataset: Table 30 lists the top 20 Pylint rules violated in the full dataset, ordered by mean density expressed in ×10−3.The table continues across two additional passages.
- C Power Analysis Parameters and Results Tables: Additional G*Power sensitivity-analysis parameters and results are provided for an unspecified grouping.The supplied material does not expose the associated parameter values or outcomes.
- C Power Analysis Parameters and Results Tables: The section includes G*Power sensitivity-analysis results for group 2 overall and for competition-expert and competition-non-expert samples.Separate tables are provided for group 2, competition experts, and competition non-experts.
- C Power Analysis Parameters and Results Tables: Further G*Power sensitivity-analysis tables cover group-2 code-expert and code-non-expert samples, alongside two additional analyses.The supplied passages identify the tables but do not provide their numerical results.