PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures

Nicolas Lacroix, Frederic Precioso, Mireille Blay-Fornarino, Sebastien Mosser

arXiv:2610.10261v1cs.SEcs.LG

TL;DR

Heterogeneous ML pipelines and constrained classifiers make stage extraction difficult, limiting empirical understanding of ML practices. This study compares SLM classifications with replicated reference methods and tests taxonomy wording; the SLM is competitive but does not outperform the reference classifiers, and classifier choice changes practice insights.

  • Problem

    Pipeline diversity and evolving algorithms and libraries make stage extraction difficult, while existing methods have constraints in performance, scope, and taxonomy consistency.

    Source: §1 · p. 2

  • Method

    The confirmatory study compares SLMs with two reference approaches using a unified taxonomy, tests synonym-based taxonomy changes, and compares inferred practice patterns with manual labels.

    Source: §1 · p. 3, §4.3 · p. 12

  • Results

    SLMbest reached F1 scores of 0.912 on DS-Pipelines and 0.748 on DASWOW, but did not outperform the reference studies; classifier methods yielded different practice patterns from human labeling.

    Source: §8 · p. 36

  • Takeaways & Limitations

    Classifier choice affects inferred data-science practices, and taxonomy wording sensitivity means current SLMs are not an immediate solution to extraction limitations.

    Source: §1 · p. 3

    Training-data contamination cannot be confidently ruled out, and selecting the prompting technique with one SLM may give it an advantage.

    Source: §7.1 · p. 35

Abstract

from arXiv · show

Context: Once defined a taxonomy of stages structuring Machine Learning (ML) pipelines (e.g. Data Preprocessing, Modeling...), extracting these stages from source code is key for better understanding ML practices. However, the diversity caused by the constant evolution of ML (e.g., algorithms, datasets) makes this task challenging. Existing approaches either rely on non-scalable manual labeling or on classifiers that do not properly support domain's diversity. These limitations call for more reliable solutions. Objective: We evaluate whether Small Language Models (SLMs) can leverage their code understanding and classification abilities to address these limitations, and enhance our understanding of practices in ML. Method: We conduct a confirmatory study based on two relevant reference works representing current limitations in the state-of-the-art. We first compare several SLMs using Cochran's Q test, then evaluate the best-performing model against reference studies via two McNemar's tests. An additional Cochran's Q test examines how taxonomy definition variations affect the SLM performance. Finally, goodness-of-fit tests compare ML practice insights from SLM classification with those from prior studies. Results: First, we found that the taxonomy wording significantly impacts classification performance. Second, the best performing SLM yielded good results, yet, without outperforming other classifiers. Third, the three classification methods led to significantly different insights, with varying effect sizes, when exploring practices of data scientists. Conclusions: Limitations of existing classification methods bias our understanding of ML practices. While current SLMs show promising results without prior fine-tuning, they still exhibit common limitations, in addition to inference high costs challenging their applicability in large-scale studies.

1 Introduction

Heterogeneous ML pipelines make reliable stage extraction difficult, limiting empirical understanding of how data scientists organize their workflows. This confirmatory study evaluates SLMs against existing classifiers, tests taxonomy wording, and compares the resulting practice insights with manual labels.

1 Introduction

  • Existing extraction methods have limited performance and scope, and differing stage taxonomies also affect empirical analyses of pipeline practices.Pipeline diversity reflects exploratory work and the rapid evolution of algorithms and libraries.Source: §1 · p. 2
  • The study tests whether SLMs can reverse-engineer ML pipeline structures and whether their classifications change with taxonomy wording.Its research questions also compare SLM accuracy with existing classifiers and assess practice insights against manual labels.Source: §1 · pp. 2–3
  • The confirmatory study replicates two reference classification methods, selects a best-performing SLM, and compares classifications using a unified taxonomy.It also tests synonym-based taxonomy mutations and compares practice patterns with manually labeled pipelines.Source: §1 · p. 3
  • The authors report that classifier limitations influence understanding of ML practices, while SLMs remain sensitive to taxonomy formulation and costly to run.They describe SLM results as promising but not an immediate solution.Source: §1 · p. 3

2 Background and Related Work

Prior reverse-engineering work spans large and small language models, structured workflow frameworks, and rule-based or supervised extraction of ML pipeline stages. The literature motivates more flexible approaches while noting that existing methods remain bounded by rules, training data, or exploratory pipeline diversity.

2.1 Reverse-Engineering and Language Models

  • Language-model research has explored code abstraction and reverse-engineering, but prior work mainly used LLMs; SLM studies suggest smaller models can perform strongly without fine-tuning.Examples include REx86 [20] and reverse-engineering code with 8B-parameter models [21].Source: §2.1 · p. 4

2.2 From Structured Frameworks to Exploratory Practices

  • Jupyter notebooks support rapid experimentation across the pipeline lifecycle, while heterogeneous exploratory workflows complicate automatic structure extraction.CRISP-DM [23] remains a historical standard, but the text describes such frameworks as often inadequate for current exploratory practices.Source: §2.2 · p. 5

2.3 Reverse-Engineering Applied to ML Pipelines Extraction

  • Static mappings and supervised classifiers have offered pipeline-stage extraction, but their library coverage, predefined rules, or classifier performance remain constrained.Reported examples include CORAL [11], DS-Pipelines, and a prediction F1 score of 0.29 for Ramasamy et al. [7].Source: §2.3 · p. 5
  • Language models are proposed as a way to address these constraints by learning code patterns from broad training data that includes ML pipelines.The authors expect this to better support domain variability; they frame it as an expectation.Source: §2.3 · pp. 5–6

3 Reference Studies and Reproduction

The study selects two reproducible reference works with complementary classification granularity and scope, then reproduces their methods and examines dataset constraints. The DS-Pipelines mapping covers only a minority of parsed instructions, and its parser output required reverse-engineering to support instruction-level comparisons.

3.1 Reference Studies Selection

  • DS-Pipelines and DASWOW were selected as reference baselines for their instruction-level static mapping and cell-level supervised classification, respectively.Selection compared depth, library breadth, and publication date after requiring empirical practice results and executable open-source reproduction packages.Source: §3.1 · p. 6

3 Reference Studies and Reproduction

  • The study reproduced both reference works using their reproduction packages.The authors report providing their reproductions in their own reproduction package.Source: §3.1 · pp. 6–7

3.2 DS-Pipelines

  • DS-Pipelines maps notebook instructions to stages using a manually created file of 404 function or class names and an 11-stage taxonomy.Only six taxonomy stages were used in its notebook analysis.Source: §3.2 · p. 7

3.2.1 Challenges

  • The original parser returned stage-acronym sequences without linking stages to source instructions, so the authors reverse-engineered it to create instruction-to-stage mappings.They extracted 4,975 mapped instructions; dependency versions were inferred from release dates when unavailable.Source: §3.2.1 · p. 7

3.2.2 Findings

  • Only 30.66% of parser-produced instructions were supported by the static mapping, which covered 404 entries despite 875 distinct function names.The authors identify manually maintaining such a mapping as unsustainable as developers introduce new functions.Source: §3.2.2 · pp. 7–8
  • The review found 18 conflicting duplicated mapping keys and four rule inconsistencies, and excluded one of 105 notebooks because it was not an ML pipeline.The practice analysis therefore treats DS-Pipelines as 104 ML pipelines.Source: §3.2.2 · p. 8

3.3 DASWOW

DASWOW provides cell-level labels across 470 notebooks, but reproduction required resolving missing dependencies and recovering usable notebook code. The authors restricted evaluation to the test split and report adequate statistical power for the effects observed.

3.3 DASWOW

  • DASWOW contains 470 notebooks divided into train, test, and validation dataframes, with cell metadata and labels for ten taxonomy stages.The reported dataframe sizes are 1,918 test, 5,833 train, and 1,927 validation rows.Source: §3.3 · p. 8

3.3.1 Challenges

  • The authors inferred missing dependency versions and recovered the original notebooks after the reproduction package’s Python code was found corrupted.They validated the notebooks and manually fixed code inconsistencies before parsing.Source: §3.3.1 · pp. 8–9

3.3.2 Findings

  • Evaluation used only the test split because train and validation data had been merged for training; the dataset was also highly class-imbalanced.Data preprocessing or exploration accounted for 49.5% of stages, while save results accounted for 1.2%.Source: §3.3.2 · p. 9

3.4 Power Analysis

  • The a priori power analysis considered three scenarios for paired tests and examined goodness-of-fit requirements at degrees of freedom 1 and 46.The minimum required sample size for the paired-test analysis was 796 cells or instructions.Source: §3.4 · p. 9, Table 1 · p. 10, Table 2 · p. 10
  • All observed effects could be detected with adequate statistical power; the measured minimum effect size was ω = 0.267, requiring a sample of 410.The conservative scenario with large degrees of freedom was not observed in the experiment.Source: §3.4 · p. 10

4 Study Preparation

The study standardized SLM inference and prepared comparisons across reference taxonomies, prompt variants, and candidate models. Qwen2.5-Coder 7B Instruct performed best on the DASWOW test set, while the taxonomy unification reduced distinctions among stages.

4.1 Inference Configuration

  • Inference responses were constrained to a JSON schema with an enum of taxonomy classes; temperature, sampling, and penalties were controlled to limit randomness and class variation.The maximum output length was set to 15 tokens.Source: §4.1 · pp. 10–11

4.2 SLMs Selection

  • The seven code-focused SLMs were selected using openness, resource constraints, and performance across HumanEval, BigCodeBench, and EvalPlus.The study limited models to fewer than 8 billion parameters and selected top performers across the three benchmarks.Source: §4.2 · pp. 11–12

4.3 Taxonomies Unification

  • The unified taxonomy grouped equivalent terms from both reference taxonomies with expert input, but merged Modeling/Training, preprocessing/exploration, and evaluation/result-visualization distinctions.Three experts contributed to the taxonomy; their overall agreement was κ = 0.89.Source: §4.3 · pp. 12–15

4.4 Prompt Creation

  • Moving notebook code into the system prompt improved weighted average MCC by 0.024; eight in-context examples raised weighted average F1 by 0.017 and MCC by 0.014.Code prompting alone did not improve performance, while conditional-branch ordering produced a slight improvement.Source: §4.4 · pp. 15–16

4.5 Best SLM Selection

  • All 21 SLM pairs differed significantly (p < 0.001), while Phi-3-Mini-128K-Instruct produced 62.674% invalid outputs after struggling with structured JSON.Invalid outputs included badly structured JSON or unknown classes.Source: §4.5 · pp. 16–17
  • Qwen2.5-Coder 7B Instruct significantly surpassed the other SLMs across three performance metrics, with weighted average F1 0.754; Gemma 3 4B scored 0.641.The comparison used the DASWOW test dataset.Source: §4.5 · p. 16, Table 6 · p. 17

5.1 RQ2. Impact of the Taxonomy Mutations

The study tested whether replacing taxonomy headwords with domain synonyms changed SLM classification. Results showed significant statistical and practical effects across the mutated taxonomies.

5.1 RQ2. Impact of the Taxonomy Mutations

  • Each mutation replaced one taxonomy headword with a Data Science-domain synonym while leaving its definitions and comments unchanged; three synonyms were collected per element.The test used the smaller DS-Pipelines dataset.Source: §5.1 · p. 18

5.1.1 Results

  • Cochran’s Q found differences among the 18 mutated taxonomies and original (Q = 4112.01, p < 0.001); 140 of 171 post-hoc pairs differed significantly.Post-hoc comparisons used McNemar tests with Holm-Bonferroni correction.Source: §5.1.1 · p. 19
  • 82.46% of mutation pairwise comparisons were significant (maximum OR 9.3); mutations shifted weighted average F1 by up to +0.108 and weighted average MCC by +0.154.The practical differences were between Data Modeling to Model Building and Model Deployment to Model Prediction.Source: §5.1.1 · p. 19

5.2 RQ1. Performance Comparison between SLM and Reference Studies

SLMbest produced good but statistically and practically different classifications from the reference methods, without outperforming them. Its errors varied by dataset and stage, while inference remained resource-intensive.

5.2.1 Comparison Process

  • SLMbest classification was compared with DS-Pipelines at instruction level and with DASWOW at cell level after grouping SLMbest instruction labels by notebook cell.The comparison used reproduced reference results converted to a unified taxonomy; the SLM was not fine-tuned.Source: §5.2.1 · pp. 20–21

5.2.2 Results

  • Both McNemar tests found statistically significant differences between SLMbest and the reference classifiers: χ2(1, N = 1635) = 61.95, p < 0.001, OR = 0.49 versus DASWOW, and χ2(1, N = 4975) = 443, p < 0.001, OR = 0 versus DS-Pipelines.The null odds ratio for DS-Pipelines reflects that its static mapping predictions cannot be improved in this comparison.Source: §5.2.2 · p. 21
  • On DS-Pipelines, SLMbest scored weighted average F1 0.912, subset accuracy 0.91, and weighted average MCC 0.838; Model Prediction MCC was 0.90 versus 0.65 for Model Evaluation.The normalized classification matrix also shows weaker performance on Data Collection and Model Evaluation than on Data Preparation and Model Prediction.Source: §5.2.2 · p. 21, Table 8 · p. 21, Figure 5 · p. 22
  • Data Preparation over-classification produced a 14.14% false positive rate, while low perplexity was correlated with over-classification; perplexity reflects token prediction, not class prediction.On DASWOW, SLMbest also misclassified 23.55% of the dataset as Data Preparation, with a 64% false positive rate, and had a 47% false negative rate for Save Results.Source: §5.2.2 · pp. 21–23, Figure 6 · p. 23, Figure 9 · p. 26
  • On DASWOW, SLMbest had higher weighted average F1 than DASWOW (0.748 vs 0.729), but lower subset accuracy (0.545 vs 0.656), weighted average MCC (0.509 vs 0.587), and Hamming loss performance (0.117 vs 0.088); Jaccard scores were similar (0.609 vs 0.6).The cell-level comparison used multi-label classifications.Source: §5.2.2 · pp. 22–23, Table 9 · p. 24, Figure 8 · p. 25
  • More than 20 minutes were needed to classify the DASWOW test set with batch invariance, compared with about 300 milliseconds for DASWOW; batching reduced SLMbest time to 10:38 minutes.Energy use was 107 Wh with batch invariance and 29 Wh without it.Source: §5.2.2 · p. 25
  • SLMbest achieved good results without fine-tuning but did not outperform the reference classifiers, notably over-classifying Data Preparation and requiring substantial resources.The authors therefore could not reject H0.1.Source: §5.2.2 · p. 26

5.3 RQ3. Impact of Classification Methods on our Understanding of ML Practices

Against expert-labeled DASWOW test data, all three classifiers yielded significantly different stage distributions and transitions, with distinct errors. These differences also affected the sequential patterns identified in DS-Pipelines.

5.3.1 Exploring the DASWOW test dataset

  • The stage-distribution analysis used the DASWOW test split because it had expert ground truth and avoided contamination from splits used to train DASWOW.Expected populations were scaled to match classifier output sizes before goodness-of-fit testing.Source: §5.3.1 · pp. 27–28, Figure 11 · p. 29
  • For stage proportions, SLMbest over-represented Data Preparation by 20% versus ground truth, while DS-Pipelines under-represented it by 35.23%; arithmetic and subscripting operations often lack mapped function calls.Across stages, mean differences were +6.43% for SLMbest, −10.47% for DS-Pipelines, and −3.19% for DASWOW, which appeared more reliable.Source: Figure 10 · p. 27, §5.3.1 · p. 28
  • All three classifiers differed significantly from ground truth in stage distributions, with small-to-moderate effect sizes (ω = 0.267–0.303); residuals differed across classifiers and affected half the stages.DASWOW and DS-Pipelines had no significant residuals in common, while SLMbest shared two-thirds of its significant residuals with DASWOW.Source: §5.3.1 · pp. 29–30, Figure 11 · p. 29
  • All classifiers differed significantly from ground truth on transitions; DASWOW was most representative (ω = 0.398), versus large effects for DS-Pipelines (ω = 0.687) and SLMbest (ω = 0.582).The comparison used transition probability matrices and residual matrices; for cells with multiple labels, each edge received probability p = 1/n.Source: §5.3.1 · pp. 30–32, Figure 12 · p. 30
  • SLMbest over-represented transitions to Data Preparation and under-represented transitions to Data Modeling, while DS-Pipelines missed all transitions from Save Results because its mapping lacked that stage.The Model Evaluation-to-Data Preparation transition was over-represented by all three methods; DASWOW also under-represented Model Prediction-to-Model Evaluation transitions.Source: §5.3.1 · p. 31

5.3.2 Identifying sequential patterns in the DS-Pipelines dataset

  • SLMbest changed overall sequential-pattern counts only slightly, but symmetric differences revealed compensating errors, including eight differing pipelines for the flexible representative-pipeline pattern.For feedback loops, SLMbest found 47 versus 52 and 12 versus 13 matches; the analysis also identified six compensating errors.Source: §5.3.2 · pp. 32–34
  • SLMbest found 58 pipeline jungles versus 65 with DS-Pipelines; false positives between preparation stages could merge them and make data preparation appear less fragmented.The authors connect this discrepancy to SLMbest’s high Data Preparation false positive rate.Source: §5.3.2 · pp. 33–34

6 Deviations From The Registered Report

The study deviated from its registered report in model selection, taxonomy mutation, model eligibility, and goodness-of-fit testing.

6 Deviations From The Registered Report

  • Seven SLMs were compared instead of the planned five, using two additional benchmarks to reduce selection bias and reflect evolving evaluation practices.Source: §6 · p. 34
  • Taxonomy mutation synonyms came from domain experts and reference-study papers rather than dictionaries, to avoid non-domain-specific synonyms biasing the results.Source: §6 · p. 34
  • The study included restricted-weight models but no closed-weight models; Gemma 3 4B was its only selected restricted-weight model.Source: §6 · p. 34
  • Goodness-of-fit testing switched from Pearson chi-squared to G-tests because classification outcomes could be dependent on surrounding pipeline cells.The authors state the change did not affect the evaluation, which remained a goodness-of-fit test.Source: §6 · p. 34

7 Threats to Validity

The study identifies internal, external, and construct-validity threats involving the reference datasets, possible training-data contamination, taxonomy scope, and the measurement of classification and practice insights.

7.1 Internal Validity

  • The reference datasets’ diversity and quality were uncontrolled, training-set contamination could not be determined confidently, and three lab experts defined the unified taxonomy.Experts assessed dataset quality; the unique taxonomy and raw-line prompting limited contamination risk, while expert agreement was evaluated.Source: §7.1 · pp. 34–35

7.2 External Validity

  • Reliance on two studies limits generalizability, and the unified taxonomy does not cover every ML domain or finer-grained pipeline stage.The authors identify language-model training and detailed data-preparation stages as examples requiring taxonomy extension.Source: §7.2 · p. 35

7.3 Construct Validity

  • Stage imbalance threatens performance assessment, mitigated with MCC and per-stage F1; broad practice insights rely on reference-study indicators, supplemented by sequential-pattern analysis.The authors note that reporting additional insights could deepen the analysis.Source: §7.3 · pp. 35–36

8 Conclusion and Future Works

The study finds that SLM classification is competitive but does not outperform reference classifiers, and that taxonomy wording and classifier choice affect results and inferred ML practices. It calls for scalable, cost-aware classification methods.

Conclusion

  • Qwen2.5-Coder 7B Instruct outperformed the other selected SLM candidates, with a maximum odds ratio of 20.38.The study reports a significant statistical and practical impact from SLM choice.Source: §8 · p. 36
  • Synonym substitutions produced statistically significant differences in 82.46% of taxonomy mutations; better headwords increased F1 by +0.108 and MCC by +0.154.The authors report these increases as practical effects of selecting more appropriate headwords.Source: §8 · p. 36
  • F1 scores of 0.912 and 0.748 on DS-Pipelines and DASWOW, respectively, show SLMbest was competitive but did not outperform reference classifiers.The methods yielded statistically significant differences; on DASWOW, SLMbest required 20 minutes and 107Wh, raising scaling concerns.Source: §8 · p. 36
  • Classifier choice significantly affected practice insights: SLMbest and reference classifiers produced patterns different from human labeling.The conclusion calls for effective classification methods to analyze data scientists’ practices.Source: §8 · p. 36

Future Works

  • Future work should move beyond static mappings, address cost and scalability, and investigate supervised classifiers using LLM embeddings.The paper reports that DS-Pipelines mapping alone supported 27.22% of code instructions on its own dataset.Source: §8 · p. 36

Ethics Declarations

The authors disclose their funding, report no conflicts of interest, and provide ethics, consent, and availability statements.

Funding

  • The work was funded by the FATES-MLOps international cooperation project and an ANR grant.The stated grants are ANR-24-IAS2-0002-01, NSERC–ALLRP-2024-599951, and ANR-24-CE25-1286-01.Source: Ethics Declarations · p. 37

Conflict of Interest

Ethics and Consent

  • The study did not involve human-subject experiments; laboratory experts helped establish the unified taxonomy, and all provided informed consent.Source: Ethics Declarations · p. 37

Availability

  • The datasets, data, and materials supporting the findings are available in the replication package.The statements provide a Zenodo DOI for the package.Source: Ethics Declarations · p. 37

Appendix

The appendix documents taxonomy definitions and prompt templates for classifying code snippets into data-science lifecycle stages.

Appendix

  • Tables 10–12 present stage definitions for the DS-Pipelines taxonomy, DASWOW taxonomy, and best-performing taxonomy mutation.Table 12 identifies the mutated headword.Source: Table 10 · p. 38, Table 11 · p. 39, Table 12 · p. 39
  • The system prompt frames classification as a data-science expert task and supplies code snippets to classify using a stage taxonomy.Source: Appendix · p. 39
  • The prompt defines Data Collection, Data Preparation, Data Modeling, Model Evaluation, Model Deployment, and Save Results, with unmatched snippets assigned to Others.Source: Appendix · p. 40
  • Examples pair snippets such as DataFrame creation, model fitting, and prediction reshaping with expected stage labels.Source: Appendix · p. 40
  • The user-prompt template asks for a snippet’s classification and includes variants that provide a prior classifier result to improve.Source: Appendix · pp. 37, 41
Loading 2610.10261v1…