Source-linked AI summary
Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence
Sebastian Raschka, Joshua Patterson, Corey Nolet
TL;DR
Machine learning applications require methods and tools that can learn from massive data while remaining computationally effective and usable. This survey reviews Python-based machine learning, scientific-computing libraries, hardware and software paradigms, and major developments across classical ML, deep learning, and related areas. It concludes that Python has evolved into a lingua franca supported by CPU and GPU advances and expanding library ecosystems.
Problem
Machine learning must support complex decision making and massive, growing data volumes while addressing computational and hardware constraints.
Method
The paper surveys major Python machine-learning topics, challenges, approaches, tools, libraries, and computational paradigms across academic and industry contexts.
Results
Python has evolved into a lingua franca for the reviewed research and application areas, supported by advances in CPU and GPU computing and library ecosystems.
Takeaways & Limitations
The review provides a starting point for further research and anticipates that Python will remain dominant for scientific computing for many years.
Takeaways & Limitations
The review does not comprehensively cover all specialized application and research areas or provide exhaustive instruction and library coverage.
Abstract
from arXiv · showhide
Smarter applications are making better use of the insights gleaned from data, having an impact on every industry and research discipline. At the core of this revolution lies the tools and the methods that are driving it, from processing the massive piles of data generated each day to learning from and taking useful action. Deep neural networks, along with advancements in classical ML and scalable general-purpose GPU computing, have become critical components of artificial intelligence, enabling many of these astounding breakthroughs and lowering the barrier to adoption. Python continues to be the most preferred language for scientific computing, data science, and machine learning, boosting both performance and productivity by enabling the use of low-level libraries and clean high-level APIs. This survey offers insight into the field of machine learning with Python, taking a tour through important topics to identify some of the core hardware and software paradigms that have enabled it. We cover widely-used libraries and concepts, collected together for holistic comparison, with the goal of educating the reader and driving the field of Python machine learning forward.
1. Introduction
The paper introduces machine learning in Python as a field shaped by automation needs, growing data volumes, performant libraries, and expanding computational and community ecosystems. It surveys major topics, tools, and trends while remaining a starting point rather than a comprehensive instructional or exhaustive review.
- Motivation: Machine learning automates complex decision making by discovering predictive rules from labeled data without explicit instructions.This extends programming’s goal of automating and augmenting tedious tasks.
- Python’s role: Python’s readability, ecosystem, and interoperability with systems-level languages have made it attractive for scientific computing and machine learning.Most recent machine learning and deep learning libraries are Python-based.
- Scope: CPython executes CPU-bound code in a single thread, while multiprocessing introduces performance trade-offs and PyPy has limited version and library compatibility.These constraints complicate using alternative Python implementations for data science, machine learning, and deep learning.
- Performance foundations: Efficient Python libraries use lower-level Fortran, C/C++, and CUDA implementations to combine usability with computational efficiency as data volumes grow.NumPy and SciPy similarly use lower-level implementations and BLAS where possible to bypass Python threading limitations.
- Community and scale: The Python ecosystem has expanded through data-science communities, cloud infrastructure, scalable data processing, and accelerated computing that make previously intractable workflows feasible.Kaggle connects practitioners and provides empirical evidence about tools used in successful machine-learning projects.
- Scope: The survey reviews major topics, challenges, taxonomies, approaches, tools, and libraries while balancing academic and industry perspectives.It is not intended as comprehensive instruction or an exhaustive list, and it assumes rudimentary Python knowledge plus familiarity with computing, statistics, and machine learning.
2. Classical Machine Learning
Classical machine learning remains important for structured data, and the paper surveys Scikit-learn and compatible extensions for practical modeling, pipelines, imbalance, ensembles, and scalable boosting. These tools emphasize consistent workflows while addressing dataset and computational challenges.
- Context: Classical machine learning remains prevalent because many applications use structured tabular datasets that are unsuitable for contemporary deep learning methods.Classical ML includes decision trees, random forests, support vector machines, and other techniques.
- Scikit-learn: Scikit-learn is a popular open-source library for classical machine learning, and the survey reviews its core library and community extensions.The extensions focus on class imbalance, ensemble learning, and scalable distributed machine learning.
- Scikit-learn: Scikit-learn’s clean, consistent API and documentation support feature engineering and classical modeling on small to medium-sized datasets.The paper defines small datasets as fewer than 1,000 training examples and medium-sized datasets as 1,000 to 100,000 examples.
- Pipelines: Its pipeline API combines preprocessing, feature engineering, and modeling estimators for end-to-end execution, while also supporting common evaluation techniques such as cross-validation.The illustrated pipeline normalizes Iris features, applies principal component analysis, and fits a linear support vector machine.
- Class imbalance: Class imbalance can bias models toward majority labels, motivating extensions that rebalance data through under-sampling, over-sampling, and related techniques.Uniform train-test splitting can produce training and test sets whose class distributions do not represent the original dataset.
- Ensembles: Ensemble methods combine models to improve stability, performance, generalization, or bias-variance control, including bagging, stacking, and gradient boosting.Stacking trains an aggregator on individual model predictions to produce a final prediction.
- Boosting: Gradient boosting is computationally challenging because it is iterative and repeatedly evaluates feature thresholds, encouraging more efficient libraries such as XGBoost and LightGBM.XGBoost approximates greedy split finding with feature histograms, while LightGBM grows trees depth-first.
3. Automatic Machine Learning (AutoML)
AutoML automates stages of machine-learning pipelines, from data preparation and feature engineering through hyperparameter optimization, model evaluation, and neural architecture search. The survey compares major tools and search strategies while highlighting computational trade-offs and the need for standardized evaluation.
- AutoML tools automate one or more machine-learning pipeline stages, reducing repetitive work and helping both non-experts and experienced engineers build models.
- 3. Automatic Machine Learning (AutoML): AutoML libraries differ in supported models and capabilities: AutoKeras targets Keras-based DNNs and neural architecture search, whereas NNI supports neural architecture search, classical ML, and feature engineering.
- 3. Automatic Machine Learning (AutoML): The field lacks a library that automates unsupervised model tuning and selection, and a 2019 open-source benchmark compares AutoML algorithms across 39 classification tasks.
- 3.1. Data Preparation and Feature Engineering: Data preparation may include automatic data-type inference, while feature engineering transforms, scales, encodes, combines, or reduces features before modeling.
- 3.1. Data Preparation and Feature Engineering: Including feature engineering in end-to-end pipeline search can improve pipeline evaluation, but substantially expands the hyperparameter space and may become computationally prohibitive.
- 3.2. Hyperparameter Optimization and Model Evaluation: Hyperparameter optimization uses methods including grid search, random search, Hyperband, Bayesian optimization, SMBO, and evolutionary computation to select configurations and evaluate models.
- 3.2. Hyperparameter Optimization and Model Evaluation: Randomized search can find similarly or better-performing models in a fraction of grid search’s computation time, while Hyperband improves resource utilization through early stopping.
4. GPU-accelerated Data science and Machine learning
GPU acceleration extends Python-based machine learning and data science by exploiting parallel hardware, interoperable libraries, and distributed execution. The section presents RAPIDS and related tools as an ecosystem for end-to-end GPU workflows, including classical ML, ETL, and scalable distributed processing.
- GPU computing: GPUs provide SIMT parallelism and high-speed memory that suit many primitive machine learning operations, including BLAS subroutines.Their parallel architecture makes GPU acceleration a natural fit for algorithms with highly parallelizable computations.
- End-to-end data science: GPU acceleration benefits the full data science pipeline, including input/output and ETL stages beyond machine learning tasks.These stages often contain highly parallelizable transformations that can exploit SIMT processing.
- RAPIDS ecosystem: RAPIDS grows the GPU-accelerated Python ecosystem by supporting existing libraries, filling missing capabilities, and promoting interoperability.It was introduced in 2018 as an open-source effort spanning data science, machine learning, and scientific computing.
- RAPIDS ecosystem: RAPIDS uses GPU-accelerated variants of familiar Python libraries and Apache Arrow to compose workflows while minimizing host-to-device transfers.Its core libraries include cuDF, cuML, and cuGraph as near drop-in replacements for Pandas, Scikit-learn, and NetworkX.
- Interoperability: DLPack and __cuda_array_interface__ enable supported libraries to exchange device-memory pointers without copying or converting underlying data.This supports zero-copy interoperability among libraries such as Numba, CuPy, and PyTorch.
- Classical machine learning: cuML fills gaps in GPU-accelerated Python machine learning by providing Scikit-learn-like preprocessing, feature extraction, model selection, and additional algorithms.Its extra coverage includes time-series algorithms not included in Scikit-learn.
- Distributed data science and machine learning: GPUDirect provides distributed multi-GPU systems with more comprehensive support for writing scalable algorithms than naive workers running local CUDA kernels.The comparison is framed as an optimization advantage for distributed GPU computing.
- Distributed data science and machine learning: RAPIDS supports distributed end-to-end machine learning with Dask, executing training on data-containing workers and broadcasting trained parameters for parallel prediction.The design covers most parametric classical machine-learning model algorithms, with a few exceptions.
5. Deep Learning
Deep learning addresses limitations of classical machine learning by learning representations from raw data, while its frameworks and algorithms continue to balance usability, computational efficiency, and scalability. The field is moving from static toward dynamic graphs and developing methods for larger models and distributed training.
- Motivation: Classical ML remains recommended for tabular datasets but depends on substantial domain expertise for feature engineering.Classical methods are often suboptimal for extracting knowledge from high-dimensional raw data such as text and images.
- Motivation: Deep learning learns salient representations from raw data through its optimization task and model architecture, reducing reliance on manual feature engineering.
- Static Graphs: Static computation graphs support code optimization, model export, and portability, but separating declaration from execution complicates real-time interaction, introspection, and debugging.
- Dynamic Graphs: Dynamic computation graphs enable real-time interaction and easier experimentation, but eager execution has high computational cost and complicates deployment where recent Python versions are unavailable.
- Scaling: 36x growth in ImageNet winner model size from 2014 to 2017 contrasted with approximately 3x GPU-memory growth, creating a single-GPU bottleneck.
- Scaling: 32x less communication per iteration than full-precision distributed SGD was achieved by SignSGD while maintaining a competitive convergence rate.
6. Explainability, Interpretability, and Fairness of Machine Learning Models
Machine-learning explainability and interpretability encompass methods for understanding model mechanics, feature effects, and the generation of predictions. The survey covers post-hoc, intrinsically constrained, visual, and interactive approaches, emphasizing their growing importance as machine learning spreads through industry.
- Concepts: Explainability communicates how model outputs are generated, whereas interpretability concerns understanding model mechanics and observing effects of input or parameter changes.
- Concepts: Local interpretability explains individual data points, while global interpretability provides an overview of a model’s decision process.
- Post-hoc Methods: LIME fits a local linear surrogate to predictions from perturbations around one datum, explaining a nonlinear decision function after training.
- Post-hoc Methods: SHAP uses Shapley values to estimate each feature’s average contribution across predictions, supporting global post-hoc explainability.
- Model Design: Regularization, monotonicity, explanation penalization, invertibility, and tree-based logical formulas provide ways to build interpretability into models or expose their decision structure.
- Interactive Visualization: Interactive visualizations can help researchers explore learned parameters and black-box behavior, including attention mechanisms and embedding neighborhoods in BERT through exBERT.
- Importance: Explainability and interpretability are increasingly important in industry, while classical ML supports feature importance, surrogate modeling, and visualization of deep neural networks.
7. Adversarial Learning
Adversarial learning examines attacks that manipulate inputs or training processes to mislead machine-learning models, with important security and physical-world implications. Python toolkits organize implementations of attack and defense methods, although the review does not exhaustively discuss them.
- Small, often imperceptible image perturbations can fool deep neural networks into making incorrect predictions.
- 100% laboratory and 85% field misclassification rates were reported for sticker-based attacks on stop signs.
- Attacks may occur during training as poisoning attacks or after training during prediction as evasion attacks.
- Python-based libraries include CleverHans, Foolbox, ART, DeepSec, and AdvBox, with varying support for attack and defense mechanisms.
- Table 1 summarizes selected evasion attack and defense mechanisms implemented in adversarial-learning toolkits, while ART also supports poisoning and extraction attacks not shown.
8. Conclusions
The conclusion reviews major advances across machine learning, data science, and scientific computing while identifying specialized areas and emerging directions. It highlights Python's continuing role alongside developments in scalable computing, libraries, reinforcement learning, and quantum computing.
- The review surveys notable advances, challenges, and current solutions across machine learning, data science, and scientific computing.
- Transformer architectures and specialized tools have increasingly dominated natural-language processing in deep learning.
- Graph convolutional neural networks are being applied to molecular structures, while scalable StumPy supports renewed interest in Python time-series analysis.
- Python probabilistic-programming tools span wrappers, probabilistic-modeling libraries, and lower-level deep-learning research frameworks.
- Quantum computing is presented as a future ML direction, but classical simulation is prohibitively slow for real-world applications.
- Reinforcement learning trains agents through trial and error to maximize long-term rewards, but its resource demands make scaling difficult.
- Python is expected to remain dominant for scientific computing, supported by CPU and GPU advances and expanding library ecosystems.
Abbreviations
The manuscript provides a list of abbreviations used throughout the review, covering concepts, models, algorithms, and computing hardware.
- The abbreviation list defines terms including AI, API, AutoML, CPU, DAG, DL, DNN, and CNN.