Source-linked AI summary
SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, SciPy 1. 0 Contributors
TL;DR
Scientific computing in Python needed a broadly capable, reliable foundation, and SciPy developed into that role through an overview of its architecture, development practices, and technical improvements. SciPy 1.0 combined extensive adoption with continued performance work, while its roadmap identified modernization and documentation needs.
Problem
Scientific Python needed fundamental algorithms and a maintainable scientific software foundation spanning common numerical problems.
Method
The paper surveys SciPy’s capabilities, architecture, development practices, and recent technical improvements, including Python-centered implementations supported by compiled code.
Results
13,096,468 SciPy downloads from PyPI and 5,776,017 via conda were recorded during 2017, establishing a lower bound on total user downloads.
Takeaways & Limitations
SciPy functions as widely used scientific-computing infrastructure, with 610 contributors, more than 19,000 accepted commits, and extensive downstream adoption.
Takeaways & Limitations
Distributed computing and GPU support are explicitly outside SciPy’s scope, and the project continues to identify modernization and tutorial gaps.
Abstract
from arXiv · showhide
SciPy is an open source scientific computing library for the Python programming language. SciPy 1.0 was released in late 2017, about 16 years after the original version 0.1 release. SciPy has become a de facto standard for leveraging scientific algorithms in the Python programming language, with more than 600 unique code contributors, thousands of dependent packages, over 100,000 dependent repositories, and millions of downloads per year. This includes usage of SciPy in almost half of all machine learning projects on GitHub, and usage by high profile projects including LIGO gravitational wave analysis and creation of the first-ever image of a black hole (M87). The library includes functionality spanning clustering, Fourier transforms, integration, interpolation, file I/O, linear algebra, image processing, orthogonal distance regression, minimization algorithms, signal processing, sparse matrix handling, computational geometry, and statistics. In this work, we provide an overview of the capabilities and development practices of the SciPy library and highlight some recent technical developments.
Introduction
SciPy provides fundamental numerical building blocks for scientific computing in Python and has grown into widely adopted infrastructure. Its development combined open tooling, a focused community, and contributions from many individuals.
- SciPy supplies algorithms and data structures for modeling and solving scientific problems, while supporting higher-level libraries such as scikit-learn and scikit-image.It is built on NumPy and is used in gravitational-wave analysis and the M87 black-hole imaging project.
- SciPy 1.0 marked a conventional API-maturity milestone for a project already described as a standard followed by others and widely adopted in research and industry.
- The project began with little funding and was written mainly by graduate students, making its eventual prominence appear anomalous against heavily funded research-software ecosystems.
- Open-tool motivations and a friendly, focused community supported SciPy’s growth and an ecosystem emphasizing implementation, documentation, testing, and better development practices.
- SciPy’s history reflects contributions from very many people, each supplying an important part of the project’s development.
Background
Scientific Python developed from a general-purpose language and numerical array ecosystem into an integrated scientific software stack. SciPy consolidated broad numerical functionality while adoption, infrastructure, and community institutions expanded around it.
- Python attracted scientists by combining readability and rapid prototyping with the ability to wrap C and Fortran libraries without requiring low-level programming.
- Numeric emerged from Matrix-SIG discussions as a multidimensional array package intended to support interoperable numerical functions.
- SciPy was created in 2001 to simplify the scientific tool stack by subsuming packages covering signal processing, special functions, sparse matrices, quadrature, optimization, and Fourier transforms.
- Scientific Python progressed from graduate-student side projects to infrastructure used by national laboratories and research institutes, including LLNL’s compute-intensive parallel efforts.
- By the middle to late 2000s, SciPy matured through broader adoption, larger conferences, improved processes, and a narrower library scope alongside a growing ecosystem of scikit packages.
- At the time of writing, SciPy contained nearly 600,000 lines of code in 16 subpackages, with over 110,000 GitHub repositories and 6,500 packages depending on it.
Architecture and implementation choices
SciPy organizes fundamental scientific algorithms into 16 subpackages, balancing broad mathematical coverage with a defined scope and maintainable implementations. Its architecture combines Python with established compiled languages and field-tested libraries.
- SciPy’s scope covers fundamental algorithms for scientific computing, while specialized ecosystem projects provide advanced statistics, machine learning, Bayesian modeling, symbolic computation, and broader image processing.
- New functionality is selected for relevance across multiple scientific fields and demonstrable importance, such as textbook status or substantial peer-reviewed use.
- SciPy targets in-memory computing on single machines across data types and process architectures, explicitly excluding distributed computing and GPU support.
- Its 16 subpackages group mathematical building blocks, data structures, algorithms, and higher-level data-analysis functions.
- The package collection includes clustering, constants, FFTs, integration, interpolation, file I/O, linear algebra, image processing, optimization, signal processing, sparse matrices, spatial algorithms, special functions, and statistics.
- Architecture and implementation choices: The misc subpackage remained in SciPy 1.0 despite efforts to deprecate or relocate its contents and remove it.
Key technical improvements
SciPy 1.0 introduced technical improvements across optimization, sparse data structures, spatial algorithms, statistical distributions, interpolation, and compiled-code integration. These developments expanded functionality, improved performance, and strengthened interfaces for scientific computing.
- Data structures: Sparse-matrix internals were rewritten: CSC and CSR iteration and slicing became up to 35% faster, while new operators, utilities, random generation, and NumPy-aligned APIs expanded sparse functionality.SuperLU was updated to version 5.2.1 as part of the low-level improvements.
- Spatial algorithms: cKDTree was rewritten in C++, gained periodic boundary conditions, released the GIL for multithreading, and improved query performance while preserving approximately loglinear time complexity.Sparse distance matrices support arbitrary Minkowski p-norms, and weighted dual-tree counting supports applications such as galaxy correlation functions.
- Unified bindings to compiled code: SciPy added Cython wrappers for BLAS, LAPACK, and special functions, plus LowLevelCallable interfaces that reduce Python-to-compiled-code calling overhead.The wrappers can avoid requiring users to manage Fortran compilers, ABI differences, and name-mangling schemes.
- Numerical optimization: SciPy 1.0 added a homogeneous self-dual interior-point method for continuous linear programming that solves more than 90 NETLIB benchmark problems.The method also provides certificates of infeasibility or unboundedness.
- Numerical optimization: The new interior-point method is faster than the existing simplex method except on the smallest problems and handles medium- and large-sized problems where simplex fails.Its solutions are typically near the center of an optimal face, motivating future crossover or sparse-simplex work.
- Numerical optimization: SciPy expanded optimization and interpolation support with differential evolution, 196 global optimization benchmarks, four trust-region minimization methods, and modular polynomial interpolators.The interpolator design supplies reusable piecewise-polynomial objects and construction algorithms.
Project organization and community
SciPy combines formal governance, maintainers, contributors, and downstream collaboration to sustain development and cross-project improvement.
- Governance: An 18-member Steering Council oversees daily development, with merge decisions deferred when substantive community objections remain.
- Governance: SciPy’s Code of Conduct defines five guidelines for openness, empathy, collaboration, inquisitiveness, and careful wording.
- Maintainers and contributors: Each six-month release cycle involves approximately 100 contributors, supported by 15 active maintainers who review contributions and advance the project.
- Downstream projects: The ecosystem supports reciprocal contributions between domain-specific projects and SciPy, including functionality first developed in an Astropy-affiliated package.
Discussion
SciPy has substantial developer activity, user adoption, and credibility, while its roadmap identifies documentation, modernization, and developer-retention needs.
- Community and adoption: 610 contributors had accepted more than 19,000 commits, while GitHub recorded roughly 20,000 unique visitors and 721 codebase clones in two weeks.
- Community and adoption: 13,096,468 PyPI downloads and 5,776,017 conda downloads in 2017 provide a lower bound on SciPy’s user downloads.
- Community and adoption: SciPy is used by 47% of machine learning projects on GitHub and is associated with LIGO/Virgo gravitational-wave observation and distribution through macOS and Intel Python.
- Future work: The post-1.0 roadmap identifies improvement areas, including expanding beyond 15 usage-tutorial sections and modernizing low-level Cython code with typed memoryviews.
- Future work: Developer turnover can reduce institutional memory and create repeated mistakes, inconsistent APIs, and drifting project scope.
Data Availability
SciPy’s source code and most data generated for the study are publicly available through GitHub and other cited repositories.
- All SciPy library source code and most study-generated data are available in the SciPy GitHub repository, with supporting code and data in other public repositories.
Author Contributions Statement
The authors identify leadership, manuscript, development, and review responsibilities across the SciPy project and paper.
- Project leaders, creators, manuscript composers, core developers, and all authors’ code, documentation, expertise, and review contributions are specified.
Consortium
The section lists contributors to SciPy 1.0 and provides their institutional affiliations. The affiliations include universities, research laboratories, and independent researchers across multiple locations.
- SciPy 1.0 includes a large contributor list spanning numerous named researchers.The listed contributors include Aditya Vijaykumar, Alessandro Pietro Bardelli, Alex Rothberg, and many others.
- Contributors are affiliated with universities and research institutions in India, Germany, the United States, France, and other locations.The affiliations name institutions such as the Tata Institute of Fundamental Research, the University of Bremen, and the University of Illinois at Urbana-Champaign.
- The affiliations also include an independent researcher and national or applied research laboratories.The list includes an Independent Researcher in Cologne and Lawrence Berkeley National Laboratory, among other organizations.