PaperlayerView exact revision on arXiv

Source-linked AI summary

Array Programming with NumPy

Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, Travis E. Oliphant

arXiv:2006.10256v1cs.MSstat.CO

TL;DR

Modern scientific data and hardware outgrew NumPy’s in-memory, single-machine model, while specialized NumPy-like projects fragmented interoperability. The paper presents NumPy’s array protocols and coordination mechanisms, establishing it as a standard tensor-computation API and central link between Python array types and technologies.

  • Problem

    Modern datasets and specialized hardware exceed NumPy’s direct capabilities, while proliferating NumPy-like APIs burden users and threaten to fragment the scientific Python community.

  • Method

    The paper describes NumPy’s ndarray and array protocols, enabling external array objects to interoperate with NumPy functions through central coordination and dispatch.

  • Results

    NumPy has become the standard API for tensor computation and a central coordinating mechanism between Python array types and technologies.

  • Takeaways & Limitations

    NumPy functions and semantics can support specialized arrays, allowing users to reuse NumPy-oriented code across array providers.

  • Takeaways & Limitations

    RandomState’s compatibility guarantee slows progress because improving its underlying code requires extending the API with additional keyword arguments.

Abstract

from arXiv · show

Array programming provides a powerful, compact, expressive syntax for accessing, manipulating, and operating on data in vectors, matrices, and higher-dimensional arrays. NumPy is the primary array programming library for the Python language. It plays an essential role in research analysis pipelines in fields as diverse as physics, chemistry, astronomy, geoscience, biology, psychology, material science, engineering, finance, and economics. For example, in astronomy, NumPy was an important part of the software stack used in the discovery of gravitational waves and the first imaging of a black hole. Here we show how a few fundamental array concepts lead to a simple and powerful programming paradigm for organizing, exploring, and analyzing scientific data. NumPy is the foundation upon which the entire scientific Python universe is constructed. It is so pervasive that several projects, targeting audiences with specialized needs, have developed their own NumPy-like interfaces and array objects. Because of its central position in the ecosystem, NumPy increasingly plays the role of an interoperability layer between these new array computation libraries.

NumPy arrays

NumPy arrays combine a compact multidimensional data structure with metadata for interpreting memory and an expressive API for indexing, vectorized operations, broadcasting, and array-aware functions. Together, these features form a productive array programming language that supports scientific computation.

  • Array representation: A NumPy array stores multidimensional data with metadata specifying its data type, shape, and strides.The array uses a pointer to memory, while its metadata describes how the stored data should be interpreted.
  • Array representation: A single array data type ensures every element occupies the same number of bytes, supporting numeric, textual, temporal, and object data.Examples include real and complex numbers, strings, timestamps, and pointers to Python objects.
  • Array representation: Shape specifies the number of elements along each axis, while the number of axes defines dimensionality, from vectors to four-dimensional color videos.A vector can have shape N, whereas color videos can have shape (T, M, N, 3).
  • Array representation: Strides map linearly stored memory onto multidimensional arrays by specifying byte jumps between rows, columns, and other axes.For a (4, 3) array of 8-byte elements, the strides are (24, 8).
  • Array programming: Indexing, operators, and array-aware functions provide a readable high-level API for accessing data and performing fast array operations.Subarray indexing can return views that share data with the original array, limiting memory usage, while vectorization expresses whole-array calculations concisely.
  • Array programming: The combination of simple in-memory representation, mathematical syntax, and utility functions forms a productive and expressive array programming language.NumPy also supports broadcasting and array-aware operations across differing dimensions and axes.

Scientific Python ecosystem

NumPy’s fast array operations and ecosystem of scientific libraries provide a unified foundation for scientific computing in Python, from exploratory analysis to application-specific research software. Its broad adoption has made NumPy and its API ubiquitous across scientific disciplines and educational settings.

  • Scientific Python ecosystem: Fast array operations and linear algebra let scientists perform their work within Python, an open-source language that is easy to learn and teach.Python supports standard tasks including data cleaning, web interaction, and text parsing.
  • Scientific Python ecosystem: NumPy, SciPy, Matplotlib, and interactive environments such as IPython or Jupyter form a foundation for array programming and exploratory data analysis.Users inspect, manipulate, visualize, and iteratively refine data before organizing statements into programs or notebooks.
  • Scientific Python ecosystem: Application-specific libraries build on NumPy and related tools for research workflows, including radio interferometry imaging, astronomical data handling, coordinate transformations, and visualization.The eht-imaging example relies on lower-level ecosystem components, while Astropy handles astronomical formats and transformations and Matplotlib generates the final black-hole image.
  • Scientific Python ecosystem: Software engineering practices such as version control, unit testing, code review, and issue tracking have spread through the NumPy ecosystem and train researchers in professional-standard development.Astropy provided hundreds of junior scientists with this experience, while community courses and workshops address gaps in formal university education.
  • Scientific Python ecosystem: The growth of data science, machine learning, and artificial intelligence has expanded scientific Python across natural and social sciences, where NumPy and its API are now ubiquitous.NumPy and its ecosystem are commonly taught in university courses, boot camps, and summer schools.

Array proliferation and interoperability

NumPy’s in-memory CPU array model led to a proliferation of specialized array implementations as datasets exceeded single-machine memory and accelerator hardware emerged. NumPy responded by becoming a central interoperability layer through protocols that dispatch operations across external arrays while preserving its high-level API.

  • Array proliferation and interoperability: NumPy’s in-memory data model cannot utilize datasets exceeding one machine’s memory or specialized accelerators such as GPUs, TPUs, and FPGAs.Scientific datasets may be distributed across machines or the cloud, while deep learning and artificial intelligence have driven demand for accelerator hardware.
  • Array proliferation and interoperability: These limitations prompted a proliferation of array implementations, including framework-specific, sparse, distributed, and NumPy-based arrays.Examples include PyTorch, TensorFlow, Apache MXNet, JAX, SciPy, PyData/Sparse, and Dask.
  • Array proliferation and interoperability: NumPy added a central coordination mechanism with a well-specified API so users could switch among specialized arrays with minimal code changes.The goal is for NumPy functions or semantics to operate on NumPy, GPU, distributed, and other specialized arrays.
  • Array proliferation and interoperability: Over four hundred popular NumPy functions use protocols that dispatch operations to originating libraries, including Dask, CuPy, xarray, and PyData/Sparse.The protocols act as contracts of operation for passing specialized arrays to NumPy functions.
  • Array proliferation and interoperability: NumPy protocols compose across distributed and multi-GPU systems, enabling highly parallel execution on multiple systems with millions of cores and minimal code changes.For example, CuPy arrays can be embedded in Dask arrays while users continue using NumPy’s high-level API.

Discussion

NumPy has grown from an open-source Python array library into core scientific infrastructure, a standard tensor-computation API, and a coordinator across array types and technologies. Its continued leadership will require interoperability work, sustained funding, and new contributors to address evolving hardware and expanding scientific data.

  • Capabilities: NumPy combines expressive array programming, C performance, and Python readability in a mature, well-tested, well-documented, community-developed library.Its simple memory model supports hand-optimized low-level code, while array protocols enable specialized hardware acceleration with minimal changes to existing code.
  • Evolution: What began as adding an array object to Python became core scientific infrastructure on which substantial scientific work depends.NumPy is expected to remain correct, fast, and stable as its role has expanded beyond a small community project.
  • Interoperability: NumPy has become the standard API for tensor computation and a central coordinating mechanism between Python array types and technologies.Work continues to expand and improve these interoperability features.
  • Future challenges: Evolving specialized hardware, more data-science practitioners, and expanding scientific data collection will shape NumPy’s challenges over the next decade.The paper identifies diminishing returns on Moore’s law and instruments such as light sheet microscopes and LSST as drivers of change.
  • Future challenges: NumPy is poised to continue leading interactive scientific computation, but doing so requires sustained funding and a new generation of graduate students and developers.These contributors will need to build NumPy for the needs of the next decade of data science.

Methods · Library organization

NumPy’s development uses GitHub-centered review, testing, issue triage, and integration workflows. Its organization centers on ndarray and ufuncs, supported by scientific-computing libraries, infrastructure utilities, and the f2py Fortran-wrapping tool.

  • Methods: Git and GitHub provide NumPy’s version-control, hosting, fork-based development, pull-request review, testing, and upstream integration workflow.Pull requests enable code review and extensive testing before modified code is integrated.
  • Methods: GitHub’s issue-tracking system collects and triages problems and proposed improvements.
  • Library organization: NumPy comprises ndarray, universal functions, scientific-computing and array-manipulation functions, infrastructure libraries, and f2py.ndarray and universal functions are generally considered the library’s core.
  • Library organization: The ndarray stores regularly strided homogeneous data in contiguous memory for efficient representation of n-dimensional data.
  • Library organization: C-written ufuncs efficiently loop over arrays and implement broadcasting across compatible input dimensions.For example, inputs shaped 3 and 2 × 1 produce an output shaped 2 × 3.
  • Library organization: NumPy’s computing libraries support array construction and transformation, searching, sorting, counting, statistics, file I/O, FFTs, linear algebra, and random-number generation.The libraries include matrix decompositions, equation solving, probability distributions, and fast Fourier transform functions.
  • Library organization: Infrastructure libraries provide utilities for testing NumPy-array code and configuring, installing, and packaging dependent Python libraries.The testing subpackage includes assert allclose(actual, desired), while distutils supports publication to PyPI.
  • Library organization: F2PY builds NumPy-aware Python wrappers for Fortran functions and remains part of NumPy for historical reasons, although NumPy itself uses no Fortran code.

Governance

NumPy’s governance relies on consensus among contributors, guided by a 12-member Steering Council, and is supported by an official Code of Conduct with procedures for reporting and responding to breaches.

  • Governance: NumPy adopted an official Governance Document on October 5, 2015, establishing consensus-based decisions in which contributors generally have veto power.A 12-member Steering Council facilitates this process and oversees daily development through coding and community contribution review.
  • Governance: NumPy’s official Code of Conduct, approved on September 1, 2018, promotes openness, empathy, collaboration, inquiry, and careful communication.It also specifies how breaches can be reported and how reports are handled.

Funding · Developers

NumPy’s funding expanded from major grants supporting technical sustainability to targeted support for users, contributors, dependencies, documentation, and websites. Its development is maintained by a relatively small active core, despite broad historical participation and increasing contribution activity.

  • Funding: 1.3M USD in 2017 grants supported technical-debt reduction and standards and architecture for more sustainable NumPy development.The Gordon & Betty Moore and Alfred P. Sloan foundations provided the grants, with Stfan van der Walt as PI managing four programmers.
  • Funding: 195K USD from the Chan Zuckerberg Initiative supported beginning-to-intermediate users, contributor growth, and OpenBLAS.Ralf Gommers was the PI, and the grant also supported OpenBLAS for accelerated linear algebra.
  • Funding: Since May 2019, Tidelift has provided annual funding for documentation and website improvements.The passage describes this support as a small annual amount.
  • Developers: 23 contributors currently have commit rights, including 17 active maintainers in 2019 and 4 paid to work full-time.Several long-term developers also maintain specific NumPy components without official maintainer status.
  • Developers: 823 contributors have submitted PRs over NumPy’s history, but a small group of active maintainers shares more than half of all contributions.This indicates that development relies heavily on a limited number of active maintainers.
  • Developers: 450 PRs per release were averaged across the five releases in 2018 and 2019, with each release attracting more than a hundred new contributors.Releases occurred about every half year; although merged PR counts fluctuated, contributions increased over the past years.

Community calls · NumPy enhancement proposals · Central role

NumPy formalized community-informed development through proposal processes, roadmaps, and weekly calls, while NEPs guide major changes and coordinate standards across the scientific Python ecosystem. Its central infrastructure role is reflected in widely adopted documentation practices and shared version-support policies.

  • Community calls: NumPy formalized feature proposals and a development roadmap to address stability demands from its many dependent scientific Python packages.The process incorporated extensive community input and feedback.
  • Community calls: Weekly community calls alternate between triage and higher-level discussion, involving developers, vendors, and other external groups.The calls provide a venue for outside organizations to raise community concerns.
  • Community calls: Development data before mid-2011 are excluded because NumPy development did not occur on GitHub before that point.The contribution data are based on GitHub pull requests, and some maintainer contributions may be uncategorized.
  • Community calls: Figure 4 tracks pull requests merged into NumPy’s master branch each quarter since 2012, separating total activity from maintainer contributions.The lower blue area represents contributions by current or previous maintainers.
  • NumPy enhancement proposals: 19 NEPs had been proposed, including 6 implemented and 4 accepted and being implemented, to manage major changes and document design decisions.NEPs are modeled after Python Enhancement Proposals and collect community input on issues.
  • Central role: NumPy’s docstring standard is widely adopted, and its NEP system helps coordinate the broader scientific Python community.This reflects NumPy’s central role in building and standardizing shared infrastructure.
  • Central role: NEP 29 recommends a common time window-based policy for supporting Python and NumPy versions across the Scientific Python ecosystem.The policy is proposed with leaders from various other projects and is intended to simplify downstream project and release planning.

Wheels build system · Recent technical improvements

NumPy wheels package compiled extensions and binary data in a standard format, simplifying installation through automated, platform-specific builds. Recent funding and coordination processes enabled large-scale changes, including testing-infrastructure updates for hardware platforms.

  • Wheels build system: Python wheels distribute libraries with Python code, compiled C extensions, and other binary data.This avoids requiring users to build NumPy from source with a C compiler and other build tools.
  • Wheels build system: Wheels made NumPy easier for users to install by providing precompiled software instead of requiring local compilation.
  • Wheels build system: A GitHub commit triggers automated NumPy wheel builds for Windows, Mac OSX, and Linux, which are uploaded to a public server.
  • Wheels build system: The wheel-building technology continually evolves, with multibuild serving as a key component when the paper was written.Multibuild was developed by Matthew Brett and other developers.
  • Wheels build system: Multibuild scripts supported continuous integration through Travis-CI for Linux and Mac OSX and Appveyor for Windows.
  • Recent technical improvements: Recent funding and a clear process for coordinating with the developer community enabled NumPy to tackle important large-scale changes.
  • Recent technical improvements: The technical-improvement effort highlighted two large-scale changes and changes to testing infrastructure supporting hardware platforms.

Array function protocol

NumPy aims to provide an interoperability layer for the growing ecosystem of specialized NumPy-like APIs and array objects. Its array ufunc and array function protocols let these arrays work with almost all NumPy functions while preserving provider-specific execution.

  • Array function protocol: Specialized NumPy-like projects make consumer code provider-specific, limiting tool compatibility and complicating transitions from NumPy to specialized arrays.This growing multitude threatened to fragment the ecosystem again.
  • Array function protocol: NumPy seeks to provide the fundamental interoperability API connecting diverse NumPy-like APIs and array objects.The goal addresses compatibility challenges created by specialized array projects.
  • Array function protocol: NumPy 1.13 introduced the array ufunc protocol, and NumPy 1.17 added the more general array function protocol.Together, these protocols allow array providers to interoperate with the NumPy API.
  • Array function protocol: The protocols allow specialized arrays to work correctly with almost all NumPy functions, with operations dispatched back to their providing library.For example, a CuPy GPU array passed to np.sum returns a GPU array.
  • Array function protocol: User-defined functions composed with NumPy can operate on multi-node distributed Dask arrays, returning a Dask array rather than a NumPy array.The example contrasts f(x_local), which returns a NumPy array, with f(x_distr), which works and returns a Dask array.

Random number generation

NumPy’s random-number API evolved from a monolithic, compatibility-focused RandomState to an extensible architecture separating bit generation, variate generation, and seeding. The newer design supports multiple bit generators, improved distribution algorithms, and interoperability with dependent libraries.

  • Legacy API: RandomState guarantees identical values for a fixed seed and call sequence, but this constraint slows improvements to the underlying implementation.The compatibility guarantee continues to apply to RandomState.
  • New API: NumPy 1.17 introduced a flexible API that separates pseudorandom bit generation, distribution-specific variate generation, and seed initialization.Bit generators produce pseudorandom bits, generators transform them into variates, and SeedSequences produce high-quality initial values.
  • Bit generators: NumPy ships four bit generators, including default 64-bit PCG64, 64-bit Philox, 64-bit SFC64, and 32-bit MT19937.MT19937 is the generator used in older NumPy versions.
  • Variate generation: The Generator adds improved sampling methods, including Ziggurat for normal, exponential, and gamma variates and Lemire’s method for bounded random integers.Unlike RandomState, Generator does not guarantee identical streams across versions when improved algorithms are introduced.
  • Extensibility: The component architecture improves extensibility by exposing distribution-transforming C functions through CFFI for reuse in NumPy and dependent libraries such as Numba.SeedSequence can also spawn child sequences that deterministically initialize multiple bit generators.

Testing on multiple architectures

NumPy extended continuous-integration testing across POWER, ARM, and IBM Z architectures, including big-endian behavior, while preparing for broader binary-wheel support on these platforms.

  • Architectural motivation: POWER and ARM support are motivated by Summit and Sierra’s POWER9 architectures, Astra’s ARM-based TOP500 entry, and ARM’s status as the world’s most widely used instruction set architecture.More than 100 billion ARM processors had been produced by 2017.
  • Continuous-integration testing: NumPy’s CI tests now include ppc64le, ARMv8, and s390x architectures, probing library behavior on IBM Z’s big-endian machines.This implements a major component of the project’s improved CI-testing roadmap.
  • Binary distribution: manylinux2014 adds binary-wheel support for ARMv8, ppc64le, and s390x, positioning NumPy for expected future binary provision on these architectures.The plan covers architectures supported by the CentOS Alternative Architecture Special Interest Group.
Loading 2006.10256v1…