PaperlayerView exact revision on arXiv

Source-linked AI summary

The Astropy Project: Sustaining and Growing a Community-oriented Open-source Project and the Latest Major Release (v5.0) of the Core Package

The Astropy Collaboration, Adrian M. Price-Whelan, Pey Lian Lim, Nicholas Earl, Nathaniel Starkman, Larry Bradley, David L. Shupe, Aarya A. Patil, Lia Corrales, C. E. Brasseur, Maximilian Nöthe, Axel Donath, Erik Tollerud, Brett M. Morris, Adam Ginsburg, Eero Vaher, Benjamin A. Weaver, James Tocknell, William Jamieson, Marten H. van Kerkwijk, Thomas P. Robitaille, Bruce Merry, Matteo Bachetti, H. Moritz Günther, Thomas L. Aldcroft, Jaime A. Alvarado-Montes, Anne M. Archibald, Attila Bódi, Shreyas Bapat, Geert Barentsen, Juanjo Bazán, Manish Biswas, Médéric Boquien, D. J. Burke, Daria Cara, Mihai Cara, Kyle E Conroy, Simon Conseil, Matthew W. Craig, Robert M. Cross, Kelle L. Cruz, Francesco D'Eugenio, Nadia Dencheva, Hadrien A. R. Devillepoix, Jörg P. Dietrich, Arthur Davis Eigenbrot, Thomas Erben, Leonardo Ferreira, Daniel Foreman-Mackey, Ryan Fox, Nabil Freij, Suyog Garg, Robel Geda, Lauren Glattly, Yash Gondhalekar, Karl D. Gordon, David Grant, Perry Greenfield, Austen M. Groener, Steve Guest, Sebastian Gurovich, Rasmus Handberg, Akeem Hart, Zac Hatfield-Dodds, Derek Homeier, Griffin Hosseinzadeh, Tim Jenness, Craig K. Jones, Prajwel Joseph, J. Bryce Kalmbach, Emir Karamehmetoglu, Mikołaj Kałuszyński, Michael S. P. Kelley, Nicholas Kern, Wolfgang E. Kerzendorf, Eric W. Koch, Shankar Kulumani, Antony Lee, Chun Ly, Zhiyuan Ma, Conor MacBride, Jakob M. Maljaars, Demitri Muna, N. A. Murphy, Henrik Norman, Richard O'Steen, Kyle A. Oman, Camilla Pacifici, Sergio Pascual, J. Pascual-Granado, Rohit R. Patil, Gabriel I Perren, Timothy E. Pickering, Tanuj Rastogi, Benjamin R. Roulston, Daniel F Ryan, Eli S. Rykoff, Jose Sabater, Parikshit Sakurikar, Jesús Salgado, Aniket Sanghi, Nicholas Saunders, Volodymyr Savchenko, Ludwig Schwardt, Michael Seifert-Eckert, Albert Y. Shih, Anany Shrey Jain, Gyanendra Shukla, Jonathan Sick, Chris Simpson, Sudheesh Singanamalla, Leo P. Singer, Jaladh Singhal, Manodeep Sinha, Brigitta M. Sipőcz, Lee R. Spitler, David Stansby, Ole Streicher, Jani Šumak, John D. Swinbank, Dan S. Taranu, Nikita Tewary, Grant R. Tremblay, Miguel de Val-Borro, Samuel J. Van Kooten, Zlatan Vasović, Shresth Verma, José Vinícius de Miranda Cardoso, Peter K. G. Williams, Tom J. Wilson, Benjamin Winkel, W. M. Wood-Vasey, Rui Xue, Peter Yoachim, Chen ZHANG, Andrea Zonca

arXiv:2206.14220v1astro-ph.IM

TL;DR

The Astropy Project faces the challenge of sustaining a community-oriented, open-source ecosystem without a single institution responsible for its maintenance or funding. This article reviews its core-package, governance, community, funding, educational, and ecosystem updates, finding that infrastructure for long-term sustainability and continued growth is being established.

  • Problem

    The Astropy Project lacks a single institution responsible for managing, funding, maintaining, and sustaining its community-developed software ecosystem.

  • Method

    The article reviews updates to astropy, Project governance, contributors, funding, ecosystem interoperability, education, future plans, and challenges.

  • Results

    The Project formalized governance, engaged contributors, received significant formal funding, and expanded its software ecosystem and educational activities.

  • Takeaways & Limitations

    The Astropy Project is establishing infrastructure intended to support its long-term sustainability and continued healthy growth.

  • Takeaways & Limitations

    Critical infrastructure and core functionality remain maintained without redundancy or unmaintained, reflecting continued reliance on volunteer effort.

Abstract

from arXiv · show

The Astropy Project supports and fosters the development of open-source and openly-developed Python packages that provide commonly needed functionality to the astronomical community. A key element of the Astropy Project is the core package $\texttt{astropy}$, which serves as the foundation for more specialized projects and packages. In this article, we summarize key features in the core package as of the recent major release, version 5.0, and provide major updates for the Project. We then discuss supporting a broader ecosystem of interoperable packages, including connections with several astronomical observatories and missions. We also revisit the future outlook of the Astropy Project and the current status of Learn Astropy. We conclude by raising and discussing the current and future challenges facing the Project.

ABSTRACT

The abstract passage consists of affiliation information identifying the Astropy Coordination Committee and an NSF Graduate Research Fellow.

  • The passage identifies the Astropy Coordination Committee.
  • The passage identifies an NSF Graduate Research Fellow.
  • The passage presents two affiliation labels: Astropy Coordination Committee and NSF Graduate Research Fellow.

1. INTRODUCTION

Python’s adoption in scientific computing has supported a broad ecosystem of community-oriented open-source tools, including Astropy’s astronomy-focused core package and interoperable affiliated packages. This article reviews recent core-package, Project, governance, funding, community, and future-planning developments.

  • Background: Python’s scientific popularity reflects volunteer and professional efforts that develop community-oriented open-source tools and foster user and developer communities.These efforts have produced a broad, feature-diverse ecosystem ranging from general-purpose to domain-specific packages.
  • Background: Astropy’s core package provides astronomy functionality for FITS data formats, astronomical coordinates, and physical units.It was among the first large open-source Python packages developed for astronomy.
  • Background: Astropy’s core-package feature development has slowed as specialized packages increasingly expand the ecosystem using its foundations.The core package now represents one component of the broader astronomy Python context.
  • Project scope: The Astropy Project unites the core package, interoperable astronomy-specific tools, and participating users, developers, and maintainers without a single responsible institution.The Project’s evolving needs include sustaining development and maintaining its broader ecosystem.
  • Article scope: The article reviews core-package updates, Project governance, contributors, funding, community expansion, and future plans and challenges.Community development and expansion have become key goals as affiliated packages broaden the ecosystem.

2. MAJOR UPDATES TO THE ASTROPY CORE PACKAGE

Astropy 5.0 expands the core package with new uncertainty, time-series, spectral-coordinate, WCS, interoperability, data-format, timing, and constants capabilities. The release also documents remaining limitations, including incomplete covariance tracking, limited masked Quantity support, and gaps in NumPy compatibility.

  • Uncertainties and Distributions: The new astropy.uncertainties subpackage propagates randomly sampled variable distributions through analyses, producing inspectable final-result distributions, including Quantity arrays.Future development will track covariances for normally distributed uncertainties, while deciding when covariance tracking should stop.
  • Time series: The new astropy.timeseries subpackage extends QTable for sampled or binned time-dependent data and provides folding, resampling, Lomb-Scargle, and box-least-squares periodograms.Masked Quantity support remains limited, hindering masks in QTable.
  • Spectral Coordinates: SpectralCoord simplifies spectral-coordinate conversions, incorporates Doppler-velocity equivalencies, and supports transformations involving observer and target reference frames.A formal APE 14-based high-level WCS interface harmonizes representations needed by missions and observatories such as JWST and Rubin Observatory.
  • Interoperability: Quantity now works with almost all NumPy v1.17+ functions, while Astropy further integrates units into modeling and cosmology through coercion, littleh, and redshift units.Some structured-array and other NumPy compatibility gaps remain.
  • Data formats and I/O: Astropy 5.0 expands data interoperability by interpreting FITS time metadata, supporting MRT, ASDF, Parquet, and QDP tables, updating ECSV to v1.0, and integrating dask with FITS writing.ECSV v1.0 adds fixed- and variable-dimension arrays and simple Python-object columns; dask arrays are computed only during FITS writing.
  • Time and constants: Time now automatically updates leap-second data for correct adjustments, and astropy.constants defaults to SI/CODATA 2018 values, affecting the Astropy ecosystem.LeapSeconds can still be manually applied for reproducibility with older code.

3. MAJOR UPDATES IN THE ASTROPY PROJECT

The Astropy Project strengthened its governance, expanded support for its contributor and user community, pursued greater inclusion, and established a more sustainable financial model. Despite growth in participation, sustaining a pipeline from users to contributors to maintainers remains a central challenge.

  • Governance: Astropy formalized governance around community authority and do-ocracy, and its first CoCo election established new roles and voting members.The election included prior and new members and was intended to broaden coordination and prevent prolonged dominance by the same people.
  • Community: More than 1500 people have contributed to the broader Astropy Project, but 90% of core-package commits in recent eras came from only 18, 34, and 44 contributors.Git commits are a simple but imperfect proxy for effort, and these figures correspond to eras with 115, 216, and 234 contributors, respectively.
  • Community: The total number of top contributors has not grown substantially, highlighting the challenge of sustaining a pipeline from users to contributors to maintainers.Across all eras, 50–60% of software engineers contributing to astropy were or are employed by STScI.
  • Diversity and inclusion: Astropy supported WoCCode peer mentoring and examined representation of marginalized groups, while conference participation identified limited pedagogical resources and expertise as a major barrier.WoCCode participants rated the program’s impact very highly and described greater comfort engaging with coding and an inclusive community.
  • Financial sustainability: The Project established a standing Finance Committee and secured four grants, including approximately $900k and $600k awards supporting infrastructure, development, mentorship, governance, education, and affiliated packages.Funds were administered by NumFOCUS, providing a neutral structure for institutional support; additional support came from Gemini Observatory and the Dunlap Seed Funding program.

4. SUPPORTING THE ECOSYSTEM OF ASTRONOMICAL PYTHON SOFTWARE · 4.1. Community-oriented infrastructure

The Astropy Project supports the broader astronomical Python ecosystem with pre-configured infrastructure for documentation, testing, and package creation. These tools promote interoperable, maintainable community packages and provide pathways for GitHub integration and continuous integration.

  • 4.1. Community-oriented infrastructure: The Astropy Project provides pre-configured infrastructure packages for community software documentation, automated testing, and new-project scaffolding.These packages are intended to help the community support and maintain its own software infrastructure.
  • 4.1. Community-oriented infrastructure: The sphinx-astropy meta-package supplies a default Sphinx configuration and Astropy-specific extensions for community documentation projects.Its extensions support API documentation and NumPy-style docstring parsing.
  • 4.1. Community-oriented infrastructure: The pytest-astropy meta-package provides a unified testing framework compatible with Astropy- and non-Astropy-affiliated community packages.It incorporates plugins for custom test headers, remote data, documentation interoperability, and dangling file-handle checking.
  • 4.1. Community-oriented infrastructure: The Astropy Package Template facilitates creation of new Python packages that leverage the Astropy ecosystem.It uses Cookiecutter to guide users through package generation.
  • 4.1. Community-oriented infrastructure: The package template generates projects with documentation and testing support included.This reduces the setup required for new packages built within the Astropy ecosystem.
  • 4.1. Community-oriented infrastructure: The package template can configure GitHub interoperability, repository access from documentation, and an example GitHub Actions continuous-integration workflow.The workflow demonstrates use of GitHub’s continuous integration tooling.

4.2. Astropy affiliated packages

Astropy affiliated packages extend the astropy core package while supporting code reuse, interoperability, testing, and documentation. Since 2018, the ecosystem has expanded in gravitational astrophysics and existing packages have gained substantial improvements, including new data access and release practices.

  • Definition: Affiliated packages extend astropy with astronomy-related functionality and support code reuse, interoperability, testing, and thorough documentation.They are developed and maintained by their respective package communities rather than solely by the Astropy Project.
  • Ecosystem expansion: The affiliated ecosystem has expanded with PyCBC, lenstronomy, ligo.skymap, and EinsteinPy, covering gravitational-wave signals, strong-lens modeling, probability-map visualization, and general relativity.These additions broaden affiliated-package support for gravitational astrophysics.
  • Package improvements: Roughly a dozen new missions and data services, including the JWST archive, became accessible through astroquery, which also adopted continuous PyPI publication from the main development branch.Formal releases still occur a few times per year.
  • Package improvements: The first stable photutils release indicates that its API will change less frequently.The passage identifies this stable release as a substantial improvement among coordinated and affiliated packages.

4.3. Connections with data archives

Astroquery is the Astropy-coordinated package for querying more than 50 online astronomical-data archives and databases. Its interfaces complement Virtual Observatory tools implemented through the Astropy-coordinated pyvo package by resembling familiar web interfaces.

  • Connections with data archives: More than 50 astroquery modules query astronomical databases and online archives of varying sizes.Astroquery is coordinated by Astropy for interacting with online archives of astronomical and related data.
  • Connections with data archives: Recent astroquery contributions have come from major archives, including ESA and MAST at the Space Telescope Science Institute.
  • Connections with data archives: Many astroquery tools rely on Virtual Observatory tools implemented through pyvo, another Astropy-coordinated package.The passage notes that pyvo has also recently become an Astropy-coordinated package.
  • Connections with data archives: Although pyvo can directly provide many astroquery functions, astroquery interfaces more closely resemble familiar web interfaces for most users.

4.4. Connections with Observatories and Missions

Astropy’s ecosystem is closely integrated with observatory and mission software, supporting JWST, CTA, and Australian Virtual Observatory data workflows while receiving substantial development contributions in return.

  • JWST: JWST pipelines, exposure-time calculators, and data-analysis tools depend on astropy and affiliated packages, with STScI developers contributing since 2010.JWST developers also led or substantially contributed to photutils, specutils, synphot, and ASDF, which supports JWST data products and workflows.
  • JWST: JWST developers have led photutils development and contributed significantly to specutils, while STScI created synphot and initially developed ASDF for JWST-related serialization.These packages support source detection, photometry, spectroscopy, synthetic photometry, and serialization of JWST WCS objects with Astropy models, units, and coordinates.
  • CTA: CTA’s low-level prototype ctapipe and high-level science tools based on gammapy both make heavy use of Astropy for core data-analysis functionality.Astropy supports units, times, coordinate transformations, tables, and FITS input/output across the CTA analysis software.
  • Data Central: Data Central uses Astropy throughout data ingestion, reduction, access, and visualization, including consistent time and coordinate handling for the AAT Archive.Its archives also use specutils loaders to standardize hosted spectra for reuse with common visualization and analysis tools.

4.5. Learn Astropy

Learn Astropy combines astronomy-focused Python education with searchable online infrastructure and community workshops. The Project is expanding the platform, tutorial catalog, and access to third-party educational content.

  • Learn Astropy: Learn Astropy develops online content and workshops covering astronomy-specific coding tasks in Python.Its content includes tutorials published as HTML from Jupyter Notebooks, alongside other educational formats.
  • Learn Astropy: 2021’s relaunch introduced full-text search and interactive filtering to improve Learn Astropy content discovery as the catalog expands.The infrastructure uses Algolia, with Learn Astropy Librarian populating and heuristically indexing metadata for formats such as Jupyter Notebooks and Jupyter Books.
  • Learn Astropy: 19 interactive Jupyter Notebook tutorials are currently hosted as static HTML pages across general and instrument-specific astronomical topics.Examples include FITS files, astronomical coordinate systems, and spectroscopic data from UVES.
  • Future directions: Future plans prioritize improving content sharing, encouraging astropy and affiliated-package materials, and indexing third-party tutorial series through a unified interface.These plans build on the stabilized backend infrastructure and aim to broaden discoverability of astronomy-specific educational content.
  • Community engagement: Learn Astropy workshops at American Astronomical Society winter meetings began with AAS 225 and shifted online during the coronavirus pandemic into basic and advanced sessions.Before the pandemic, full-day workshops had as many as 90 participants and a dozen facilitators.
  • Community engagement: The Project also engaged the community through NumFOCUS Sponsored Projects booths at AAS Meetings 235 and 237, offering Q&A sessions and webinars about open-source astronomical tools.Exhibit-hall funding came alternately from NumFOCUS and the Moore Foundation.

5. FUTURE PLANS FOR THE ASTROPY PROJECT

The Astropy Roadmap tracks agreed priorities for improving the Project’s health and stability, while ongoing challenges include sustaining maintainers, infrastructure, diversity, and communication with varied user communities.

  • Astropy Roadmap: The Astropy Roadmap records high-level actionable priorities and is regularly revisited at coordination meetings alongside a project board of specific issues.The document is static but updated as needed to track progress and define new versions.
  • Astropy Roadmap: Roadmap items are color-coded by required resources, with green items underway and supported by completion plans.Improving documentation and educational-material discoverability through the Learn website is given as an underway example.
  • Supporting an active group of maintainers: Maintaining Astropy relies on a roughly even split between volunteer effort and institutional support across code, educational materials, coordinated packages, and infrastructure.The Project includes the core package, Learn Astropy repositories, and Project infrastructure packages.
  • Supporting an active group of maintainers: The astropy.io.fits subpackage is largely in maintenance mode and maintained by just one contributor despite its significant codebase and user community.The Project has funding for longer-term sustainability, but many needs require sustained funding rather than short-term contracts.
  • Diversity of users, contributors, maintainers, and coordinators: Astropy faces underrepresentation among non-English-speaking or -teaching users and maintainers, despite global use of its core-package documentation.Figure 5 reports users per country accessing the documentation in 2021 and shows voting and coordination membership concentrated in the United States and, less so, Europe.
  • Long-term and sustained funding for maintaining infrastructure: The Project must secure sustained support for low-visibility infrastructure and develop broader communication with intersecting communities whose needs are only partly understood.Infrastructure includes CI, test plugins, documentation builds, and package delivery; relevant communities include operations, archives, researchers, students, educators, and the public.

6. CONCLUSIONS

The Astropy Project remains an active, community-oriented project with growing reach and impact, while establishing infrastructure for long-term sustainability and healthy ecosystem growth. Its continued operation depends on contributions, external organizations, and services supporting development and documentation.

  • Conclusions: The Astropy Project has broad and growing impact across astronomy, astrophysics, and wider scientific software communities.It continues to serve as an active, community-oriented software project with wide reach.
  • Conclusions: The Project is establishing infrastructure to support long-term sustainability and continued growth of its software ecosystem, developers, and users.This infrastructure is intended to ensure long-term sustainability and healthy community growth.
  • Conclusions: Google Summer of Code funds several students per year to work on Astropy-related projects, and participants often become long-term contributors.The community also receives support from organizations outside traditional academic institutions.
  • Conclusions: Astropy relies heavily on GitHub, Azure Pipelines, CircleCI, and Read the Docs for code hosting, continuous integration, and documentation.These web services are described as essential to the packages’ current form.

APPENDIX · A. LIST OF AFFILIATED PACKAGES

Appendix A provides a registry of Astropy-affiliated packages, listing each package’s PyPI name, maintainers, and associated citations. The registry spans packages supporting diverse astronomical analysis, visualization, modeling, data access, and scientific workflows.

  • A. LIST OF AFFILIATED PACKAGES: The registry identifies affiliated packages by package name, PyPI name, maintainer(s), and citation(s).The table is explicitly titled “Registry of affiliated packages” and provides these four columns.
  • A. LIST OF AFFILIATED PACKAGES: Affiliated packages include gwcs, gala, feets, HENDRICS, galpy, Glue, Halotools, and gammapy.These entries are listed with maintainers and, where shown, literature citations.
  • A. LIST OF AFFILIATED PACKAGES: The registry also covers ginga, marxs, lenstronomy, naima, linetools, mocpy, ligo.skymap, and pyspeckit.These entries extend the list across visualization, instrument simulation, lens modeling, spectral analysis, coordinate, and gravitational-wave workflows.
  • A. LIST OF AFFILIATED PACKAGES: The appendix shows that the affiliated-package registry spans multiple continued pages.Repeated “Table 1 continued on next page” markers and repeated column headers document the table’s continuation.
  • A. LIST OF AFFILIATED PACKAGES: Additional entries include poliastro, PyDL, omnifit, PyCBC, PyVO, photutils, regions, pyregion, spectral-cube, and sncosmo.The table continues across pages and retains the same package, PyPI, maintainer, and citation structure.
  • A. LIST OF AFFILIATED PACKAGES: The registry further lists SpectraPy, sbpy, reproject, specutils, synphot, statmorph, stingray, and spherical_geometry.These entries include package names, corresponding PyPI names, maintainers, and citations where provided.
Loading 2206.14220v1…