PaperlayerView exact revision on arXiv

Source-linked AI summary

Marvin: A Toolkit for Streamlined Access and Visualization of the SDSS-IV MaNGA Data Set

Brian Cherinka, Brett H. Andrews, José Sánchez-Gallego, Joel Brownstein, María Argudo-Fernández, Michael Blanton, Kevin Bundy, Amy Jones, Karen Masters, David R. Law, Kate Rowlands, Anne-Marie Weijmans, Kyle Westfall, Renbin Yan

arXiv:1812.03833v1astro-ph.IMastro-ph.GA

TL;DR

Large, spatially interconnected MaNGA data products are difficult to store, query, and explore locally because of their volume and coarse chunking. Marvin addresses this gap with a Python package, API, web application, and remote database, while its abstractions are intended to support similar systems beyond MaNGA.

  • Problem

    MaNGA’s large, spatially interconnected data set is difficult to access, query, visually explore, and analyze at fine scale using locally stored FITS files.

  • Method

    Marvin combines client software, an API, a web application, and a PostgreSQL database with a release-independent data model for accessing MaNGA products.

  • Results

    Marvin provides a toolkit for MaNGA data use and is being abstracted into a foundation for similar systems across astronomy and other scientific disciplines.

  • Takeaways & Limitations

    Marvin’s abstractions are intended to support reusable data-distribution and analysis workflows beyond the MaNGA survey.

  • Takeaways & Limitations

    Local analysis still requires package installation, and Marvin’s single-user focus limits collaborative science pending integration with SciServer.

Abstract

from arXiv · show

The Mapping Nearby Galaxies at Apache Point Observatory (MaNGA) survey, one of three core programs of the fourth-generation Sloan Digital Sky Survey (SDSS-IV), is producing a massive, high-dimensional integral field spectroscopic data set. However, leveraging the MaNGA data set to address key questions about galaxy formation presents serious data-related challenges due to the combination of its spatially inter-connected measurements and sheer volume. For each galaxy, the MaNGA pipelines produce relatively large data files to preserve the spatial correlations of the spectra and measurements, but this comes at the expense of storing the data set in a coarsely-chunked manner. The coarse chunking and total volume of the data make it time-consuming to download and curate locally-stored data. Thus, accessing, querying, visually exploring, and performing statistical analyses across the whole data set at a fine-grained scale is extremely challenging using just FITS files. To overcome these challenges, we have developed \marvin: a toolkit consisting of a Python package, Application Programming Interface (API), and web application utilizing a remote database. \marvin's robust and sustainable design minimizes maintenance, while facilitating user-contributed extensions such as high level analysis code. Finally, we are in the process of abstracting out \marvin's core functionality into a separate product so that it can serve as a foundation for others to develop \marvin-like systems for new science applications.

1. INTRODUCTION

MaNGA combines spatially resolved spectroscopy with a rapidly growing, high-dimensional data volume, making fine-grained access and analysis difficult. Marvin addresses these challenges through integrated software and remote data infrastructure.

  • Data-access challenges: Remote access is necessary because users often cannot store the full data set locally, while selecting useful subsets may itself require querying the whole data set.Bandwidth limitations further constrain how much data can be transferred.
  • MaNGA data set: MaNGA produces hundreds of spectra per galaxy and more than 150 spatially resolved maps spanning diverse physical properties.These measurements are organized in three-dimensional datacubes with two spatial dimensions and one wavelength dimension.
  • MaNGA data set: The survey’s scale reaches over 10,000 galaxies and approximately 10 terabytes in final summary data products.The data include cubes, row-stacked spectra, and derived analysis maps.
  • Data-access challenges: Coarsely chunked files make spatially resolved querying, exploratory analysis, and visualization difficult without extensive manual preparation and cross-matching.This storage structure makes global-property queries easier than fine-grained spatial queries.
  • Marvin: Marvin presents a software toolkit designed to address these challenges through client tools, a web interface, server-side capabilities, and a streamlined MaNGA workflow.The paper describes Marvin as software for accessing, querying, exploring, and analyzing MaNGA data.

2. CORE DESIGN

Marvin redesigns an unsustainable prototype into a toolkit for accessing, querying, visualizing, and analyzing MaNGA data through reusable components and multiple data-access paths. Its abstractions reduce file, database, API, and maintenance overhead while supporting spatially resolved analysis and future reuse.

  • Prototype limitations: The prototype used static visualizations, global-property queries, large galaxy-level files, and manual file creation, limiting interactive exploration, fine-grained retrieval, extensibility, and sustainability.Its web components also lacked command-line equivalents, forcing users to recreate interoperable analysis tools.
  • Design philosophy: Marvin’s design philosophy targets on-demand access, interactive exploration, minimal downloads, spatially resolved queries, and spaxel-level statistical analysis.The framework is intended to support user-developed tools that can be contributed back for code reuse.
  • Core components: Marvin combines a Multi-Modal Access system, REST API, Brain core package, and programmatic DataModel with interconnected Python tools and a web application.The tools provide multiple entry points while allowing users with different expertise to access the same data.
  • Motivation: The system is motivated by MaNGA’s scale, which approaches the storage and analysis limits of ordinary local computing and foreshadows larger future surveys.The final MaNGA release is expected to be of order 10 TB.
  • Multi-Modal Access: The Multi-Modal Access system supports local files, local databases, and remote API calls while preserving consistent tool behavior across data origins.Files offer portability, databases support efficient indexed queries, and APIs return requested subsets without requiring local downloads.
  • Reusable abstractions: Marvin’s Brain abstracts recurring management tasks into reusable base classes, while the DataModel preserves consistent data-product access across releases and formats.These abstractions address logistical barriers such as path management, HTTP requests, SQL access, and changing FITS structures.

3. PROGRAMMATIC TOOLS

Marvin’s programmatic tools unify access to MaNGA data products, metadata, queries, masks, and extensible analysis code. They simplify spatially resolved retrieval and analysis across local files, databases, and remote services.

  • Programmatic tools: Marvin provides Python tools for searching, accessing, downloading, interacting with, and analyzing MaNGA data while connecting its data products through a built-in data model.The tools support repeatable, science-grade analyses and reduce the need to manage many underlying files manually.
  • Data Product Tools: The core Cube, RSS, Maps, and ModelCube tools correspond to major DRP and DAP products and inherit shared functionality from a common core object.They are designed to do more than wrap FITS files by sharing logic such as the Multi-Modal Access system.
  • Sub-Region Tools: Sub-region tools expose individual fibers, spaxels, and bins with plotting functions and access to associated DRP and DAP properties.The Bin tool can also access all spaxels belonging to a bin and generate masks for it.
  • Query Tools: Marvin’s query system uses simplified SQL-like filters and automatically constructs the complex table joins needed to retrieve selected MaNGA properties.Users can query global galaxy properties through a web form or Python class without writing the full SQL statement.
  • Utilities: The Maskbit utility translates mask integers into set bits and labels, supports label-based mask creation, and enables searches on targeting-mask bits.For example, it can identify labels such as “NOCOV” and “DONOTUSE” from a native mask value.
  • Contributed Code: Marvin prioritizes MaNGA interfacing over downstream analysis tools but supports extensions such as BPT classification and user-contributed drop-in analysis code.These extensions can manipulate existing tools, perform analysis, or return plots or data for reuse by others.

4. MARVIN WEB

Marvin Web provides an intuitive browser entry point for visual exploration and querying of MaNGA data. Its pages are backed by the same tools used programmatically, allowing browser workflows to be reproduced locally.

  • Web interface: Marvin Web is designed for quick visual exploration of MaNGA data, while more rigorous analysis remains available through programmatic tools or user scripts.The browser interface aims to reduce the barrier for users new to the field.
  • Web features: The web application includes Galaxy, Query, Plate, and Image Roulette pages for interactive inspection, searching, plate-level browsing, and random galaxy sampling.The Image Roulette page supports browsing the variety in the 10,000-galaxy sample.
  • Galaxy page: The Galaxy page provides point-and-click exploration of galaxy spectra, maps, and NSA metadata.Users can interact with individual galaxies and inspect outputs from the MaNGA DRP and DAP.
  • Query page: The Query page searches DRP and DAP summary-file properties through a simplified interface built on the Marvin Query tool.Its interface combines a text-based filter, selectable return parameters, and a guided query constructor.
  • Web-to-code workflow: Web features map to equivalent Marvin Tool commands, and page-specific code snippets let users reproduce browser actions in a local iPython terminal.The shared tool foundation supports the same galaxy and query capabilities across web and local workflows.

5. MARVIN BACK-END

Marvin’s back end combines layered remote access, lazy loading, and a PostgreSQL database to support interactive visualization, spatial queries, and selective MaNGA retrieval.

  • REST-like API: The REST-like Marvin API offers direct HTTP, a Python Interaction class, and high-level Marvin Tools as three access layers for remote data acquisition.Users can choose low-, mid-, or high-level access depending on how much request handling they want to manage.
  • REST-like API: The Interaction class wraps HTTP requests with exception handling, default parameters, response checks, and conversion of returned data into Python types.This middle layer simplifies remote calls for users who do not want to manage HTTP directly.
  • Remote retrieval: Marvin Tools use lazy loading so the API returns only the information needed for a request and fetches additional data through later calls.This approach is used to minimize server load.
  • Database: The PostgreSQL database underpins interactive visualization, spatially resolved queries, and selective data retrieval for MaNGA metadata and pipeline outputs.It stores DRP and DAP products and target and NSA catalog information across dedicated schemas.
  • Database querying: The Marvin Query page presents simplified filters and guided SQL construction for searching MaNGA data through the web interface.The guided builder is intended for constructing complex queries without manually writing the complete SQL statement.

6. WORKFLOW

Marvin streamlines MaNGA analysis by handling sample selection, data access, linkage, visualization, and interpretation within a flexible ecosystem. It supports remote, targeted access and shared analysis tools while reducing repetitive workflow effort.

  • Marvin streamlines sample selection, data access, data linkage, interaction, and interpretation by enhancing existing steps or automatically handling logistical tasks.The framework is designed to reduce the overhead of the existing MaNGA workflow.
  • Marvin supports rapid preliminary visual exploration through its web interface without requiring data downloads or code.This is especially useful when iterating over selection criteria during exploratory analysis.
  • Marvin enables complex queries on maps and spectra, including searches based on spatially resolved quantities rather than only global properties.The paper gives high central star-formation-rate surface density as an example query.
  • Marvin can download only explicitly requested substructures, such as a single map or spectrum, minimizing unnecessary bandwidth and local disk use.Traditional downloads provide entire data cubes or hundreds of maps in a file.
  • Marvin provides standardized tools for avoiding duplicate downloads and linking map quantities to spectra across local and remote data.The same tools support different access modes, while built-in linkage reduces custom preparation.
  • Marvin's web interface and Python package provide visualization tools for fast exploratory iteration, while its ecosystem supports shared and reusable analysis code.The modular design also allows components to be used independently or within a single Python environment.

7. SUSTAINABILITY

Marvin's sustainability strategy combines accessible distribution, public version control, searchable and versioned documentation, community feedback, error monitoring, and automated testing. These practices are intended to support long-term maintenance and reliable software development.

  • Marvin treats sustainability as dependent on easy installation, understandable code, community interaction, continuous testing, and rapid patch updates.The paper frames these practices as conditions for successful software adoption and development.
  • Marvin is distributed through PyPI, where pip simplifies installation by handling software dependencies automatically.The package-based distribution is intended to reach a wide audience and streamline installation.
  • Marvin uses a BSD 3-Clause open-source license, with code and SDSS dependencies versioned through git and hosted in public GitHub repositories.These choices support transparent software development and broad freedom of use.
  • Marvin's documentation covers public-facing code through docstrings, Sphinx-generated API references, searchable web pages, tutorials, and versioned documentation.Read the Docs regenerates web documentation when the master branch changes and preserves documentation by code version.
  • User feedback from workshops, tutorials, and GitHub Issues has directly improved Marvin's documentation by exposing gaps developers might miss.The paper emphasizes that users provide perspectives distinct from those of developers.
  • Sentry catalogs errors and tracebacks, links failures to relevant software and system information, and can automatically create GitHub Issues.This helps developers monitor crashes and identify failure modes more efficiently.
  • Marvin uses automated testing and coverage tools across Python code, web and API routes, and simulated front-end interactions.The testing infrastructure includes pytest, Travis-CI, Coveralls, Flask testing, Selenium, and Pytest-Flask.

8. SUMMARY AND FUTURE POTENTIAL

Marvin provides an integrated toolkit for accessing, querying, visualizing, and analyzing MaNGA data, while its planned abstractions aim to support reusable systems beyond MaNGA. The release emphasizes remote access, extensibility, sustainability, and future collaboration, while acknowledging current installation and single-user limitations.

  • Summary: Marvin combines the Multi-Modal Data Access System, Python tools, remote querying, a web front-end, and open-source development practices.The MMA switches between local files and a remote database, while Python tools and web interfaces streamline interaction with MaNGA data.
  • Summary: Its remote query system lets users query the entire MaNGA data set from Python without downloading the full sample.
  • Future potential: Marvin is currently specific to MaNGA, but its design is presented as applicable to other astronomy data sets, scientific disciplines, and especially IFU surveys.
  • Future potential: The planned Brain abstraction is intended to provide a reusable framework connecting local and remote access, programmatic tools, querying, and web interfaces.Applications beyond a blank-slate template would subclass the framework and add domain-specific functionality.
  • Limitations and development: Local analysis still requires package installation and Marvin focuses on single-user software usage, limiting collaborative science.The authors describe SciServer integration as a route toward remote, persistent, shareable analysis without local data hosting or installation.
  • Community development: Marvin’s extensible open-source structure allows community members to contribute tools, features, and functions that expand its scientific usefulness.

A.1. Illustration of the Marvin MMA

The Marvin MMA lets a Cube be accessed through a local file, local database, or remote API while returning the same Cube object interface.

  • MMA access: A Marvin Cube can be opened by filename or object ID, with the latter checking local database or file sources before remote access.
  • MMA access: When no local version is found, the MMA switches to remote API mode to retrieve the object.
  • MMA access: All access paths produce the same Marvin Cube instance wrapping the specified DRP datacube.

A.2. Map Plotting

Marvin’s Maps tools support remote map retrieval, arithmetic and logarithmic transformations, instrumental-broadening correction, and customizable multi-panel plotting.

  • Map plotting: Map arithmetic and logarithms are used to construct the [N ii]-to-Hα flux ratio before plotting.
  • Map plotting: The plotting method is simple and customizable, with an optional signal-to-noise threshold shown for the line-ratio panel.

A.3. Metallicity Science Use Case

This use case shows how Marvin selects galaxies, derives gas-phase metallicity from emission-line maps, masks to star-forming spaxels, and visualizes metallicity maps and radial gradients. The resulting gradient plots encode radius relative to effective radius, metallicity, and elliptical azimuthal angle for three galaxies.

  • Spaxel selection: BPT masks select star-forming spaxels, while bad-data and non-star-forming spaxels are combined into the plotting mask.The mask combines Marvin’s BPT classification with the metallicity map mask and the DONOTUSE label.
  • Visualization: The workflow produces gas-phase metallicity maps and radial gradients for star-forming spaxels only.Marvin map arithmetic is used to compute metallicities before plotting the maps and gradients.
  • Sample selection and data access: Marvin queries galaxies by global parameters, converts the results into map objects, and lazily loads [NII] and Hα emission-line maps.The example limits the query to three galaxies selected by stellar mass and g−r color.
  • Visualization: Figure A2 plots metallicity against radius in units of effective radius for three galaxies, with color encoding elliptical azimuthal angle.The displayed galaxies are 7960-3704, 8466-9102, and 8552-12702; the color scale spans 0–360 degrees.
  • Metallicity calculation: Gas-phase metallicity is computed as 12 + log(O/H) from the [NII]/Hα ratio using the Pettini & Pagel (2004) N2 relation.The calculation takes N2 = log10([NII]/Hα) and applies 12 + log(O/H) = 8.90 + 0.57 × N2.

B. Marvin TUTORIALS

Marvin tutorials provide layered, example-driven instruction for accessing and analyzing MaNGA data through the web interface and Tools. They range from an end-to-end [N II]/Hα exercise to plotting, sample selection, and spaxel-level analysis.

  • Tutorial structure: The documentation tutorials use sequential exercises and example Jupyter notebooks to familiarize users with working with MaNGA data via Marvin.Each tutorial builds on previous steps to provide a layered learning experience.
  • Example science cases: The Lean Tutorial demonstrates an end-to-end calculation of the [N ii]/Hα ratio for star-forming spaxels in galaxies with log(M⋆) = 10–11 M⊙.It is presented as an example project using Marvin from start to finish.
  • Example science cases: The Plotting Tutorial progresses from single-map or single-spectrum plots to customized multi-panel map plots.The examples increase in plotting complexity.
  • Example science cases: The Sample Selection Tutorial covers searches across the Primary, Secondary, and Color-Enhanced MaNGA main samples.These samples are identified as components of the MaNGA main sample.
  • Tutorial scope: Additional tutorials cover Marvin’s web interface and Tools, including spaxel spectra, custom-masked maps, unique bins, and binned-spectrum extraction.The main Marvin documentation contains more advanced tutorials than the introductory DR15 examples.
Loading 1812.03833v1…