PaperlayerView exact revision on arXiv

Source-linked AI summary

The Illustris Simulation: Public Data Release

Dylan Nelson, Annalisa Pillepich, Shy Genel, Mark Vogelsberger, Volker Springel, Paul Torrey, Vicente Rodriguez-Gomez, Debora Sijacki, Gregory F. Snyder, Brendan Griffen, Federico Marinacci, Laura Blecha, Laura Sales, Dandan Xu, Lars Hernquist

arXiv:1504.00362v2astro-ph.COastro-ph.GAastro-ph.HEastro-ph.IM

TL;DR

The paper addresses the need to make complex hydrodynamical simulation data usable for studying baryonic galaxy formation and comparison with observations. It releases the Illustris suite and provides direct files, an API, and supporting documentation for searching, extracting, and analyzing the data. The release is broad and accessible, but the authors emphasize modeling uncertainties and cautions when interpreting results or comparing them with observations.

  • Problem

    Dark-matter-only simulations cannot directly predict gas, stars, and black holes, motivating accessible public hydrodynamical data for observationally relevant studies.

  • Method

    The paper releases Illustris data through native HDF5 files and a REST web API supporting searches, particle extraction, merger-tree access, visualization, and analysis.

  • Results

    The release provides broad access to six Illustris realizations, including 136 snapshots, halo and subhalo catalogs, two merger trees, and supplementary products.

  • Takeaways & Limitations

    Users can analyze Illustris data locally or request reduced products programmatically, with documentation and examples in IDL, Python, and Matlab.

  • Takeaways & Limitations

    Modeling uncertainties and known simulation problems must be considered before drawing scientific conclusions or comparing outputs with observations.

Abstract

from arXiv · show

We present the full public release of all data from the Illustris simulation project. Illustris is a suite of large volume, cosmological hydrodynamical simulations run with the moving-mesh code Arepo and including a comprehensive set of physical models critical for following the formation and evolution of galaxies across cosmic time. Each simulates a volume of (106.5 Mpc)^3 and self-consistently evolves five different types of resolution elements from a starting redshift of z=127 to the present day, z=0. These components are: dark matter particles, gas cells, passive gas tracers, stars and stellar wind particles, and supermassive black holes. This data release includes the snapshots at all 136 available redshifts, halo and subhalo catalogs at each snapshot, and two distinct merger trees. Six primary realizations of the Illustris volume are released, including the flagship Illustris-1 run. These include three resolution levels with the fiducial "full" baryonic physics model, and a dark matter only analog for each. In addition, we provide four distinct, high time resolution, smaller volume "subboxes". The total data volume is ~265 TB, including ~800 full volume snapshots and ~30,000 subbox snapshots. We describe the released data products as well as tools we have developed for their analysis. All data may be directly downloaded in its native HDF5 format. Additionally, we release a comprehensive, web-based API which allows programmatic access to search and data processing tasks. In both cases we provide example scripts and a getting-started guide in several languages: currently, IDL, Python, and Matlab. This paper addresses scientific issues relevant for the interpretation of the simulations, serves as a pointer to published and on-line documentation of the project, describes planned future additional data releases, and discusses technical aspects of the release.

1. Introduction

Illustris addresses the baryonic blind spot of dark-matter-only simulations by modeling gas, stars, and black holes across cosmological volumes. This release makes the resulting simulation data broadly accessible through downloads, an API, and documentation.

  • Motivation: Dark-matter-only simulations cannot directly predict baryonic components such as gas, stars, and black holes.These components underpin much observational astronomy.
  • Motivation: Cosmological hydrodynamical simulations model the coupled evolution of dark matter and cosmic gas while incorporating stars, black holes, feedback, and heavy-element processes.
  • Motivation: Simulations now span volumes of ≳100 Mpc while resolving galaxy structure at ≲1 kpc scales, enabling predictions for diverse astrophysical problems.
  • Data access: Public dissemination of simulation and observational data is crucial for confronting theoretical predictions with measurements across cosmic time.
  • Data access: The Illustris release offers direct access to snapshots, group catalogs, merger trees, and supplementary files, plus an API for searches, extraction, visualization, and analysis.Documentation and examples are provided in IDL, Python, and Matlab.
  • Paper scope: The paper serves as a user guide covering the simulations, data products, access methods, release architecture, scientific cautions, and community considerations.

2. Description of the Simulations

Illustris consists of six realizations of a large cosmological volume, varying resolution and baryonic physics, and uses comprehensive physical models with the Arepo moving-mesh code. The simulations support studies of diverse galaxy-formation phenomena but retain modeling uncertainties and documented limitations.

  • Simulation suite: The six runs share a (106.5 Mpc)^3 volume and cover three resolutions, each with full baryonic physics and a dark-matter-only analog.Illustris-1 is the flagship highest-resolution run.
  • Simulation suite: The simulations evolve dark matter, cosmic gas, stars, and supermassive black holes from z = 127 to z = 0.
  • Scientific scope: Figure 1 illustrates the range of accessible objects and phenomena across dark matter, stars, gas, and black holes at multiple scales.
  • Physical models: Full-physics runs include cooling, star formation, stellar evolution and enrichment, galactic outflows, black-hole growth, and quasar- and radio-mode AGN feedback.
  • Numerical methods: Arepo couples continuum hydrodynamics with self-gravity using an unstructured, moving Voronoi tessellation and Godunov-based fluid integration.
  • Cautions: The released data require careful interpretation because the simulations involve modeling uncertainties and known problems that can affect scientific conclusions and observational comparisons.

3. Data Products

The release provides the full Illustris simulation suite through snapshots, catalogs, merger trees, subboxes, and supplementary products. Data can be inspected and extracted through native files and programmatic access, with organization and unit conventions documented for analysis.

  • Core products: The release contains 136 snapshots from z = 40 to z = 0 for six runs spanning high, intermediate, and low resolution with full-physics and dark-matter-only realizations.Snapshots include up to five resolution-element types.
  • Core products: FoF halo catalogs, Subfind subhalo catalogs, and the SubLink and LHaloTree merger trees are provided for every snapshot and run.Supplementary catalogs include selected stellar-property products.
  • Snapshot organization: Snapshots are stored in chunks, and particle data are sorted by group number, subgroup number, and binding energy rather than spatial position.FoF halos may span chunks, while subhalos are contiguous only within a single group.
  • Snapshot contents: HDF5 snapshots organize gas, dark matter, tracers, stars and wind particles, and black holes into PartType groups, with complete field descriptions documented by particle type.
  • Snapshot contents: The documented unit system uses kpc/h for lengths, 10^10M⊙/h for masses, and km/s for velocities, with scale-factor conversions for comoving quantities.
  • Subboxes: Four subboxes per full-physics run provide higher-time-resolution spatial cutouts, but their spacing is variable and they are not periodic or group-cataloged.

3.2. Group Catalogs

Illustris provides FoF halo and Subfind subhalo catalogs, along with two merger-tree formats that connect subhalos across snapshots. The catalogs and tree links support both object organization and efficient traversal of merger histories.

  • Group Catalogs: Each snapshot has a group catalog containing FoF and Subfind objects, with HDF5 groups for headers, groups, subhalos, and offsets.Particle or cell IDs are not stored directly in the catalogs; snapshot ordering follows group membership.
  • Group Catalogs: FoF groups use linking length b = 0.2 on dark matter particles, with gas, stars, and black holes attached through their nearest dark matter particles.
  • Group Catalogs: Subfind identifies gravitationally bound substructures using all particle types and assigns those particles to subhalos.
  • Merger Trees: SubLink and LHaloTree connect subhalos across snapshots using descendant, first-progenitor, and next-progenitor links, with additional links available in SubLink.The root descendant, last progenitor, and main leaf progenitor links exist only for SubLink trees.
  • Merger Trees: SubLink stores each tree in sequential HDF5 files, while its depth-first IDs and pointers enable efficient extraction of progenitor branches and subtrees.Subhalo IDs connected by any SubLink pointer are guaranteed to reside in the same tree file.

3.4. Supplementary Data Catalogs

The release adds post-processed catalogs focused on Illustris-1 galaxies at z = 0, covering synthetic observables, photometric morphologies, and stellar structural properties. These products have defined stellar-mass or particle-count limits and standardized viewing or measurement configurations.

  • Scope: The supplementary effort focuses on properties of Illustris-1 galaxies at z = 0 above a stellar mass limit of M⋆≳10^9M⊙.
  • Stellar Mocks: Multi-band Images and SEDs: SUNRISE produces synthetic stellar images and integrated spectra, including 36-band spatially resolved maps and integrated SEDs for galaxies with M⋆>10^10M⊙.Approximately 7000 galaxies exceed this higher threshold; lower-mass galaxies receive integrated SEDs down to 500 star particles.
  • Photometric Non-Parametric Stellar Morphologies: The morphology catalog measures Gini, M20, C, rP, and rE in rest-frame u, g, i, and H bands from four viewing directions.It is available for essentially all z = 0 subhalos with M⋆>10^9.7M⊙, with 42531 sources per filter when viewing directions are treated independently.
  • Stellar Circularities, Angular Momenta, Axis Ratios: The stellar-structure catalog provides circularities, angular momenta, and axis ratios for Illustris-1 galaxies above 10^9M⊙ within twice the stellar half-mass radius.Additional circularity fields use the distribution of individual stellar circularity parameters, and analogous quantities are provided for the full stellar content.

4. Data Access

Illustris provides two complementary data-access paths: direct HDF5 downloads for local analysis and a web-based API for programmatic search, extraction, visualization, and analysis. These paths can be combined, with documentation and examples supporting common workflows.

  • Combined workflows: Users can combine local downloads with API requests, for example searching a locally downloaded catalog before extracting selected merger trees or particle data.This workflow avoids downloading full merger-tree or snapshot files when only selected objects are needed.
  • Direct files: Direct downloads provide raw Illustris data products in portable, self-describing HDF5 format for local analysis.Released products include snapshots, group catalogs, merger trees, and supplementary catalogs; large outputs are split into downloadable file chunks.
  • Web-based API: The web-based API reduces response sizes by supporting searches, data extraction, visualization, and analysis without downloading full data files.It can search group catalogs, extract particle data, access merger trees, and request visualization or derived-property functions.
  • Search and extraction: API requests can list simulations and catalog objects, search catalog fields by numeric ranges, retrieve object fields, and return full or selective particle cutouts.Cutouts may include specified particle or cell types and fields, while searches combine restrictions over supported fields.
  • Analysis functions: The API supports merger-tree traversal, halo-subhalo relationship queries, full or selective raw downloads, crossmatching with dark-matter-only analogues, visualization, and derived-property retrieval.Merger-tree functions include complete histories, main progenitor branches, and links across adjacent snapshots.
  • Request details: API searches use AND-combined field restrictions, while cutout queries specify particle types and fields; authentication requires a registered user's API key.Keys may be supplied in the query string or HTTP header, and browser requests are automatically authenticated for logged-in users.

5. Architectural and Implementation Details

The release prioritizes direct downloads and an API facade over raw SQL for complex hydrodynamical data, emphasizing local analysis, data-volume reduction, and accessible programmatic workflows. Its REST API supports discoverable, read-only resources and light requests, while omitting long-running computation and remote user storage.

  • 5. Architectural and Implementation Details: The release limits relational-database use to group catalogs and hides the database behind API endpoints rather than exposing raw SQL queries.Common supported tasks become easier, while complex or specialized queries are unavailable.
  • 5. Architectural and Implementation Details: Hydrodynamical data complexity led the authors to emphasize local processing and API-based data-volume reduction before transfer.Examples include downloading one galaxy’s particles or merger history without acquiring an entire snapshot or merger tree.
  • 5. Architectural and Implementation Details: A survey of 125 field respondents found that search, cutout, and programmatic download interfaces were preferred over raw SQL; direct HTTP downloads were favored, especially FITS and HDF5.Approximately 70% of respondents were graduate students, postdocs, or faculty, evenly split between observers and theorists.
  • 5. Architectural and Implementation Details: The REST API is read-only and stateless, uses HTTP GET requests and URL-identified endpoints, and returns JSON by default.Its HATEOAS design lets users discover resources through absolute URLs without knowing the API structure in advance.
  • 5. Architectural and Implementation Details: HDF5 was selected as a practical, self-describing binary format aligned with the simulation’s native output and supporting in-memory hyperslab extraction.The trade-off is that requested responses must be temporarily loaded into memory rather than streamed.
  • 5. Architectural and Implementation Details: The system supports light queries expected to finish within a few seconds but provides no queued batch queries, long-running jobs, or per-user remote storage.The back end currently focuses on storage and delivery, without temporary guest access to local compute resources.

6. Scientific Remarks and Cautions

Illustris reproduces many observed galaxy and gas properties, but its broad dynamical range and complex physics introduce modeling uncertainties and specific tensions with observations. Users should account for these cautions before drawing scientific conclusions or comparing outputs with data.

  • 6. Scientific Remarks and Cautions: Illustris-1 reproduces many observational facts about galaxy demographics, galaxy properties, and large-scale gas distribution using a comprehensive galaxy-formation model.The model is intended to account for primary processes believed important for galaxy formation and evolution.
  • 6. Scientific Remarks and Cautions: Modeling uncertainties arise from the simulations’ enormous dynamical range and the variety and complexity of their physical phenomena.The authors advise considering identified problems and cautions before advancing scientific conclusions or observational comparisons.
  • 6.1. Caveats with the Illustris Galaxy Formation Model: Feedback and star-formation implementations produce inefficient quenching and related discrepancies in simulated galaxy populations and halo gas content.The authors identify stellar feedback, AGN feedback, and possibly the adopted star-formation recipe as contributors.
  • 6.1. Caveats with the Illustris Galaxy Formation Model: The simulated galaxy color distribution lacks clear red-blue bimodality, and about 10 percent of selected z=0 disk galaxies show strong ring-like features.Ring-like features are associated with a possible spurious stellar-clump population identified by Subfind.
  • 6.1. Caveats with the Illustris Galaxy Formation Model: Late-time gas within R500c is underestimated by a factor of 3-10 in halos of M500c∼10^13−14M⊙, and several X-ray and Sunyaev-Zel’dovich predictions conflict with observational constraints.The gas deficit is attributed to the too-violent operation of radio-mode feedback; the cluster Sunyaev-Zel’dovich predictions are described as unreliable.
  • 6.1. Caveats with the Illustris Galaxy Formation Model: Comparisons with observations require care because observational results may disagree and simulated variables must be mapped carefully to observationally derived quantities.The authors describe some quantitative tensions as necessarily partial or inconclusive where direct constraints are lacking.
  • 6.2. Numerical Choices and Data Interpretation: Metallicities, low-mass Subfind objects near host centers, and low-mass black holes in small satellite subhalos require careful interpretation.The cautions reflect uncertain enrichment yields, possible fragmentation artifacts, and artificial black-hole repositioning or reseeding.

7. Community Considerations

The release asks users to cite the relevant simulation and supplementary-product papers, while encouraging collaboration and community contributions. It also outlines planned catalogs, mock observations, merger histories, halo properties, and future simulation products.

  • 7. Community Considerations: Publications using Illustris data should cite the release paper and original project paper, with additional citations for extensive analyses and supplementary products.The Illustris website maintains the up-to-date citation list.
  • 7. Community Considerations: The full Illustris-1 snapshots may be prohibitive for most users to acquire or store, so projects needing the entire set may benefit from collaboration and local guest compute access.The collaboration welcomes joint projects involving closer interaction with members who hold full data copies.
  • 7. Community Considerations: Community contributions can include analysis code or computed data products that may be run across simulations or snapshots and published through the API.Results may remain private until an initial publication and then be released publicly.
  • 7.3. Additional Data Releases: Planned Rockstar catalogs and Consistent-Trees merger trees will provide alternative group and merger histories for all six Illustris boxes and enable consistency checks.Their identified subhalo populations and tree construction differ from Subfind, LHaloTree, and SubLink.
  • 7.3.2. Additional Supplementary Data Catalogs: Planned supplementary releases include multi-band stellar mocks, maps of mass and kinematics, strong-lensing catalogs, black-hole merger data, and stellar assembly histories.The stellar assembly and merger-history data are intended for all subhalos at all Illustris-123 snapshots.
  • 7.3. Additional Supplementary Data Catalogs: Future products also include dark-matter halo structural catalogs, stellar-halo mocks, lightcone images in HST and JWST filters, and potentially smaller, higher-resolution, or larger-volume simulations.The planned halo catalogs include density-profile fits, concentration estimates, formation times, and shapes.

8. Summary and Conclusions

The Illustris project publicly releases six simulation realizations and a broad set of snapshots, catalogs, merger trees, subboxes, and supplementary galaxy data. These products are available through direct files and a web-based API, with documentation for community use.

  • Six Illustris realizations span three resolution levels with full baryonic physics and dark-matter-only analogs.
  • The released data cover a (106.5 Mpc)^3 volume and include several million bound structures in the high-resolution realizations, with ∼7000 well-resolved z = 0 galaxies above 10^10M⊙.
  • 136 redshift snapshots, friends-of-friends and Subfind catalogs, SubLink and LHaloTree merger trees, four high-time-resolution subboxes, and supplementary galaxy catalogs are released.
  • ∼265 TB of data are accessible through direct raw-file downloads or web-based API queries for common search, extraction, and analysis tasks.

Appendix A: Snapshot Data Details

Appendix A documents the organization and contents of the Illustris snapshot data. It covers file structure, subboxes, headers, and fields for the simulated particle and cell types.

  • Table A.1 details file organization for the six runs, including file counts, average data-type sizes, and approximate total volume per run.
  • Snapshot files are documented through header information and field listings for gas, dark matter, tracer particles, stars, and black holes.
  • Table A.3 records each subbox's environment, matter overdensity, center, dimensions, and volume fraction relative to the full box.
  • Tracer-particle documentation lists thirteen auxiliary values and indicates whether each field resets to zero after a snapshot is written.

Appendix B: Group and Merger Tree Data Details

Appendix B specifies the fields and organization of group catalogs and merger-tree files. It documents FoF halos, Subfind subhalos, shared headers and offsets, and the SubLink and LHaloTree structures.

  • Tables B.1–B.3 describe fields in the FoF halo and Subfind subhalo catalogs, including mass-related conventions for wind-phase cells.
  • Tables B.4 and B.5 document group-catalog headers and offsets, including file-chunk counts and missing-tree indicators represented by −1.
  • Tables B.6 and B.7 list fields for the SubLink and LHaloTree merger trees, which also contain copied subhalo fields ordered according to tree structure.

Appendix C: Supplementary Data Details

Appendix C documents supplementary catalogs for selected Illustris-1 z = 0 galaxy properties. These include photometric morphology, stellar circularity, angular momentum, and axis-ratio measurements.

  • Table C.1 describes photometric non-parametric stellar morphologies across four bands and four camera views.
  • Table C.2 details stellar circularities, angular momenta, and axis ratios measured within 10R_E and for all stars in the subhalo.

Appendix D: API Examples and Reference

The appendix demonstrates the Illustris API through Python examples covering metadata access, subhalo searches, merger-tree traversal, and selective snapshot downloads. It also points readers to endpoint reference tables and multilingual getting-started documentation.

  • Documentation: The API documentation includes a getting-started guide and cookbook of common tasks in Python, IDL, and Matlab.The appendix presents four Python examples drawn from the online documentation.
  • Documentation: A helper function handles HTTP requests, raises errors for unsuccessful responses, parses JSON, and saves binary responses to files.This helper supports both structured API responses and downloadable data products.
  • Subhalo queries: At z = 0, the API returns subhalo fields including total mass and stellar half-mass radius for a specified Subfind ID.The example queries Illustris-1 subhalo 0 and prints mass 22174.8 and stellar half-mass radius 12.395.
  • Subhalo queries: At z = 2, a mass-range search returns 550 subhalos and exposes the Subfind IDs of the first five results.The example searches 10^11.9 M⊙ < M < 10^12.1 M⊙.
  • Merger trees: Following SubLink descendant links maps five specified z = 2 subhalos to their z = 0 descendants.The reported descendant IDs are 30465, 41396, 99148, 51811, and 194303 for the five starting IDs.
  • File downloads: The API can download only selected fields from a full snapshot, reducing the example's storage requirement from ∼1.5 TB to ∼17 GB.The request extracts stellar positions, masses, and metallicities from 512 HDF5 files.
Loading 1504.00362v2…