Source-linked AI summary
The IllustrisTNG Simulations: Public Data Release
Dylan Nelson, Volker Springel, Annalisa Pillepich, Vicente Rodriguez-Gomez, Paul Torrey, Shy Genel, Mark Vogelsberger, Ruediger Pakmor, Federico Marinacci, Rainer Weinberger, Luke Kelley, Mark Lovell, Benedikt Diemer, Lars Hernquist
TL;DR
The paper addresses how to make the full IllustrisTNG simulation suite broadly usable without requiring researchers to download its large datasets. It releases the simulations and associated catalogs, trees, and snapshots, while extending online API tools for visualization, analysis, and browser-based computation. The result is approximately 1.1 PB of TNG data directly accessible online, alongside explicit cautions about unresolved ISM physics, numerical convergence, and scientific interpretation.
Problem
The paper provides a comprehensive public data release and access framework for the large IllustrisTNG simulation suite.
Method
The release combines complete simulation outputs, catalogs, merger trees, supplementary data, web-based analysis and visualization tools, and a remote JupyterLab interface.
Results
Approximately 1.1 PB of IllustrisTNG data are directly accessible online, including full simulations, snapshots, catalogs, merger trees, subboxes, and supplementary products.
Takeaways & Limitations
Online access and browser-based computation aim to maximize the scientific return from the TNG simulations without extensive local computational resources.
Takeaways & Limitations
Cold neutral and molecular ISM phases, giant molecular clouds, and individual massive-star birth sites are unresolved, so dense-ISM observables such as CO masses require care.
Abstract
from arXiv · showhide
We present the full public release of all data from the TNG50, TNG100 and TNG300 simulations of the IllustrisTNG project. IllustrisTNG is a suite of large volume, cosmological, gravo-magnetohydrodynamical simulations run with the moving-mesh code Arepo. TNG includes a comprehensive model for galaxy formation physics, and each TNG simulation self-consistently solves for the coupled evolution of dark matter, cosmic gas, luminous stars, and supermassive blackholes from early time to the present day, z=0. Each of the flagship runs -- TNG50, TNG100, and TNG300 -- are accompanied by lower-resolution and dark-matter only counterparts, and we discuss scientific and numerical cautions and caveats relevant when using TNG. Full volume snapshots are available at 100 redshifts; halo and subhalo catalogs at each snapshot and merger trees are also released. The data volume now directly accessible online is ~1.1 PB, including 2,000 full volume snapshots and ~110,000 high time-resolution subbox snapshots. Data access and analysis examples are available in IDL, Python, and Matlab. We describe improvements and new functionality in the web-based API, including on-demand visualization and analysis of galaxies and halos, exploratory plotting of scaling relations and other relationships between galactic and halo properties, and a new JupyterLab interface. This provides an online, browser-based, near-native data analysis platform which supports user computation with fully local access to TNG data, alleviating the need to download large simulated datasets.
2 Description of the Simulations
IllustrisTNG comprises three cosmological gravo-magnetohydrodynamical simulation volumes with a shared galaxy-formation model, spanning complementary volume and resolution regimes. This release covers the simulations, their physical data, and evidence of broad scientific applicability.
- Simulation suite: IllustrisTNG evolves dark matter, gas, stars, and supermassive black holes from z = 127 to z = 0 using AREPO.The suite includes a comprehensive galaxy-formation model and newly incorporates cosmic magnetism.
- Simulation suite: TNG50, TNG100, and TNG300 use roughly 50, 100, and 300 Mpc boxes, respectively, with lower-resolution and dark-matter-only counterparts.The volumes trade statistical reach against mass resolution: TNG300 supports rare massive objects, while TNG50 provides more than a hundred times better mass resolution.
- Data scope: The release includes all 100 redshift snapshots, halo and subhalo catalogs, merger trees, multiple resolution levels, and high-time-resolution subboxes.The highest-resolution realizations contain 2×2160^3, 2×1820^3, and 2×2500^3 resolution elements for TNG50-1, TNG100-1, and TNG300-1, respectively.
- Resolution: TNG snapshots resolve adaptive gas-cell sizes, with the highest spatial resolution concentrated in star-forming gas inside galaxies and the largest cells in low-density intergalactic gas.Figure 2 summarizes the distributions and their median values for the three high-resolution simulations at z ∼ 0.
- Validation and applications: TNG has reproduced observationally consistent galaxy, cluster, circumgalactic, and intergalactic properties while supporting novel studies of galaxy formation and evolution.Reported applications include galaxy morphologies, metallicity relations, magnetic fields, jellyfish galaxies, mock 21-cm maps, molecular hydrogen, and machine-learning analyses.
3 Data Products
The data products provide complete simulation outputs together with catalogs and merger trees needed to analyze halos, subhalos, and their evolution.
- Snapshots: The release provides 100 snapshots for every run, spanning dark matter, gas, tracers, stars and winds, and supermassive black holes.Runs are available at high, intermediate, and low resolutions, with consistent fiducial TNG galaxy-formation physics and matched dark-matter-only analogs.
- Catalogs and trees: Every snapshot includes friends-of-friends halo catalogs and Subfind subhalo catalogs, from which released SubLink and LHaloTree merger trees are generated.Supplementary catalogs add further computations and modeling for varied scientific topics.
3.1 Snapshots
TNG snapshots are whole-volume, multi-component datasets sampled at 100 epochs, with distinct full and mini formats and group-based particle organization. Their fields include standard metadata plus shock, elemental-abundance, and magnetic-field information.
- Snapshot formats: Each run has 100 whole-volume snapshots, comprising 20 full snapshots and 80 mini snapshots with a subset of fields.Snapshots are stored in manageable on-disk chunks.
- Snapshot organization: Particles are sorted by FoF group number, Subfind subhalo number, and binding energy rather than spatial position.Inner-fuzz particles follow each group’s subhalos, while outer-fuzz particles are placed at the snapshot end; halos can span file chunks.
- Snapshot contents: HDF5 snapshots contain metadata groups and particle-type groups for gas, dark matter, tracers, stars and winds, and black holes.Dark-matter-only runs contain a single particle-type group.
- Units and metadata: Comoving quantities use documented units and can be converted to physical quantities with the appropriate power of the scale factor.Online field listings provide dimensions, units, and descriptions.
- Additional fields: TNG adds shock-finder outputs, nine tracked elemental abundances, metal-tracking fields, and magnetic-field quantities to the snapshot data.These include EnergyDissipation, Machnumber, GFM Metals, GFM MetalsTagged, MagneticField, and MagneticFieldDivergence.
3.1.3 Tagged Metals
Tagged-metal fields record the metals and iron ejected by different enrichment channels, but they track the last star ejecting material rather than its original production site.
- SNIa, SNII, and AGB fields record metals ejected by Type Ia supernovae, Type II supernovae, and stellar winds, respectively.
- NSNS records stochastic mass ejection from neutron-star mergers, with physical masses requiring multiplication by α/α0.
- FeSNIa and FeSNII record iron ejected by Type Ia and Type II supernovae, while the remaining fields follow related enrichment channels.
- The tags identify the last star from which an element was ejected, not where that element was originally created.For example, AGB winds can re-eject iron previously produced by Type Ia and Type II supernovae.
3.3.1 SubLink
The SubLink and LHaloTree merger trees are stored as large, split HDF5 data structures with different file organizations.
- SubLink is split across sequential HDF5 files named tree extended.[fileNum].hdf5.For TNG100-1, file numbers run from 0 to 19; for TNG300-1, they run from 0 to 125.
- LHaloTree is split across HDF5 files named trees sf1 99.[chunkNum].hdf5, containing disjoint merger trees in TreeX groups.TNG100-1 has 80 chunks and TNG300-1 has 320 chunks in the examples given.
3.3.3 Offsets Files
Offsets files map halos and subhalos to their particle locations in group catalogs and snapshots, supporting efficient local data loading.
- Offset numbers identify where each halo or subhalo begins in group-catalog and snapshot files.
- Local helper scripts require downloading the offset files for the snapshots being analyzed.The web API and its particle cutouts do not require offsets.
- TNG stores offsets separately as one offsets *.hdf5 file per snapshot, unlike Illustris.
3.3.4 The ‘simulation.hdf5’ file
The optional simulation.hdf5 file presents an entire simulation through virtual links to data stored in other files, simplifying access without duplicating the underlying datasets.
- simulation.hdf5 is an optional convenience file that encapsulates the data of an entire simulation.It is not required by the public scripts.
- HDF5 virtual datasets make simulation.hdf5 a collection of symbolic links to subsets of datasets in other HDF5 files.
- Users must still have the corresponding snapshot, group-catalog, or supplementary files that the links reference.
- The virtual-dataset organization makes loading snapshots, catalog fields, and particles for a selected halo or subhalo straightforward.Supplementary catalogs can also be virtually inserted into snapshots or group catalogs.
- The release also includes initial conditions and post-processed supplementary catalogs supporting additional analyses.The initial conditions cover all TNG and original Illustris volumes, while supplementary catalogs are computed from raw outputs.
4 Data Access
TNG offers three complementary access modes, ranging from downloading raw files to browser-based computation directly beside the simulation data. Its enhanced API adds searchable catalogs, data extraction, merger-tree visualization, plotting, and on-demand galaxy and halo visualizations, while warning that outputs require scientific judgment.
- Access modes: TNG supports local downloads, API-based extraction with local analysis, and browser-based remote analysis through JupyterLab.These approaches can also be combined, such as searching a downloaded catalog before extracting selected merger trees or particle data.
- Web-based API: The API searches subhalos, retrieves particle cutouts and merger histories, and downloads targeted snapshot subsets.Requests can be made through a web browser or programmatically in a script.
- Remote analysis: The remote-analysis platform hosts the datasets and analysis environment together, eliminating the need to download data or run calculations locally.The service is intended to facilitate broad, open access to large cosmological simulation datasets.
- Exploration and visualization: The release adds interactive 2D and 3D explorers for examining derived imagery and group-catalog structure across the simulation volume.The 3D Explorer represents halos in the simulation box and supports rotation, zooming, and navigation.
- Exploration and visualization: Merger-tree tools provide interactive and static views of subhalo assembly histories, with node size mapped to halo mass and color to instantaneous sSFR.The static view emphasizes merger events and a subhalo’s path toward quiescence.
- Catalog analysis: The group catalog plotter produces publication-quality relationship plots using density histograms, median relations, percentiles, and optional third-quantity coloring.The third quantity can reveal structure underlying scatter in relations such as stellar mass versus halo mass.
- Exploration and visualization: On-demand visualization renders particle-level projections and can return both publication-quality figures and the underlying data.Available projections include gas density or temperature, stellar line-of-sight velocity, and dark matter velocity dispersion.
- Cautions: Visualization outputs can become nonsensical when requests conflict with TNG’s effective two-phase ISM model, which does not resolve cold, dense ISM phases.Users are advised to understand the scientific limitations of the requested analysis.
5 Scientific Remarks and Cautions
TNG improves agreement with observations relative to the original Illustris simulation, while retaining important observational, numerical, and unresolved-physics caveats. Users should treat comparisons and multi-resolution analyses carefully.
- Observational scope: TNG100-1 and TNG300-1 show no comparably significant observational tensions in initial comparisons, while reproducing diverse galaxy properties and scaling relations across cosmic time.The model remains physically plausible but simplified, and the authors invite further observational scrutiny.
- Observational tensions: Observational comparisons require careful mock measurements because inconsistent observations and imperfect mappings from simulated to observed quantities limit stronger conclusions.The authors explicitly avoid more specific tension quantification because observational results may disagree across properties and cosmic time.
- Observational tensions: Possible tensions include alpha-enhanced Milky Way-like stars, low massive-black-hole accretion rates, excess red disk-like galaxies, and sharper quenched-fraction trends.These issues are reported as possible tensions rather than definitive failures of the model.
- Numerical considerations: Non-cosmological satellite fragments can contaminate galaxy catalogs, especially as low-mass, baryon-dominated objects near host centers at late times.The Subfind algorithm cannot inherently distinguish these baryonic fragments from satellites formed through cosmological structure formation; SubhaloFlag assists identification.
- Numerical considerations: Cold neutral and molecular interstellar-medium phases, giant molecular clouds, and massive-star birth sites are unresolved, requiring care when modeling dense-ISM observables.The paper specifically gives CO masses as an example of an observable affected by this coarse multiphase approximation.
- Numerical considerations: The degree of numerical convergence depends on the specific property, so analyses combining TNG boxes or resolutions require explicit convergence checks.Resolution convergence is described as complex when multiple realizations are analyzed together.
6 Community Considerations
The release is framed as a community resource supported by citation guidance, future catalog and simulation plans, and contributions to its data products and API. Large full snapshots motivate continued investment in remote and on-demand access.
- Community access: Full TNG100-1 and TNG300-1 snapshots can be prohibitive to transfer or store, with one snapshot requiring roughly 48 hours to five days at 10 MB/s.The cited examples are approximately 1.5 TB for TNG100-1 and 4.1 TB for TNG300-1.
- Community contributions: Users are invited to contribute analysis code, derived catalogs, supplementary products, and fast halo or subhalo routines that can be integrated into the API.The API can expose integrated routines as on-demand results for individual objects.
- Future releases: Future releases are planned to include Rockstar- and Velociraptor-based group catalogs, which will identify subhalo populations differently from Subfind, especially during mergers.The planned Consistent Trees assembly histories also differ fundamentally from LHaloTree and SubLink.
- Future releases: The project includes roughly 100 model-variation simulations that change one fiducial-model choice or parameter, enabling robustness studies through matched 25 Mpc/h volumes.Their relatively small volume limits the available statistics.
- Future releases: The API roadmap includes on-demand zoom initial conditions, longer-running radiative-transfer and other mock-observation calculations, and browser-based exploration tools.Examples include SKIRT, spectral HI, X-ray datacubes, and intergalactic quasar-absorption sightlines.
7 Architectural and Design Details
The release builds on the original Illustris platform by combining downloadable data with remote computation through a web-based API and JupyterLab. Prior usage demonstrates substantial community demand for this architecture.
- Usage: The original Illustris release attracted 2,122 registered users, 269 million API requests, and approximately 2.15 PB of data transfer in three and a half years.It also supported millions of particle cutout and merger-tree extraction requests.
- Usage: The original release produced 163 publications, with recent Illustris papers typically written without collaboration-team members, indicating broad external use.Only one of the ten most recent Illustris papers came from the team.
- Remote computation: TNG adds remote data with remote computation through per-user, containerized JupyterLab instances with read-only access to simulation filesystems.This extends the earlier modes of local computation over downloaded data or downloaded API subsets.
- Backend design: The API retains a REST architecture and relational backend while considering direct bitmap-indexed searches over structured HDF5 data as a future alternative.The proposed approach would reserve the database for lightweight metadata and query binary simulation data directly.
- Backend design: GraphQL is identified as a possible future API standard because declarative, typed queries could unify detailed requests to simulation data.The paper presents this as a development target rather than a deployed feature.
8 Summary and Conclusions
The release makes the full IllustrisTNG suite and associated catalogs, trees, subboxes, and supplementary products publicly available. Its online access tools are designed to broaden analysis of representative synthetic galaxy populations without requiring extensive local resources.
- Data release: The release includes TNG50, TNG100, and TNG300, each with lower-resolution realizations and a dark-matter-only analog.It provides full snapshots, halo and subhalo catalogs, merger trees, high-time-resolution subboxes, and supplementary catalogs.
- Data release: TNG100-1 contains approximately 20,000 well-resolved z = 0 galaxies above 10^9 M⊙, while TNG300-1 includes more than ten million gravitationally bound structures.The sampled galaxies span mass, type, environment, and assembly history within ΛCDM.
- Access and analysis: Approximately 1.1 PB of TNG data is directly accessible online through downloads, scripts, documentation, API functions, on-demand visualization and analysis, and remote JupyterLab.The stated goal is to maximize scientific return while reducing dependence on extensive local computational resources.
Appendix A: Simulation Data Details
Table A.1 organizes file-structure and data-volume details for all twenty TNG runs, covering both baryonic and dark-matter-only simulations.
- Table A.1 reports file-chunk counts, average full-snapshot sizes, corresponding group-catalog sizes, and estimated total simulation data volumes.
Appendix B: Web-Based API Examples
Appendix B illustrates web-based API functionality through explicit request URLs and Python examples. The examples cover metadata retrieval, object searches, particle cutouts, merger trees, and selective snapshot downloads.
- The API examples show how request URLs specify simulations, snapshots, halos or subhalos, and requested data products.
- A subhalo query retrieves total mass and stellar half-mass radius for a specified object at z = 0.
- Mass-range filtering at z = 2 returns Subfind IDs for subhalos matching specified total-mass bounds.
- Selective particle-field requests can reduce a TNG300-1 snapshot download from approximately 4.1 TB to approximately 20 GB.The example requests only stellar coordinates, masses, and metallicities across 600 HDF5 files.