Source-linked AI summary
AION-1: Omnimodal Foundation Model for Astronomical Sciences
Liam Parker, Francois Lanusse, Jeff Shen, Ollie Liu, Tom Hehir, Leopoldo Sarra, Lucas Meyer, Micah Bowles, Sebastian Wagner-Carena, Helen Qu, Siavash Golkar, Alberto Bietti, Hatim Bourfoune, Nathan Casserau, Pierre Cornette, Keiya Hirashima, Geraud Krawezik, Ruben Ohana, Nicholas Lourie, Michael McCabe, Rudy Morel, Payel Mukhopadhyay, Mariel Pettee, Bruno Regaldo-Saint Blancard, Kyunghyun Cho, Miles Cranmer, Shirley Ho
TL;DR
Astronomy lacks a unified framework for jointly modeling heterogeneous observations from different instruments and modalities. AION-1 addresses this gap with modality-specific tokenization and multimodal masked modeling, achieving competitive or state-of-the-art results across diverse tasks while remaining effective with far fewer labels.
Problem
Astronomy lacks a unified framework for modeling heterogeneous, noisy data from different instruments, while prior approaches often isolate modalities or support limited modality combinations.
Method
AION-1 uses tokenizers to homogenize 39 astronomical modalities and a multimodal masked-modeling objective in a single encoder-decoder architecture.
Results
AION-1 achieves or surpasses state-of-the-art performance across diverse downstream tasks and maintains high R2 and classification accuracy with two to three orders of magnitude fewer labels than supervised baselines.
Takeaways & Limitations
AION-1 provides a frozen, multimodal foundation for downstream astronomical workflows using lightweight probes or small task-specific heads.
Takeaways & Limitations
AION-1 inherits the selection functions of its pretraining data, which may influence downstream predictions without recalibration.
Abstract
from arXiv · showhide
While foundation models have shown promise across a variety of fields, astronomy still lacks a unified framework for joint modeling across its highly diverse data modalities. In this paper, we present AION-1, a family of large-scale multimodal foundation models for astronomy. AION-1 integrates heterogeneous imaging, spectroscopic, and scalar data using a two-stage architecture: modality-specific tokenization followed by transformer-based masked modeling of cross-modal token sequences. The model is pretrained on five large-scale surveys: Legacy Survey, Hyper Suprime-Cam (HSC), Sloan Digital Sky Survey (SDSS), Dark Energy Spectroscopic Instrument (DESI), and Gaia. These span more than 200 million observations of stars, galaxies, and quasars. With a single frozen encoder, AION-1 achieves strong results on a broad suite of downstream tasks, including galaxy and stellar property estimation, galaxy morphology classification, similarity-based retrieval, galaxy image segmentation, and spectral super-resolution. We release AION-1 model variants ranging from 300 M to 3.1 B parameters. Beyond astronomy, AION-1 provides a scalable blueprint for multimodal scientific foundation models that can seamlessly integrate noisy, instrument-specific observations. All code, tokenizers, pretrained weights, and a lightweight evaluation suite are released under an open-source license.
1 Introduction
Astronomy lacks a unified foundation-model framework for heterogeneous, noisy observations spanning instruments and modalities. AION-1 addresses this gap with tokenization and multimodal masked modeling, achieving broad downstream performance and low-data effectiveness.
- Motivation and contribution: AION-1 unifies 39 heterogeneous astronomical modalities across imaging, spectroscopy, and scalar measurements for galaxies, stars, and quasars.The model is designed for arbitrary combinations of observations and multiple physical phenomena.
- Method: AION-1 first homogenizes diverse observations through bespoke tokenization, then trains a transformer encoder-decoder with multimodal masked-token prediction.The objective learns relationships across modalities simultaneously rather than treating each modality in isolation.
- Empirical scope: AION-1 maintains high R2 and classification accuracy with two to three orders of magnitude fewer labels than fully supervised baselines.This low-data capability matters when high-quality annotations are scarce or expensive.
- Emergent capabilities: AION-1’s embeddings organize objects along physically meaningful directions and support rare-object retrieval that surpasses current state-of-the-art astronomy retrieval methods.The model also supports arbitrary observation combinations for downstream fusion and cross-modal generation.
- Broader significance: The tokenization, masked-modeling, and cross-modal-generation strategies target heterogeneity, noise, and instrument-specific idiosyncrasies in scientific data.The authors present this as a blueprint applicable beyond astronomy, including settings without strong physical models.
- Empirical scope: AION-1 achieves competitive or state-of-the-art performance across property estimation, morphology classification, image segmentation, and spectral super-resolution using lightweight probes or task-specific heads.The encoder can support downstream tasks without bespoke per-task architecture changes.
2 Related Work
Existing foundation models often use language to bridge modalities, while astronomy-specific approaches have focused on narrower modality pairs or phenomena. AION-1 extends this work toward billion-parameter, arbitrary-modality modeling across diverse astronomical objects.
- General multimodal models: General-purpose multimodal models have achieved strong visual reasoning and generation, but primarily rely on language to connect modalities.Language may be unavailable for scientific observations, motivating alternative fusion strategies.
- Alternative fusion: Early-fusion models learn mappings between modalities without requiring language, offering an alternative for scientific data integration.Examples include Chameleon, 4M, and PerceiverIO.
- Astronomy-specific approaches: Astronomy-specific efforts include supervised morphology models and self-supervised multimodal approaches involving image-spectra, image-text, or time-series-photometry pairs.These approaches address selected astronomical tasks or modality combinations.
- AION-1’s position: AION-1 expands scale and scope by targeting billion-parameter models and unifying 39 modalities across galaxies, stars, and quasars.The model spans roughly 200 million unique measurements.
3 Data
AION-1 is pretrained on five public astronomical surveys containing complementary imaging, spectroscopic, photometric, astrometric, and object-level measurements. Pairwise cross-matches and individual survey datasets provide fused views of astronomical objects for multimodal pretraining.
- Survey corpus: The pretraining corpus combines Legacy Survey and HSC galaxy imaging, SDSS and DESI spectra, and Gaia stellar spectra, photometry, and astrometry.The surveys cover galaxies, stars, and quasars with complementary measurement types.
- Legacy Survey: The Legacy Survey sample contains roughly 122 million galaxies after Southern Galactic Cap, magnitude, and quality selections.The data use cropped 96×96 images and associated flux, infrared, reddening, and shape descriptors.
- Modalities: Each survey contributes distinct modalities, including multiband images, optical spectra with inverse-variance and wavelength data, redshifts, Gaia spectral coefficients, astrometry, and broad-band fluxes.These inputs preserve measurement uncertainty and survey-specific information where available.
- HSC: HSC contributes roughly 2.5 million galaxies from deep, high-resolution imaging selected using magnitude, visit, and quality criteria.The sample uses cropped 96×96 images and photometric and shape-tensor modalities.
- Spectroscopic surveys: SDSS provides roughly 4 million galaxies and stars with medium-resolution optical spectra, while DESI contributes roughly 1 million selected galaxies, stars, and quasars from its early data.Both spectroscopic datasets include flux, wavelength, inverse-variance, and pipeline redshift information.
- Gaia: Gaia supplies BP/RP spectral coefficients, four-parameter astrometry, and mean G, BP, and RP fluxes for Milky Way sources with available mean spectra.The spectra are represented by 110 Gauss–Hermite coefficients.
- Cross-survey construction: Pairwise reciprocal sky matches within 1 arcsec create datasets whose objects aggregate modalities from two parent surveys.Pretraining samples both individual-survey datasets and these cross-matched fused views.
4 Tokenization of Astronomical Data Modalities
AION-1 uses modality-specific tokenizers to convert heterogeneous astronomical images, spectra, and scalars into compact, standardized representations. The image, spectrum, and scalar tokenizers accommodate survey differences while preserving measurement information for multimodal modeling.
- Overview: Modality-specific tokenizers standardize images, spectra, and scalar measurements across instruments and surveys before transformer processing.They provide aligned representations within each modality while supporting multiple instruments and survey inputs.
- Image tokenizer: Images from HSC and Legacy Survey are mapped into nine survey-specific channels, with masks tracking unavailable bands.Distinct bands remain separate because the surveys differ in wavelength coverage, pixel scale, zero-point, and noise.
- Image tokenizer: FSQ quantization with roughly 2^12 image codes balances reconstruction loss and codebook utilization.The selected configuration uses levels {8, 5, 5, 5}, while reconstruction loss appears to plateau near this size.
- Image tokenizer: Image-tokenizer training achieves a final reconstruction score of L_NLL = 0.00775.Training uses inverse-variance-weighted Gaussian NLL and incorporates survey-provided noise and pixel masks.
- Spectrum tokenizer: Spectra are normalized and interpolated onto a shared 8,704-point wavelength grid before ConvNeXt encoding and LFQ quantization.The common grid spans 3500–10462.4 Å at 0.8 Å spacing and removes survey-specific wavelength and dispersion differences.
- Spectrum tokenizer: R^2 = 0.994 and mean mask AUC = 0.92 are obtained for spectrum-token reconstruction and reliability prediction.The tokenizer is trained with flux-likelihood, mask-accuracy, and quantizer-commitment losses.
- Scalar tokenizer: Scalar tokenization maps empirical cumulative probabilities into standard-normal space before parameter-free equal-width binning.With K = 1024 bins, median absolute reconstruction error is below typical measurement uncertainties.
5 Multimodal Masked Modeling
AION-1 trains a transformer encoder-decoder by masking tokenized observations across modalities and reconstructing the missing content. Sampling observed and target tokens across the full token pool encourages both intra-modal and cross-modal relationships while modality and provenance embeddings identify each input’s source.
- Model objective: AION-1 randomly masks tokenized inputs across available modalities and reconstructs the masked content with a transformer encoder-decoder.The objective follows a multimodal masked-modeling scheme adapted for heterogeneous astronomical data.
- Model objective: Observed and target tokens are sampled across the entire token pool, enabling learning of both intra-modal and cross-modal relationships.The target subset is predicted from the observed subset rather than requiring all modalities to be present simultaneously.
- Embeddings: Each modality and source receives dedicated token, modality, and positional embeddings, preserving instrument provenance in the input representation.Different instruments therefore receive different modality embeddings even when they produce the same data type.
- Token sampling: A global input budget is filled by sampling one modality first and then drawing remaining tokens from other modalities.This replaces inefficient sampling that frequently produced mostly empty batches for modalities with widely different sequence lengths.
- Token sampling: Output-token counts are sampled from a Beta distribution skewed toward zero to match iterative inference patterns.The procedure is intended to cover output distributions expected under cosine-schedule, ROAR-style sampling.
6 AION-1 Family of Models
The AION-1 family comprises three model sizes trained with the same multimodal masked-modeling setup. Scaling is evaluated through Legacy Survey test losses with and without Gaia stellar data, while the model variants follow a T5-like size convention.
- Scaling: Legacy Survey test losses are compared across three model sizes with and without the Gaia stellar set.The figure reports increased loss when Gaia data are included and provides the model-variant size breakdown.
- Model variants: AION-1 is released in Base, Large, and XLarge variants with 300M, 800M, and approximately 3B parameters.All versions use AdamW, a 256-token input budget, and a 128-token output budget during pretraining.
7 Evaluation on Downstream Tasks
AION-1 supports posterior sampling and cross-survey conditional generation from heterogeneous astronomical inputs. These capabilities produce increasingly constrained redshift estimates and recover high-resolution spectral structure, while the sampler’s outputs are not guaranteed to be calibrated joint posteriors.
- 7.1 Out-of-the-Box Capabilities: AION-1 generates posterior samples for any trained modality by iteratively resampling query tokens conditioned on visible context.The frozen decoder produces categorical distributions, and ROAR progressively promotes sampled tokens to inputs until all query positions are resolved.
- 7.1.1 Redshift Posterior Estimation: Redshift posteriors contract from broadband photometry to photometry plus imaging and then to a high-resolution DESI spectrum.The three contexts are evaluated for a representative Legacy Survey galaxy, with richer observations yielding more informative estimates.
- 7.1.2 Spectral Super-Resolution: AION-1 recovers DESI spectral line centers, widths, and amplitudes from low-resolution Gaia BP/RP coefficients within narrow posterior uncertainty bands.The generated spectra closely track the true DESI measurements despite the coarse conditioning input.
- 7.1.2 Spectral Super-Resolution: The generated spectral samples should not be interpreted as calibrated joint posteriors because the iterative sampler may underrepresent multi-token dependencies.This limitation applies even when samples visually and quantitatively track key spectral features.
7.2 AION-1 Embeddings
AION-1’s frozen encoder produces modality-agnostic embeddings that support multimodal downstream prediction with lightweight adaptation. Across property estimation, morphology, segmentation, and stellar tasks, the model remains competitive or superior, especially when labels are scarce.
- 7.2 AION-1 Embeddings: AION-1 embeddings support images, spectra, scalars, and arbitrary modality combinations without an additional fusion module.Inputs are tokenized, concatenated when multimodal, and passed through the frozen encoder before task-specific calibration.
- 7.2.2 Galaxy Morphology Classification: AION-1-L tops all baselines except ZooBot in galaxy morphology classification, exceeding EfficientNet by +7.2 pp and DINOv2 by +15.8 pp.It trails ZooBot by only −2.4 pp while using two orders of magnitude fewer labeled images during inference.
- 7.2.3 Galaxy Image Segmentation: AION-1’s frozen encoder plus a small decoder outperforms the supervised U-Net for galaxy image segmentation.The evaluation uses intersection-over-union on held-out images.
- 7.2.4 Stellar Parameter Estimation: Adding Gaia geometric information improves metallicity prediction, while XP spectra provide a further ∼15–20% absolute gain in [Fe/H] R2.With high-resolution DESI spectra, AION-1 matches the supervised ConvNeXt baseline.
- 7.2.5 Low-Data Performance: AION-1 reaches R2 ∼0.5 for physical properties and ≥80% for galaxy morphology with only 10^2−10^3 labels, while supervised models remain near-zero R2 or below 70% accuracy.The reported low-data comparison spans property estimation and morphology classification.
- 7.2.5 Low-Data Performance: AION-1 plateaus by ∼10^3−10^4 training examples, whereas baselines require an order of magnitude more data to approach similar performance.The results indicate faster data saturation for the frozen foundation-model representation.
7.3 Rare Object Detection
AION-1 embeddings enable zero-shot retrieval of rare astronomical objects using cosine similarity. The benchmark covers spirals, mergers, and strong lenses, including highly uncommon lensing examples, and compares AION-1 with several foundation and vision baselines.
- Set-Up: The embedding-based setup returns the top candidates by sorting all corpus objects according to cosine similarity with the query embedding.Mean-pooled encoder outputs are used for query and candidate galaxies.
- Galaxy-Zoo Retrieval Experiments: The retrieval benchmark uses cosine-similarity rankings and nDCG@10, with soft volunteer vote fractions defining relevance for Galaxy Zoo morphologies.Queries are high-confidence exemplars drawn from a roughly 171,000-galaxy sample.
- Results: AION-1 outperforms every baseline on retrieval across spirals, mergers, and strong lenses, including lenses whose true abundance is below 0.1%.AION-1-XL nearest neighbours are described as visually convincing across all three classes.
7.4 Emergent Transfer Properties
AION-1 exhibits transfer across unseen survey pairings and imaging domains. It generates DESI spectra from HSC images despite never training on that pair, and a classifier trained on Legacy Survey embeddings transfers to HSC with near-native accuracy.
- 7.4.1 Out-of-Distribution Conditional Generation: AION-1 generates DESI spectra from HSC images even though HSC–DESI pairs were absent during pretraining.The generated spectra closely track ground-truth DESI spectra for both quiescent and star-forming galaxies.
- 7.4.2 Survey-Invariant Representations: A classifier trained on Legacy Survey embeddings transfers directly to HSC images and reaches 84–86% accuracy across AION-1 scales.The zero-shot result essentially matches performance on the native Legacy Survey domain.
- 7.4.2 Survey-Invariant Representations: The transferred morphology performance persists despite approximately twofold depth differences, distinct filter sets, and different pixel scales.The result supports survey-agnostic morphology representations within the evaluated domains.
8 Conclusion
AION-1 unifies 39 heterogeneous astronomical modalities in one masked-modeling foundation architecture and performs diverse downstream tasks with lightweight probes. Its main strengths are broad multimodal transfer and strong low-data performance, while quantization, mean pooling, selection effects, and generative calibration remain important caveats.
- Core contribution: AION-1 integrates imaging, spectroscopy, and scalar metadata across instruments without bespoke per-task architecture changes, using lightweight probes or task-specific heads.The model homogenizes 39 modalities from five public surveys with tokenizers and multimodal masked modeling.
- Downstream performance: AION-1 achieves or surpasses state-of-the-art results across property estimation, morphology classification, image segmentation, and spectral super-resolution with lightweight adaptation.The reported task suite spans both imaging and spectral workflows.
- Downstream performance: Two to three orders of magnitude fewer labels still yield high R2 and classification accuracy than fully supervised baselines when annotations are scarce or expensive.The conclusion identifies low-data performance as a critical capability for astronomical applications.
- Emergent capabilities: Zero-shot semantic retrieval discovers rare objects such as strong gravitational lenses with better scores than previous state-of-the-art astronomical foundation models.The latent space is described as organizing objects along physically meaningful directions, supporting rare-observation retrieval.
- Transfer and accessibility: Survey-invariant representations enable transfer of downstream tasks between telescopes, while the released code, tokenizers, weights, and evaluation suite support adoption.The authors position the model as a shared backbone for fusing observations, prototyping analyses, and mining outliers.
- Limitations and future work: Quantization limits information content, mean-pool retrieval embeddings may be suboptimal, pretraining selection functions can affect predictions, and masked decoding may produce mis-calibrated or under-correlated long-token samples.The paper suggests richer aggregation, recalibration for shifted selections, and autoregressive or diffusion alternatives for coherent joint sampling.
9 Contributions
The paper assigns contributions across project leadership, data, pretraining, downstream evaluation, computing, writing, and advisory roles.
- Project leads are Francois Lanusse and Liam Parker.
- The data team covers image, spectrum, scalar, and segmentation-map data contributions.
- The pretraining and downstream-evaluation teams are separately identified.
- Additional roles include computing and optimization, manuscript writing, and advisory support.
10.1 Data
The paper draws on data and software associated with multiple astronomical surveys and observatories, with extensive institutional and funding acknowledgements.
- The Legacy Surveys combine DECaLS, BASS, and MzLS observations from several telescopes and observatories.
- The project acknowledges supporting software, including LSST software and Legacy Survey use of NEOWISE data.
- The surveys and missions are supported by extensive institutional, governmental, and foundation funding arrangements.
- The data sources include HSC, DESI, SDSS, and Gaia collaborations or missions.