PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniSpectra: A Unified Foundation Model for Universal Spectra Representation Learning

Md Khairul Islam, Zeyu Xia, Adam S. Bolton, Judy Fox

arXiv:2601.15351v2astro-ph.IMcs.AI

TL;DR

Pixel-indexed models struggle to align spectra across instruments with different sampling, often requiring lossy resampling. OmniSpectra uses physical log-wavelength coordinates and a resolution-aware Fourier decoder; across the reported tasks, it matches or exceeds specialized models without resampling or per-instrument adaptation.

  • Problem

    Pixel-indexed positional codes assign different representations to the same wavelength across instruments, making cross-instrument alignment difficult and requiring lossy resampling.

    Source: Introduction · p. 1

  • Method

    OmniSpectra replaces integer pixel indexing with continuous physical log-wavelength coordinates and uses a resolution-aware implicit Fourier decoder to reconstruct at native coordinates.

    Source: Introduction · pp. 1–2

  • Results

    OmniSpectra matches or exceeds specialized models on source classification, stellar and galaxy property estimation, and redshift prediction without resampling or per-instrument adaptation.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    A shared model can handle heterogeneous native spectra without resampling or per-instrument adaptation while matching or exceeding specialized models across the reported tasks.

    Source: Introduction · p. 2

    The positional encoding’s period range is tuned to approximately 3,600–18,000 Å, performance at substantially different wavelengths is unevaluated, and current representations are unimodal.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Standard sequence models encode position by token rank rather than physical coordinate, typically forcing practitioners to destructively resample continuous physical signals such as spectra observed across heterogeneous instruments. We identify this coordinate mismatch as a structural limitation in cross-instrument representation learning, and propose OmniSpectra, a wavelength-native foundation model for spectral representation learning: a continuous-coordinate masked autoencoder that discards the integer pixel indexing of sequence modeling. Mapping physical coordinates to a logarithmic basis turns the multiplicative scaling induced by cosmological redshift into a uniform additive shift. In continuous rotary position embeddings, this shared shift cancels from relative token distances rather than producing grid-dependent positional distortion. We further counter the spectral bias inherent to implicit neural decoders with a physics-grounded, resolution-gated Fourier decoder that provides anti-aliasing. Pretrained on eight surveys and benchmarked against seven baselines, our model reaches state-of-the-art performance with only 12.2M parameters while outperforming substantially larger models, indicating that encoding physical structure can substitute for parameter scale.

Introduction

Heterogeneous instruments make shared spectral representation difficult: pixel-indexed models require lossy resampling that can blur narrow features and add interpolation artifacts. OmniSpectra instead encodes spectra in physical log-wavelength coordinates and reconstructs at native coordinates, matching or exceeding specialized models across several tasks without resampling or per-instrument adaptation.

Introduction

  • Instrument heterogeneity creates a choice between per-instrument models and lossy resampling, which blurs narrow spectral features and introduces interpolation artifacts.A shared representation must ingest native spectra across heterogeneous instruments.Source: Introduction · p. 1
  • Pixel-indexed positional encoding assigns different codes to the same physical wavelength across detectors, leaving cross-survey attention geometry misaligned even when tokens carry wavelength.The mismatch can require lossy resampling and impair representation transfer.Source: Introduction · p. 1
  • OmniSpectra replaces integer pixel indexing with physical log-wavelength coordinates, where redshift dilation becomes translation, and decodes spectra at native coordinates.Its resolution-aware implicit Fourier decoder supports a shared set of weights without a fixed grid.Source: Introduction · p. 1, Figure 1 · p. 2
  • Pretrained on 6.4M spectra from eight surveys, OmniSpectra matches or exceeds specialized models across source classification, property estimation, and redshift prediction without resampling or per-instrument adaptation.The reported property-estimation scope includes stellar and galaxy properties.Source: Introduction · p. 2

Related Work

Prior work addresses variable-length inputs, continuous coordinates, physics-informed representations, and spectroscopic modeling, but the approaches differ in how they handle instrument-dependent sampling and wavelength alignment. OmniSpectra’s distinguishing choice is relative logarithmic wavelength coordinates, making redshift a uniform translation in attention.

Continuous-coordinate models

  • Variable-length models and relative-position methods can remain tied to instrument-dependent token indices, while continuous-coordinate models accommodate irregular sampling.Related approaches span neural operators, implicit representations, and set-based models.Source: Related Work · p. 2

Physics-informed representations

  • Fourier-feature networks address standard MLP spectral bias, allowing coordinate-conditioned decoders to recover narrow, high-frequency features.The bias concerns the network’s own Fourier spectrum, not the observed spectrum.Source: Related Work · p. 2

Spectroscopic models

  • Spectroscopic models range from fixed-grid, single-instrument systems to native-grid and instrument-specific tokenizers; OmniSpectra instead uses relative logarithmic coordinates so redshift is a uniform attention translation.Spender uses known redshift to resample onto a common rest-frame grid, whereas AION-1 tokenizes instruments separately and Shen et al. use absolute sinusoidal wavelength embeddings.Source: Related Work · p. 2

Methodology

OmniSpectra represents spectra through instrument-independent content tokens and continuous physical-wavelength coordinates, then reconstructs masked values at native coordinates with a resolution-aware Fourier decoder.

Problem Statement

  • Instrument spectra are discrete samples at instrument-specific coordinates, and sampling density differs by as much as 25×.Source: Problem Statement · p. 2

Attention-Pooled Patch Tokenizer

Log-Wavelength Positional Encoding

Implicit Resolution-Gated Fourier Decoder

Experimental Setup

OmniSpectra is pretrained on native-grid spectra from eight surveys and evaluated against pretrained and supervised baselines. The setup uses frozen-encoder transfer for foundation models and task-specific downstream splits and metrics.

Datasets

  • 6.4M spectra from eight surveys span resolving power R ≈ 1,500–34,000 and 3,865–16,384 native pixels, with no resampling or interpolation.The surveys cover optical and near-infrared data; each spectrum enters on its native grid.Source: Experimental Setup · p. 4

Training and Baselines

  • The 12.2M-parameter model uses a 384-dimensional, six-layer encoder and a 128-dimensional, four-layer decoder, pretrained for 200 epochs with 50% patch masking.Comparisons include four pretrained models, two supervised single-instrument models, and GaSNet-III for redshift only; Shen et al. and Perceiver use the same corpus and matched step budget.Source: Experimental Setup · p. 4
  • Figure 4 contrasts linear and Fourier decoding from shared encoder context, then compares ungated and gated reconstructions.The caption describes the two comparisons but supplies no quantitative outcome.Source: Figure 4 · p. 5

Transfer learning protocol

  • Foundation models use frozen encoders with a lightweight cross-attention pooling head, while supervised baselines train end-to-end per task.Models train for 10 downstream epochs, with selection by validation loss.Source: Experimental Setup · p. 5

Downstream data splits

  • Galaxy-property and redshift tasks use 232.7K DESI EDR–PROVABGS cross-matched samples with an 80:20 split; other tasks use 12,500 samples per instrument, with 20% held out.Results average over five random seeds. Classification reports F1; regression reports R2 and σMAD, with catastrophic-outlier rate added for redshift.Source: Experimental Setup · p. 5

Results and Analysis

OmniSpectra reconstructs survey spectra on their native wavelength grids with shared weights, closely tracking ground truth. It leads on all three stellar parameters for both SDSS and APOGEE and remains strongest as labels are reduced.

Cross-Instrument Reconstruction

Stellar Parameter Estimation

  • OmniSpectra is best on Teff, log g, and [Fe/H] for both SDSS and APOGEE, and remains strongest as labels are reduced.Its gap from the baselines remains large as more labels are added.Source: Stellar Parameter Estimation · p. 5
  • AION-1 is omitted because its public spectral weights do not cover APOGEE and its official SDSS checkpoint trails every other pretrained model by a large margin.Source: Stellar Parameter Estimation · p. 5

Galaxy Property Estimation

On DESI PROVABGS galaxy properties, OmniSpectra outperforms in most cases, substantially on three of four targets. For source classification, it achieves the best F1 for every SDSS class and leads in the low-data regime.

Galaxy Property Estimation

  • OmniSpectra outperforms on most DESI PROVABGS galaxy-property cases, substantially for three of four targets.The targets are sSFR, mass-weighted age, mass-weighted metallicity, and stellar mass.Source: Galaxy Property Estimation · p. 5, Table 3 · p. 6

Source Classification

Redshift Estimation

On DESI PROVABGS redshift estimation, OmniSpectra achieves the best R2-score, ties GaSNet-III on σMAD, and has a nearly fivefold lower outlier rate.

Redshift Estimation

  • OmniSpectra achieves the best R2-score, ties GaSNet-III on σMAD, and has a nearly 5× lower outlier rate.Table 5 and Figure 8 report R2, σMAD, and outlier rate on DESI PROVABGS.Source: Redshift Estimation · p. 6, Table 5 · p. 6

Ablation Study

The ablation finds task-dependent, non-additive component effects, while cross-survey tests show robust transfer for wavelength-encoding models and sharper redshift degradation for index-based Perceiver.

Ablation Study

  • Table 6 swaps tokenizer, positional-code, and reconstruction components for conventional alternatives, comparing Base with all alternatives against Full with all proposed components.The reconstruction alternative removes the Fourier basis and gate; −Gate retains the Fourier decoder but removes LSF-dependent attenuation.Source: Ablation Study · pp. 6–7, Table 6 · p. 7

Cross-Survey Generalization

  • With SDSS-only pretraining, wavelength-encoding models remain robust on DESI and APOGEE, while Perceiver degrades sharply on redshift; OmniSpectra has the higher average.APOGEE lies entirely outside the SDSS pretraining wavelength range.Source: Figure 7 · p. 7, Cross-Survey Generalization · p. 7

Ablation Study

  • Component effects are task-dependent and non-additive: positional code matters most on APOGEE, the LSF gate on source classification, and some partial configurations underperform Base.The passage describes these findings as evidence of component interactions.Source: Ablation Study · p. 7

Inference Efficiency

Native-pixel patching shortens inference sequences and improves throughput, while the wavelength encoding’s tuned period range and unimodal representations bound the reported scope.

Inference Efficiency

  • On identical DESI batches of 64, OmniSpectra and Shen et al. use shorter sequences and achieve the highest throughput; OmniSpectra uses far less memory than per-pixel and fixed-length baselines.Native-pixel patching makes encoder sequence length scale with the observed spectrum rather than a fixed grid.Source: Inference Efficiency · p. 7

Limitations

  • The positional encoding’s period range is tuned to approximately 3,600–18,000 Å, and performance at substantially different wavelengths has not been evaluated.The continuous log-wavelength encoding can be evaluated outside the training instruments’ wavelength union without architectural modification.Source: Limitations · p. 7
  • Learned representations are currently unimodal; extending them to other modalities, such as through CLIP-style alignment, remains future work.Source: Limitations · p. 7

Conclusion

OmniSpectra addresses cross-instrument coordinate mismatch with wavelength-native codes and physics-grounded inductive biases. It reports state-of-the-art performance across several tasks and data regimes with 12.2M parameters.

Conclusion

  • OmniSpectra targets coordinate mismatch across heterogeneous instruments in continuous physical signals.Source: Conclusion · p. 7
  • Log-wavelength coordinates turn redshift into an additive shift that cancels in RoPE’s relative attention, while hardware resolution gating supplies another physics-grounded bias.Source: Conclusion · p. 7
  • With 12.2M parameters, OmniSpectra reaches state-of-the-art results in classification, redshift, and property regression across full- and low-data regimes.The conclusion attributes the gains to encoding structure rather than pretraining breadth.Source: Conclusion · p. 7

Related papers

Loading 2601.15351v2…