Source-linked AI summary
Towards Large-Scale Heterogeneous Data Organization for Scientific Foundation Models: A Nuclear Fusion Case Study
Nathaniel Chen, Kouroche Bouchiat, Peter Steiner, Azarakhsh Jalalvand, SangKyeun Kim, Egemen Kolemen
TL;DR
Fusion foundation-model training requires organizing heterogeneous, sparse tokamak data spanning multiple sampling rates, tensor structures, and evolving physics. The paper characterizes 23 diagnostics, analyzes window-size and input-complexity trade-offs, and proposes data-loading considerations. It concludes that these analyses provide a template for organizing scientific data at scale, while identifying further scaling-law, benchmark, and cross-device work.
Problem
Fusion data are heterogeneous, sparse, and shaped by nonstationary physics, creating challenges for organizing scalable datasets for foundation-model training.
Method
The paper categorizes tokamak measurements, analyzes window-size scaling and input complexity, and identifies data-loading and cross-device design considerations.
Results
5 orders of magnitude separate the lowest and highest reported sampling rates, spanning 0.04 Hz to 2000 Hz across fusion diagnostics.
Takeaways & Limitations
The window-size analysis and design recommendations provide a template for organizing heterogeneous scientific data at scale.
Takeaways & Limitations
Future work must develop full scaling laws, benchmark downstream tasks, and evaluate cross-device transfer.
Abstract
from arXiv · showhide
Training effective foundation models requires massive and organized datasets, yet scientific domains such as nuclear fusion present unique challenges due to largely heterogeneous and sparse data. Here we characterize the data used in developing such a model: with over 20 sensor types spanning 5 orders of magnitude in sampling rate, mixed tensor structures (point measurements, spectrograms, images), and nonstationary physics. We analyze our input complexity and discuss trade-offs between temporal context and frequency resolution. Our analysis provides a template for representing multi-modal fluctuation data at scale, with implications for both multi-modal control systems and nuclear fusion.
1 INTRODUCTION
Scientific foundation-model training faces data-organization challenges in fusion because tokamak measurements are heterogeneous, sparse, and shaped by nonstationary physics. The paper characterizes this landscape and identifies design considerations for scalable standardized datasets.
- Motivation: Tokamak experiments combine multivariate text, time series, and images across spatial, temporal, and spectral dimensions, with heterogeneous sensor sampling and sparse observability.Some actuators can disable nearby sensor pickup even when data remains present, while turbulent non-stationary physics complicates standard data loading.
- Motivation: Prior fusion studies address disruption forecasting, profile prediction, and cross-diagnostic inference separately, using task-specific preprocessing that hinders scalable standardized datasets.Recent work also reduces spectral-data latent spaces, but remains distinct from a shared data-loading approach.
- Contributions: The paper characterizes fusion-data heterogeneity and its implications for foundation-model training.
- Contributions: The study categorizes 23 DIII-D measurements by sampling rate, tensor structure, and physics role, then analyzes window-size scaling and data-loading trade-offs.
2 THE FUSION DATA LANDSCAPE
Fusion diagnostics combine diverse sampling rates, tensor structures, modalities, and nonstationary signals. These properties create representation and loading challenges, including severe spectral imbalance and difficulties with normalization.
- Data organization: 23 diagnostics span six physics categories and 12 broader categories when represented using a 100ms window.
- Data heterogeneity: Sampling rates span roughly 5 orders of magnitude, from 0.04 Hz for Thomson scattering to 2000 Hz for high-resolution magnetics.A common resampling rate either discards high-frequency physics or expands low-frequency channel dimensions.
- Data heterogeneity: Fusion inputs include 2D time series (C, T), 3D spectral tensors (C, F, T), and images (H, W, 1), requiring models to handle heterogeneous tensor ranks.
- Data heterogeneity: Spectrograms from magnetic and density fluctuations comprise 87% of total elements while representing only 5 of 23 unique modalities.This imbalance may cause models to overfit spectral features while underweighting scalar quantities important for understanding physics.
- Data heterogeneity: Turbulent transport evolves on ∼10ms timescales while global equilibrium evolves over ∼1s, and full-discharge normalization may fail to capture this multi-scale structure.Thermal noise can also obscure coherent structures.
- Representations: Fusion diagnostics are commonly represented as scalar, spectral, or image-like data, with scalar forms emphasizing amplitude and spectral forms emphasizing frequency.The Short Time Fourier Transform is used for spectral representations because of its versatility.
3 DESIGN CONSIDERATIONS
Design choices for fusion foundation-model pipelines center on window size, preprocessing location, and structured sparsity. These choices balance temporal and spectral information while accommodating heterogeneous and missing measurements.
- 3.1 WINDOW SIZE TRADE-OFFS: Larger windows capture lower frequencies but sacrifice temporal localization, making window choice a frequency-resolution and temporal-context trade-off.
- 3.1 WINDOW SIZE TRADE-OFFS: Alfvén eigenmodes at 100–250 kHz require w ≥1ms for spectral resolution, whereas tearing modes at 1–20 kHz permit w ≥100ms windows for mode-locking dynamics.
- 3.1 WINDOW SIZE TRADE-OFFS: A 100ms baseline halves autoregressive roll-out depth relative to 50ms and yields an input sequence of approximately 398K inputs.The 50ms baseline is recommended for modeling a good amount of the kinetics, while 100ms reduces the number of prediction steps for a 5000ms shot from 100.
- 3.2 PRECOMPUTATION VS. ON-THE-FLY PROCESSING: Precomputation can include spectrogram generation and spatial interpolation, while window extraction, padding, normalization, and augmentation can remain on the fly.Spectrogram preprocessing reduces 2000 Hz waveforms to 500 Hz with 128 frequency bins.
- 3.2 PRECOMPUTATION VS. ON-THE-FLY PROCESSING: Wavelet and multiscale transforms may improve performance but introduce additional overhead for baseline tasks.
- 3.3 HANDLING SPARSITY AND MISSING DATA: Channel masking and cross-device alignment address structured sparsity, intermittent diagnostics, and varying device configurations.Mapping diagnostics to common categories supports transfer across DIII-D, KSTAR, EAST, and future ITER data.
4 CONCLUSION
The paper identifies major challenges in organizing nuclear-fusion data for foundation-model training and proposes a scalable data-organization template. Future work will extend this analysis through scaling laws, downstream benchmarks, and cross-device experiments.
- The study identifies extreme sampling-rate diversity, mixed tensor structures, spectral data dominance, and physics-driven non-stationarity as key training challenges.
- Window-size scaling analysis and design recommendations provide a template for organizing scientific data at scale.
- Future work will develop scaling laws across model size and data volume, benchmark downstream tasks, and test cross-device transfer.
T. C. Luce. Development of Steady-State Advanced Tokamak Research in the DIII-D Tokamak.
The supplied passages contain bibliographic information for a Fusion Science and Technology article and a DIII-D Thomson-scattering diagnostic-upgrade article. They do not provide substantive findings from either work.
- Fusion Science and Technology volume 48, issue 2 includes pages 1212–1225 from October 2005.
- The Fusion Science and Technology record lists ISSN 1536-1055 and 1943-7641.
- Ponce-Marquez and colleagues reported a Thomson scattering diagnostic upgrade on DIII-D in Review of Scientific Instruments, 81(10):10D525.
E. J. Strait. Magnetic diagnostic system of the DIII-D tokamak. Review of Scientific Instruments, 77
The supplied passages list bibliographic records spanning tokamak turbulence, a time-series embedding method, and disruption prediction with artificial intelligence. They do not state the methods or findings of these works beyond their titles and publication details.
- Surko and Slusher authored “Waves and Turbulence in a Tokamak Fusion Plasma,” published in Science in 1983.
- TOTEM is identified as “TOkenized Time Series EMbeddings for General Time Series Analysis,” with an arXiv record dated January 2025.
- Vega and colleagues published work on disruption prediction with artificial intelligence techniques in tokamak plasmas in Nature Physics in 2022.
M A Van Zeeland, G J Kramer, R Nazikian, H L Berk, T N Carlstrom, and W M Solomon.
The supplied passages provide bibliographic information for an Alfvén-eigenmode observation paper and a paper on scaling vision transformers. No substantive results are stated.
- The Alfvén eigenmode observation record concerns DIII-D measurements using two-colour CO2 interferometry.
- The observation paper appeared in Plasma Physics and Controlled Fusion, volume 47, issue 9, pages L31–L40, in September 2005.