PaperlayerView exact revision on arXiv

Source-linked AI summary

Building high accuracy emulators for scientific simulations with deep neural architecture search

M. F. Kasim, D. Watson-Parris, L. Deaconu, S. Oliver, P. Hatfield, D. H. Froula, G. Gregori, M. Jarvis, S. Khatiwala, J. Korenaga, J. Topp-Mugglestone, E. Viezzer, S. M. Vinko

arXiv:2001.08055v2stat.MLcs.LGphysics.ao-phphysics.comp-phphysics.plasm-ph

TL;DR

Accurate emulators for slow scientific simulations are difficult to build because machine-learning methods can require prohibitively expensive training data and suitable architectures are challenging to choose. DENSE jointly searches for an architecture and trains it using a super-architecture. The paper reports accurate emulation across varied simulations, with acceleration up to 2 billion times, while noting input-dimension and high-variability limitations.

  • Problem

    Machine-learning emulators need large training datasets for accuracy, but slow simulations can make those datasets prohibitively expensive, and choosing a suitable architecture is difficult.

    Source: §I · p. 1

  • Method

    DENSE uses neural architecture search to select a case-suited architecture while training it within a super-architecture of probabilistically selected operations.

    Source: §I · pp. 1–2

  • Results

    DENSE achieved the best results among tested non-deep-learning techniques in all cases, with loss up to 14 times lower, and emulation speedups up to 2 billion times.

    Source: A. Emulator results · p. 5, §IV · p. 6

  • Takeaways & Limitations

    The reported acceleration supports using DENSE emulators for expensive simulations, including real-time simulation-based diagnostics, uncertainty quantification, and extensive parameter exploration.

    Source: §IV · p. 6

    DENSE has not been tested with direct one-, two-, or three-dimensional inputs and is limited to simulations with a few scalar inputs.

    Source: A. Limitations · p. 6

Abstract

from arXiv · show

Computer simulations are invaluable tools for scientific discovery. However, accurate simulations are often slow to execute, which limits their applicability to extensive parameter exploration, large-scale data analysis, and uncertainty quantification. A promising route to accelerate simulations by building fast emulators with machine learning requires large training datasets, which can be prohibitively expensive to obtain with slow simulations. Here we present a method based on neural architecture search to build accurate emulators even with a limited number of training data. The method successfully accelerates simulations by up to 2 billion times in 10 scientific cases including astrophysics, climate science, biogeochemistry, high energy density physics, fusion energy, and seismology, using the same super-architecture, algorithm, and hyperparameters. Our approach also inherently provides emulator uncertainty estimation, adding further confidence in their use. We anticipate this work will accelerate research involving expensive simulations, allow more extensive parameters exploration, and enable new, previously unfeasible computational discovery.

I. INTRODUCTION

Slow simulations make the large training datasets needed for accurate machine-learning emulators expensive to generate, while existing models and manual architecture search leave accuracy challenges. DENSE combines architecture search and training within a super-architecture whose operations and connections can adapt to each simulation.

I. INTRODUCTION

  • Machine-learning emulators need large training datasets for accuracy, but generating them with slow simulations can be prohibitively expensive.Accurate simulations are also needed for applications such as rapid screening and online diagnostics.Source: §I · p. 1
  • Prior emulator methods may miss correlations among output points, while manually finding a suitable CNN architecture requires time and domain expertise and can yield suboptimal results.CNNs provide a useful prior for natural signals, but that prior depends on choosing an appropriate architecture.Source: §I · p. 1
  • DENSE uses efficient neural architecture search to find a case-suited network architecture while training it, including when training data are limited.The method is named Deep Emulator Network SEarch (DENSE).Source: §I · p. 1
  • Its super-architecture connects input and output nodes through groups of candidate operations, with one operation selected per group according to its trainable network variable.Candidate operations include convolutions; zero layers allow architecture complexity to vary, and identity layers provide skip connections.Source: §I · pp. 1–2, Figure 1 · p. 2
  • Training updates selected-operation weights against a loss, then ranks sampled architectures on separate validation data to update their selection probabilities.The validation ranking uses Huber loss and zero-mean CMA-ES rewards; the Huber loss is used to reduce outlier influence and increase robustness.Source: §I · p. 2

II. RESULTS

The results section covers ten scientific simulation cases spanning several disciplines and output types. Its figures present emulator comparisons, inverse-problem evaluations, and prediction-uncertainty comparisons, while the supplied passages do not report the plotted numerical outcomes.

II. RESULTS

  • The study evaluates ten cases spanning astrophysics, high-energy-density physics, fusion, seismology, climate science, biogeochemistry, and inertial confinement fusion.Cases include XRTS, OTS, ELMs, XES, Halo, SeisTomo, GCM, MOPS, ICF JAG, and ICF JAG Scalars.Source: §II · p. 2
  • The cases range from 3 to 14 scalar input parameters and produce outputs from scalars to multiple 2D signals.Datasets for simulations taking less than one CPU-hour were generated from 14,000 random-input runs; available resources limited datasets for more expensive simulations.Source: §II · pp. 2, 4
  • Figure 2 compares representative original-simulation outputs with DENSE emulator outputs and shows their relative differences.The caption directs readers to Figure 7 for outputs from the other eight cases.Source: Figure 2 · p. 3
  • Figure 3 presents loss-function ratios against other emulator methods and GPU speedups relative to the original simulations.The caption identifies non-deep-learning methods and a manually designed deep network as the loss comparators.Source: Figure 3 · p. 3
  • Figures 4 and 5 present inverse-problem comparisons, using retrieval-error histograms for XRTS and Halo and posterior-distribution comparisons for XES and XRTS.Figure 4 distinguishes full-simulation and emulator retrievals; Figure 5 includes posterior histograms and parameter scatter plots.Source: Figure 4 · p. 3, Figure 5 · p. 4
  • Figure 6 compares emulator prediction uncertainties with simulation outputs for ICF JAG, Halo, ELMs, and XRTS.The caption identifies orange emulator-output distributions and green simulation outputs for the 1D cases.Source: Figure 6 · p. 4

A. Emulator results

DENSE outputs generally match simulations, including MOPS and GCM with just 410 and 39 data points. It outperforms tested non-deep-learning emulator techniques and can deliver very large speedups.

A. Emulator results

  • Emulator outputs generally match actual simulations, even with only 410 MOPS data points and 39 GCM data points.With limited data, a suitable architecture prior matters because complex models with poor priors can overfit and perform poorly on out-of-sample data.Source: A. Emulator results · p. 4
  • DENSE selects architectures using validation performance, while randomly choosing operations during training regularizes weight updates against overfitting.These mechanisms support emulation when only limited datasets can be generated.Source: A. Emulator results · p. 4
  • For GCM, DENSE is 110 million times faster on a like-for-like CPU basis and over 2 billion times faster with a GPU.The simulation takes about 1,150 CPU-hours; emulator queries take milliseconds to seconds on one CPU core.Source: A. Emulator results · p. 5
  • Up to 14 times lower loss function than the best-performing non-deep-learning model, with DENSE best across all tested cases.The reported advantage was significant in most cases.Source: A. Emulator results · p. 5
  • Against a manually designed deep network, DENSE performs similarly in most cases but improves the loss function considerably for Halo and MOPS.The manually designed model used size-3 convolutional layers and skip connections, with hyperparameters tuned for OTS.Source: A. Emulator results · p. 5

B. Emulators for inverse problems

The authors test whether DENSE emulators can support parameter retrieval and Bayesian posterior sampling in place of costly simulations. Retrieval errors and posterior distributions are similar to simulation-based results, while the emulator workflows take far less time.

B. Emulators for inverse problems

  • Relative retrieval errors from emulators are very similar to those from actual simulations.The comparison uses relative retrieval error histograms for two cases.Source: B. Emulators for inverse problems · p. 5
  • About 800 ms on a single GPU for emulator-based parameter retrieval, versus up to 2 days for XES simulations even on 32 CPU cores.The retrieval uses a simulated test signal with about 1% added noise; the retrieved parameters are compared against known values.Source: B. Emulators for inverse problems · p. 5
  • Emulator-based retrieval also makes parameter retrieval possible for resource-intensive MOPS and GCM simulations that were previously too expensive.The paper frames this as a consequence of completing retrieval in less than a second rather than hours or days.Source: B. Emulators for inverse problems · p. 5
  • Emulator-based posterior distributions are very similar to simulation-based distributions and capture correlations between parameters.Sampling 200,000 XES samples takes over 22 days with simulations but only seconds with emulators; building the XES emulator takes about 14,000 samples and 8 hours of training.Source: B. Emulators for inverse problems · p. 5

C. Prediction uncertainty

DENSE estimates prediction uncertainty from randomized architectures, producing regions of confidence and uncertainty. The estimate can be overconfident, and the method is limited for multidimensional inputs and high-variability regions.

C. Prediction uncertainty

  • DENSE estimates prediction uncertainty by treating randomized architectures as a special case of Monte Carlo dropout.The expected prediction and variance are computed across architectures sampled from the super-architecture using the final network parameters.Source: C. Prediction uncertainty · pp. 5–6
  • Regions of emulator confidence and uncertainty are illustrated in Figure 6.The prediction mean and variance are obtained from architectures sampled from the super-architecture using final network parameters.Source: C. Prediction uncertainty · p. 6
  • Prediction uncertainty can be smaller than the prediction error, indicating overconfidence; early stopping can resolve this, while tuning remains future work.Source: C. Prediction uncertainty · p. 6

A. Limitations

  • DENSE is limited to a few scalar inputs; multidimensional inputs can be parameterized as scalars or handled with dimensionality reduction.Source: A. Limitations · p. 6
  • DENSE learns poorly in high-output-variability regions, which tend to require more samples; intelligent parameter-space sampling is a proposed remedy.Source: A. Limitations · p. 6

B. Applications

DENSE emulators are presented as a route to faster diagnostics and optimization when simulations are expensive. Across diverse scientific cases, the paper reports up to 2 billion-fold acceleration and applications including inverse problems, uncertainty quantification, and parameter exploration.

B. Applications

  • Emulators can accelerate inverse-problem diagnostics without sacrificing solution quality, supporting real-time interpretation in complex systems such as tokamaks and particle accelerators.Source: B. Applications · p. 6
  • For expensive simulations run only a few times, emulators can propose parameter guesses for testing; a more accurate cheap model may reduce required simulation runs.Source: B. Applications · p. 6

IV. CONCLUSIONS

  • Up to 2 billion-fold acceleration enables tasks including real-time simulation-based diagnostics, uncertainty quantification, and extensive parameter exploration.Source: §IV · p. 6

A. Test cases

  • In the Shatsky Rise tomography case, uncertainty quantification would otherwise require hundreds of thousands of software runs.Source: A. Test cases · p. 8
  • An accurate emulator was built for a three-parameter climate model from as few as 39 data points, despite an annual run costing about 1150 CPU-hours.Source: A. Test cases · p. 9

B. Super-architecture

The DENSE super-architecture combines fully connected and convolutional layers, with selectable layer operations and always-present identity skip connections.

B. Super-architecture

  • The super-architecture begins with two fully connected layers, followed by selectable convolutional-layer combinations with ReLU nonlinearities.Most nodes have 64 channels, and signal size grows from 4 to 250; ICF JAG uses output size 64 × 64.Source: B. Super-architecture · p. 9
  • Every layer group includes an identity skip connection, while a zero operation lets DENSE remove a layer and make the network shallower.Source: B. Super-architecture · p. 9
  • For differing node sizes, modified transposed convolutions expand signals with trainable constants filling the inserted positions; other layers use nearest-neighbor upsampling.Source: B. Super-architecture · p. 9

C. Hyperparameters

The same hyperparameters were used across cases, after selection by validation-loss minimization for OTS with CMA-ES.

C. Hyperparameters

  • CMA-ES selected hyperparameters to minimize OTS validation loss, and the resulting settings were used for every case.Source: C. Hyperparameters · p. 9

D. Other emulator builder methods

The comparison emulators use scikit-learn defaults, with ensembles for methods that predict only one output and a specified CycleGAN setup for two cases.

D. Other emulator builder methods

E. Solving inverse problems with emulators

The inverse-problem procedures use CMA-ES for parameter retrieval and affine-invariant ensemble MCMC for Bayesian posterior sampling, with the same retrieval conditions for simulations and emulators.

E. Solving inverse problems with emulators

  • CMA-ES parameter retrieval for XRTS and Halo used population size 32 and a maximum of 1200 function evaluations.The same algorithm parameters and conditions were used for retrievals via simulations and emulators.Source: E. Solving inverse problems with… · p. 9
  • Bayesian posterior sampling for XRTS and XES used affine-invariant ensemble MCMC with 256 walkers and 200,000 samples.Source: E. Solving inverse problems with… · p. 9
  • The likelihood is uniform inside the specified band and zero outside it, with bands of 0.035 J/keV/ster for XES and 3.5% for XRTS.The band settings follow ref. 27.Source: E. Solving inverse problems with… · p. 9

F. Derivation of prediction uncertainty

The uncertainty derivation treats architecture randomization as a special case of dropout and approximates the intractable posterior over network weights with a variational distribution.

F. Derivation of prediction uncertainty

Related papers

Loading 2001.08055v2…