PaperlayerView exact revision on arXiv

Source-linked AI summary

Resampling with neural networks for stochastic parameterization in multiscale systems

Daan Crommelin, Wouter Edeling

arXiv:2004.01457v1math.NAcs.LGcs.NAphysics.ao-phphysics.comp-ph

TL;DR

Multiscale simulations can be too expensive when all scales must be resolved, while parameterizations of unresolved processes need to account for uncertainty. The paper uses neural-network classification to guide stochastic resampling and validates the method on Lorenz 96, finding that sufficient lagged input and stochasticity support its performance.

  • Problem

    Explicitly resolving all scales can be computationally too expensive, and many machine-learning parameterizations are deterministic despite uncertainty in subgrid-scale responses.

    Source: §1 · pp. 1–2

  • Method

    The method classifies reference observations into output bins with a neural network, then conditionally resamples unresolved data using the predicted bin probabilities.

    Source: §4 · pp. 6–7, §7 · p. 23

  • Results

    On Lorenz 96, the method was validated at two parameter settings when sufficient time-lagged variables were included, and its stochastic surrogate had more robust overall performance.

    Source: §7 · p. 23

  • Takeaways & Limitations

    The Lorenz 96 tests indicate that the feature vector’s lagged-variable content matters, especially for the more challenging bimodal setting.

    Source: §6 · p. 22

    For vector-valued unresolved terms, independently generated elements may form a vector inconsistent with the full model if dependencies are poorly learned.

    Source: §4.1 · p. 9

Abstract

from arXiv · show

In simulations of multiscale dynamical systems, not all relevant processes can be resolved explicitly. Taking the effect of the unresolved processes into account is important, which introduces the need for paramerizations. We present a machine-learning method, used for the conditional resampling of observations or reference data from a fully resolved simulation. It is based on the probabilistic classiffcation of subsets of reference data, conditioned on macroscopic variables. This method is used to formulate a parameterization that is stochastic, taking the uncertainty of the unresolved scales into account. We validate our approach on the Lorenz 96 system, using two different parameter settings which are challenging for parameterization methods.

1. Introduction

Explicitly resolving all scales can make multiscale simulations too expensive, so reduced models need parameterizations for unresolved processes. The paper proposes combining neural-network probabilistic classification with resampling to make data-driven parameterization stochastic.

1. Introduction

  • Explicitly resolving relevant scales can be computationally too expensive, motivating reduced models that parameterize unresolved processes.The paper focuses on data-driven parameterization using observations or simulation data.Source: §1 · pp. 1–2
  • Most proposed machine-learning parameterizations are deterministic, while stochastic formulations can account for uncertainty in subgrid-scale responses.The relevance of stochastic formulations is also connected to the Mori-Zwanzig formalism.Source: §1 · p. 2
  • The paper combines neural-network probabilistic classification with resampling or bootstrapping for machine-learning-based stochastic parameterization.The approach builds on earlier work on parameterization with resampling.Source: §1 · p. 2

2. Preliminaries

The preliminaries describe a coupled macro–micro system in which unresolved microscopic processes affect the macroscopic variables. The paper seeks a cheaper reduced model by replacing that effect with a stochastic, memory-dependent surrogate.

2. Preliminaries

  • The system couples macroscopic variables x with microscopic variables y, and the unresolved contribution σ(y) must be represented to obtain a reduced model involving only x.The dimensions may differ substantially, and the paper does not assume strong scale separation.Source: §2 · p. 3
  • The surrogate replaces σ with a data-inferred approximation that is stochastic and depends on past states.The paper relates stochasticity and memory dependence to Mori-Zwanzig theory.Source: §2 · p. 3
  • The study focuses on additive coupling, where σ enters the x equation as an added term; multiplicative coupling is not tested.The discrete-time system is obtained using a numerical integration scheme with time step Δt.Source: §2 · p. 3
  • The model structure accommodates modular setups in which macromodel steps call micromodels to advance microscopic states and update the coupling term.A micromodel may use multiple smaller time steps within one macromodel step.Source: §2 · p. 4

3. A stochastic surrogate model from data

The stochastic surrogate learns to resample unresolved contributions from reference observations conditioned on finite histories of macro- and subgrid-scale variables. The approach retains stochasticity and addresses the high-dimensional input challenge of feature-space binning by learning a mapping to output bins.

3. A stochastic surrogate model from data

  • Reference observations are time series of macroscopic states x and unresolved contributions r from the full model, assumed to have no significant observational error.The data may come from full-model simulations or physical measurements.Source: §3 · p. 4
  • The surrogate samples r_(j+1) conditionally on past x and r, represented by a finite-memory feature vector.The feature vector may include lagged x and r values, with memory depth J; its definition can be modified.Source: §3 · pp. 4–5
  • Unlike local-bootstrap and nearest-neighbor schemes, this method resamples r while advancing the coupled, non-resampled x with the reduced model.The earlier schemes do not involve this coupling to another model.Source: §3 · p. 5
  • Resampling selects an observed unresolved contribution rather than its conditional expectation, retaining uncertainty and avoiding averages that may be inconsistent with the full model.The sampled observations originate from the full model, although their match to the current feature vector depends on the resampling procedure.Source: §3 · pp. 5–6
  • Binning input features suffers from exponential growth in the number of bins as input dimension and memory depth increase.The earlier feature-binning implementation used observations in the same bin to select the associated next unresolved contribution.Source: §3 · p. 6
  • Binning output observations avoids that feature-space curse, and a neural network learns the mapping from features to output bins for resampling.The output-bin approach requires learning a mapping because there is no simple direct map from the feature vector to an output bin.Source: §3 · p. 6

4. Resampling by neural network classification

The method combines output binning with neural-network probabilistic classification to resample unresolved-process data conditioned on macroscopic features. For vector-valued outputs, separate softmax layers scale the binning linearly with dimension, but dependencies among output elements may be misrepresented if the network is poorly trained.

4. Resampling by neural network classification

  • The method bins observed outputs and trains a neural network to predict the probability of each output bin from the feature vector, then resamples an observation from the selected bin.This avoids binning the potentially high-dimensional input feature space.Source: §4 · pp. 6–7
  • The quantized softmax network (QSN) treats output bins as classes and uses a softmax output to represent their conditional probabilities.The described scalar-output version has 2J + 2 inputs and M output classes.Source: §4 · p. 7
  • Unlike a deterministic network that predicts the unresolved output directly, this approach resamples outputs from the full-model observations using a stochastic mapping.The resampled outputs remain consistent with the full model, though they may be poorly matched to the feature vectors.Source: §4 · p. 8

4.1. Vector-valued r and x

  • For vector-valued outputs, separate softmax layers bin each element, yielding N × Nbins bins overall and linear scaling with output dimension.The layers share feature inputs and hidden layers.Source: §4.1 · pp. 8–9
  • Separate elementwise resampling can misrepresent dependencies, so the resulting output vector may be inconsistent with the full model when the network is poorly trained.Alternatively, clustering can resample an entire observed vector at once, preserving its full-model consistency, though the sample may still mismatch its features.Source: §4.1 · p. 9
  • Local parameterizations use only a grid point’s local state and history, but may fail to capture spatial correlations.Source: §4.1 · p. 9

5. Numerical experiments

The experiments assess whether QSN-based resampling can reproduce time-averaged macroscopic statistics in the two-layer Lorenz 96 system. They test two parameter settings with no clear temporal gap between macroscopic and microscopic variables.

5. Numerical experiments

  • The reduced model targets time-averaged statistics of the macroscopic variables, not exact trajectories of the coupled system.The evaluation considers properties such as probability density and autocorrelation functions.Source: §5 · p. 10

5.1. Model equations Lorenz 96

  • The proof-of-concept benchmark is the two-layer Lorenz 96 system, a toy model for the atmosphere.Its N macroscopic ODEs are each coupled to L microscopic variables.Source: §5.1 · pp. 10–11
  • Two-step Adams–Bashforth integration with ∆t = 0.01 generates training data; prediction solves only the Xn equation using a QSN surrogate for rn.Source: §5.1 · p. 11
  • With ϵ = 0.5, the model has no clear temporal gap between macroscopic and microscopic variables, making parameterization more challenging.The experiments use unimodal and bimodal settings that differ in hx.Source: §5.1 · p. 11
  • The unimodal setting has nearly Gaussian Xn and rn distributions, whereas the hx = −2 setting has nonsymmetric, bimodal distributions.Figure 4 shows the Xn-versus-rn scatter plots for the two parameter settings.Source: §5.1 · pp. 11–12, Figure 4 · p. 13

5.2. Learning procedure

The QSN is trained on reference data, then evaluated within the coupled reduced model. The test assesses whether that system predicts held-out macroscopic statistics.

5.2. Learning procedure

  • The QSN uses three hidden layers of 256 leaky-ReLU neurons, RMSProp with learning rate 0.001, and 10,000 training iterations.Input features are standardized to zero mean and unit standard deviation.Source: §5.2 · p. 13
  • Experiments include both a local surrogate trained at one spatial location and a stochastic surrogate for the full vector r with N softmax layers.Source: §5.2 · p. 13
  • The evaluation withholds the final 50% of data and tests the coupled ODE–neural-network simulation against the macroscopic statistics of that test set.The neural network is evaluated as a source term in the reduced model, rather than as the final model by itself.Source: §5.2 · p. 13

5.3. Results: verification

The verification tests whether the QSN reproduces bin assignments from lagged conditioning features and reports low typical misclassification. A separate time-series comparison is used to check the random resampling software.

5.3. Results: verification

  • Figure 5 compares training-data bin indices with QSN-predicted indices across twice-lagged Xn features, and indicates that the network learns a good bin-index representation.Source: Figure 5 · p. 14, §5.3 · p. 14
  • Typical misclassification error for each softmax layer is roughly 3–4%.Source: §5.3 · p. 14
  • The authors plot the reference data ro j+1 and stochastic estimate ˜rj+1(do j) over time to verify the random resampling software.Source: §5.3 · p. 14

5.4. Results: validation

The validation compares reduced-model statistics with the full two-layer solution, testing memory depth, stochastic versus deterministic resampling, and X-only versus X–r conditioning. Short memory works in the unimodal case, while the harder bimodal case needs substantially more history; stochastic resampling helps the local surrogate, and adding r does not yield robust statistical predictions.

5.4. Results: validation

  • Statistics are evaluated on held-out data from t ∈ [500, 1000] after training on t ∈ [0, 500], and include distributions and temporal and spatial correlations for X and r.The simulations run from t = 0 to 1000; statistics are averaged over spatial locations.Source: §5.4 · p. 14

5.4.1. Short vs long memory

  • In the unimodal setting, a QSN using only X(j) and X(j−9) produces a good match to full-system statistics despite its short memory.The reduced model and full two-layer solution are compared in the probability-density, auto-correlation, and cross-correlation figures.Source: §5.4.1 · p. 15, Figure 7 · p. 16, Figure 8 · p. 16
  • With 75 lagged X vectors, the QSN captures the reference statistics in the bimodal setting, whereas 10 lagged vectors clearly fail.Tests with 25 and 50 lagged vectors also failed to yield good statistics.Source: §5.4.1 · p. 17, Figure 13 · p. 19, §5.4.2 · p. 19

5.4.2. Stochastic vs deterministic

  • For the local surrogate, stochastic resampling significantly affects performance; in the hx = −1 setting, it outperforms the deterministic surrogate on auto-correlation, with similar results for other statistics.For full-vector surrogates, the deterministic approach gave results similar to resampling.Source: §5.4.2 · pp. 17, 21, Figure 16 · p. 21

5.4.3. X-only vs X −r conditioning

  • Including r in the conditioning features can lower training error, especially for local surrogates, but did not produce robust statistical predictions.The authors note that this finding is in line with results for conditional GANs, where X-only models outperformed models conditioned on both X and r [15].Source: §5.4.3 · p. 21

5.5. Software

The source code used to generate the results is available for download.

  • The source code used to generate the results can be downloaded from.Source: §5.5 · p. 21

6. Discussion and future challenges

The discussion identifies memory, flow complexity, and training strategy as open challenges for extending the method beyond the tested Lorenz 96 settings. It also points to feature design and more complex flows as directions for future work.

6. Discussion and future challenges

  • Memory mattered for the challenging bimodal setting, while the optimal memory depth is problem dependent.For unimodal settings, the amount of memory was not of major importance; feature design is left as an open research question.Source: §6 · p. 22
  • Applying the approach to more complex flows remains future work, with clustering proposed for handling large output dimensions.A two-dimensional ocean model is identified as a test problem.Source: §6 · p. 22
  • Offline training gave satisfactory Lorenz 96 results, but could lead to online instabilities in more complicated problems.The authors identify training the network while it is part of the larger dynamical system as a direction of interest.Source: §6 · pp. 22–23

7. Conclusion

The method conditionally resamples subgrid-scale data to form a stochastic parameterization, and it was validated on Lorenz 96 at two parameter settings. Validation required sufficient time-lagged features, while the stochastic surrogate showed more robust overall performance.

7. Conclusion

  • A neural network predicts probabilities over non-overlapping output-data bins conditioned on time-lagged macroscopic features, then resamples a reference subgrid-scale point.The resulting stochastic surrogate replaces the original subgrid-scale term; validation examines long-term statistics of resolved macroscopic variables.Source: §7 · p. 23
  • The stochastic surrogate showed more robust overall performance across tests on Lorenz 96 at two parameter settings.The method was validated provided the feature vector included a sufficient number of time-lagged variables.Source: §7 · p. 23
Loading 2004.01457v1…