Source-linked AI summary
When a neural surrogate cannot accelerate a solver: runtime share, closed-loop drift, and the economics of uncertainty gating in a stiff coupled simulation
L. Thümmler, T. Kuroda
TL;DR
Learned surrogates are often proposed to speed expensive inner solver blocks, but evidence from end-to-end multiphysics runtimes remains limited. This paper evaluates that proposal in a neutrino–matter coupling solve using complete-step profiling, surrogate replacement, multi-network comparisons, and deployment gating. The surrogate ties the solver at best, while runtime-share limits, confounded offline metrics, and distribution shift prevent acceleration.
Problem
Evidence is limited on whether learned surrogates for expensive inner solver blocks deliver end-to-end speedups in production multiphysics simulations.
Method
The paper evaluates surrogate replacement in a neutrino–matter coupling solve using end-to-end profiling, fourteen-network comparisons, and out-of-distribution gating.
Results
A 5.8× cheaper-per-call surrogate merely ties the solver end-to-end, consistent with a ≈1.20× Amdahl limit from the target routine’s 16.9% critical-rank runtime share.
Takeaways & Limitations
Per-call cost is insufficient to predict acceleration; deployment decisions require runtime-share measurement and end-to-end evaluation against the production solver.
Takeaways & Limitations
Single-run survival horizons are unreliable evidence of stability, as five fresh seeds survived 666–1862 steps versus an original 2625-step measurement.
Abstract
from arXiv · showhide
Learned surrogates for expensive inner solver blocks are a widely pursued route to faster multiphysics simulation. We report a controlled, end-to-end negative result and identify three structural barriers, none of them a deficiency of the network we trained. The testbed is the implicit Newton solve coupling energy-dependent neutrino radiation to matter in a general-relativistic radiation-hydrodynamics code, its most expensive physics routine per call. First, per-call cost and share of runtime are different quantities, and only the second bounds acceleration. An exclusive self-time profile puts the target block at 16.9% of critical-rank wall clock, capping any surrogate at ~1.2x by Amdahl's law. A surrogate 5.8x cheaper per call merely ties the solver, and the configuration stable enough to run without fallback reaches only parity. Second, offline accuracy cannot rank surrogates for deployment: across fourteen networks the pooled Spearman error-versus-survival correlation (rho = +0.73) is a between-family confound that vanishes under control (rho = -0.04). Third, a correct out-of-distribution gate cannot accelerate a loop that leaves its training distribution. We give the break-even deferral fraction in closed form: because the visited states sit 73x off the data manifold, the gate defers 96.8 to 99.7% of cells, almost invariant to surrogate quality. Including its own cost, the gated loop is a 0.94 to 0.96x slowdown. We further separate stability from fidelity: a never-crashing gated run accumulates a linear -19.9% density bias over 6000 steps. The error is a directed, ballistically accumulating bias, not the variance-driven divergence the autoregressive literature targets.
1. Introduction
Replacing an expensive solver block with a fast learned surrogate does not necessarily accelerate the full simulation: runtime share and closed-loop behavior impose constraints that offline benchmarks do not measure. This paper evaluates those constraints end to end in an implicit neutrino–matter coupling solve and reports measured acceleration limits.
- Motivation: Offline surrogates can exceed 0.99 held-out R^2 and deliver one- to two-order-of-magnitude per-call speedups, but these benchmarks do not establish faster simulations.The introduction distinguishes per-call performance from end-to-end acceleration.
- Structural barriers: Runtime share bounds end-to-end acceleration by Amdahl’s law, while closed-loop error compounding makes per-step accuracy a poor deployment predictor.The paper identifies these as two quantities offline benchmarks do not measure.
- Testbed: 3.4 × 10−4 s per cell per step makes the implicit neutrino-matter coupling solve the code’s most expensive physics routine per call, called once per cell per step.The testbed is a one-dimensional general-relativistic radiation-hydrodynamics code.
- Contribution: The paper explicitly reports negative and null results alongside positive ones, separating measured claims, modelled claims, and open questions.The stated structure covers the testbed, protocol, results, discussion, guidance, and conclusion.
- Measured acceleration budget: 16.9% of critical-rank wall clock limits any surrogate to ≈1.2×, despite the target routine being the motivating per-call optimization target.The introduction also reports 10.5% of mean wall clock and distinguishes that share from per-call cost.
2. Problem setting
The study targets a stiff, per-cell Newton solve coupling matter to multi-group M1 neutrino radiation in a general-relativistic radiation-hydrodynamics code. Its spatially uncoupled state-in/state-out interface enables a residual-MLP surrogate, evaluated across distinct collapse regimes using independent numerical trajectories.
- Solver interface: Each grid cell and time step invokes NeuMatIntImplicit to solve stiff matter–radiation coupling from local hydrodynamic and 20-group radiation inputs.The logged interface includes density, entropy or temperature, electron fraction, time step, conserved energy, momentum, and incoming radiation moments.
- Solver interface: The per-cell state-in/state-out interface has no explicit spatial coupling, omitting mesh topology, neighbor stencils, and time history.Despite this straightforward interface, radiation updating consumes roughly 80% of per-call solver time versus 20% for matter.
- Simulation regimes: The collapse is divided into infall, bounce, and post-bounce regimes because numerical behavior does not necessarily transfer between them.Results are reported separately where regime distinctions matter.
- Data and model: The dataset uses one 9.6 M⊙ z9.6 progenitor with the DD2 equation of state, providing 2.74M training records and 4.3M independent full-resolution test records.The independent run follows a different numerical trajectory through state space.
- Data and model: The selected surrogate is a residual multilayer perceptron with 1.8M parameters, using transformed and standardized inputs and predicting residuals in transformed space.The architecture sweep found width-512/3 superior to width-256/6, with width helping more than depth.
3. Evaluation protocol
The evaluation substitutes a frozen offline-trained network into a fresh collapse and judges deployment by closed-loop survival and physical validity. Matched comparisons, failure localization, and repeated training seeds distinguish surrogate behavior from host-code noise and measurement artifacts.
- End-to-end substitution test: The decisive test replaces NeuMatIntImplicit with the frozen network in a fresh collapse, then measures survival steps and whether the state remains physical.The network sets conserved variables and electron fraction; the equation-of-state epilogue recovers the primitive state.
- End-to-end substitution test: The protocol is an offline-trained substitution test, with crashes defined by host-code equation-of-state range and causality guards rather than numerical convenience.These guards make failure a physically meaningful violation.
- Failure localization: Selective identity-map replacements isolate which predicted output channel causes a failure, while setting every channel to identity validates the framework independently of the surrogate.The inert-network control leaves the coupling inactive while preserving the evaluation framework.
- Controlled comparisons and profiling: End-to-end comparisons use identical starting states, step counts, thread and rank counts, startup exclusion, bidirectional run order, and exclusive self-time profiling.Disjoint profiling leaves are validated to sum to measured sweep duration within 0.2%.
- Seed replication and survival analysis: Five or more retrainings per configuration vary only the random seed, restart from the same verified state, and report distributions and rank tests instead of point comparisons.The identical data partition and training recipe are retained across seed variations.
- Seed replication and survival analysis: The deterministic simulation produces identical step counts across five repeat runs, so seed-dependent spread measures learned-model sensitivity rather than host-code numerical noise.The simulation contains no random number generator; seeds vary training initialization and shuffle order.
4. The acceleration budget
The target routine is expensive per call but occupies only 16.9% of critical-rank wall clock, limiting ideal end-to-end acceleration to about 1.20×. A surrogate 5.8× cheaper per call therefore only ties the production solver, while load imbalance and communication constrain further gains.
- Runtime share and Amdahl bound: 16.9% of critical-rank wall clock bounds an ideal replacement at about 1.20× by Amdahl’s law.The corresponding mean wall-clock share is 10.5%, while a balanced compute-only idealisation reaches 27.4% and about 1.38×.
- Runtime share and Amdahl bound: 5.8× cheaper per call merely ties the solver end-to-end, at 0.296 against 0.318 s per step.The width-256 batched surrogate costs 5.9 × 10−5 s per cell versus 3.4 × 10−4 s for the solver.
- Scaling and limitations: 11.3% solver share after eightfold per-rank volume growth still leaves communication a real but minority effect, not a solver-dominated regime.The solver share rises from 7.5% to 11.3%, while surface-bound communication falls from 13.5% to 8.4%.
- Load imbalance: 1.82× compute imbalance on the busiest rank makes load imbalance a larger wall-clock lever than the coupling solve.The busiest rank performs 53.2 s versus 29.3 s for the mean rank post-bounce, with modeled balancing yielding 1.13 to 1.40×.
- Scaling and limitations: Removing solver cost can migrate the critical path to an idle-dominated rank, making realized gains smaller than a naive Amdahl estimate.The decomposition is static and contiguous in radius, and a quick non-uniform rebalance would require architectural changes and a broad routine audit.
5. Per-step accuracy is not a deployment metric
Per-step accuracy is not a reliable deployment metric: strong in-distribution and one-step results coexist with severe closed-loop failures. Across surrogates, offline error provides no usable within-configuration ranking signal for survival or physical validity.
- In-distribution accuracy: R2 = 0.982 on a held-out training-distribution split is optimistic because the random split leaks near-duplicate states from one autocorrelated trajectory.On the never-trained trajectory, energy-change R2 falls to 0.06 to 0.32 and momentum-change R2 to 0.29 to 0.45.
- Closed-loop behavior: R2 = 0.98 per call does not prevent a network from terminating the live simulation within two steps.The naive surrogate crashes within tens of steps, demonstrating that per-call accuracy does not ensure closed-loop survival.
- Configuration comparisons: Median survival rises from 21 to 47 steps with density-based routing to two narrow experts, while a single narrow specialist reaches 75 steps.Routing costs no additional inference because exactly one expert is evaluated per cell; specialisation carries most of the benefit.
- Offline ranking: 0.150 versus 0.023 one-step error can accompany survival medians of 75 versus 21, while matched models differing by 2.95% offline error survive 56 versus 397 steps.Unified evaluation sets reveal that own-set scoring can create an apparent monotone relationship, whereas common-set scoring shows no usable ranking resolution.
- Offline ranking: ρ = +0.73 across fourteen surrogates reverses to ρ = −0.04 after controlling for narrow versus broad family, showing pooled correlation is a between-family confound.Within-family correlations are not statistically significant, so offline error does not rank survival within a configuration.
6. Closed-loop failure: diagnosis
Closed-loop failure stems first from an unlearnable conserved-energy increment, then from distributed off-manifold drift. Reconstructing energy through the equation of state removes density collapse, but channel-wise safeguards do not stop the remaining instability.
- Unlearnable energy increment: 59% of 105 cell updates have bit-exact zero relative energy change, while transformed-target signal is 1.5 × 10−11 versus 8.3 × 10−8 prediction error.The median relative change is exactly zero, its 99th percentile is 5.7 × 10−5, and the maximum is 3 × 10−4; the signal-to-error gap is ∼5500×.
- Conservation diagnosis: A conservation-by-construction layer is structurally infeasible because the representable radiation-energy change is paired with a matter-energy change below floating-point resolution.The apparent conservation residual is the same sub-resolution mismatch, not a distinct physics violation.
- Algebraic reconstruction: 102 steps is the extended closed-loop survival after predicting Ye and entropy and computing conserved energy exactly through the equation of state.This construction removes density collapse and preserves consistency with the conserved rest-mass density.
- Off-manifold drift: 73× larger nearest-neighbour distance, 61% beyond the 99th-percentile Mahalanobis shell, and entropy reaching 62 versus a real maximum of 7.8 characterize visited states.These thermodynamic states lie outside trajectories occupied by real cells, so adding more real data cannot address the shift.
- Distributed instability: ∼550 steps is the unchanged crash time after clamping electron fraction, clamping entropy, or flooring density, because the abort migrates to another guard.The result indicates distributed whole-state drift rather than failure of a single channel.
7. Stabilisation strategies and their limits
Stabilisation attempts largely failed to remove drift, while dataset aggregation and contractivity extended closed-loop survival only under specific conditions. A per-cell gate stabilised the full test window but delivered no acceleration, underscoring that stability and speed remain distinct.
- Training and re-anchoring: Mean closed-loop Ye error 0.61 was unchanged by eight-step rollout training, which failed to improve long-horizon stability.Training on one trajectory did not cover the off-trajectory states visited during deployment.
- Training and re-anchoring: Horizon 557 still crashed after re-anchoring every 50 steps, while N ∼3 would forfeit the speedup.The drift spreads across the state before a single true solver correction can recover it.
- Dataset aggregation: Best-horizon selection and recency weighting raised survival to 557 steps, 5.5× the baseline (102), whereas naive aggregation oscillated without converging.Naive horizons followed 102 →150 →116 →286 →163, and the final model was not the best.
- Contractivity and reproducibility: ∼36% longer horizon from contractivity stacked with aggregation to reach 2625 closed-loop steps, 26× the un-aided baseline, but five fresh seeds had median 1086.The five fresh runs survived 666, 985, 1086, 1183 and 1862 steps, placing 2625 outside their distribution.
- Regime transfer and gating: Mean post-bounce survival fell from 64 steps to 6 after aggregation, while a per-cell gate stabilized more than 1000 steps but saved nothing.Aggregation helped pre-bounce yet worsened all five post-bounce model pairs; the gate enforces the identity where surrogate calls would occur.
8. The economics of uncertainty gating
Uncertainty gating is economically restrictive: although the detector prevents crashes, it routes nearly every cell to the exact solver, producing a measured slowdown rather than acceleration. Its deferral rate is driven by how far states leave the training manifold, not by surrogate accuracy.
- The economics of uncertainty gating: Break-even deferral is independent of runtime share, but gating cannot amortise when g > 1 − r; with r = 0.17, this requires g > 0.83.The runtime share determines the magnitude of the gain or loss, not whether gating breaks even.
- The economics of uncertainty gating: 0.996 area under the curve separates large blow-up errors, while 0.66 detects small accumulating drift errors only weakly.The gate prevents the crash but is much less sensitive to gradual drift.
- The economics of uncertainty gating: 96.8% median deferral leaves the network evaluating only 3.2% of cells, because the visited states have broadly left the training manifold.The 77.3% minimum occurs only during the startup transient.
- The economics of uncertainty gating: 0.96× end-to-end performance at f = 0.119 and 0.94× at f = 0.169 show that gated execution is a measured slowdown, including detection cost.These values use g = 0.391, d = 0.968, and r = 0.17.
- The economics of uncertainty gating: 0.96 area under the curve versus 0.91 improves drift detection, and 1365 versus 1038 steps extends stability, yet the better detector still defers 94% of cells.The ensembledisagreement detector improves the stable horizon without changing the fundamental economics of near-total solver routing.
9. Stability is not fidelity
Crash-freedom does not establish trajectory fidelity: the gated run remained stable while central-density error grew linearly to −19.9%. This drift was non-self-correcting and was measured only in the pre-bounce regime.
- Stability is not fidelity: 99.88% of cells were deferred to the solver while the network evaluated 0.12% in the matched gated run.The run began from an identical verified state and was compared with a solver-only reference through 6000 pre-bounce steps.
- Stability is not fidelity: −19.9% central-density relative error accumulated by the end of the 6000-step run despite the gated loop never crashing.The error stayed below 0.2% for roughly 609 steps, then grew linearly with fitted slope −3.6 × 10−5 to −3.7 × 10−5 per step.
- Stability is not fidelity: The central-density error grew linearly rather than accelerating, consistent with directed bias rather than variance-driven instability.The linear fit had R2 > 0.993 across each reported measurement window.
- Stability is not fidelity: Deferring all but a fraction of a per cent of cells to the exact solver did not pull the trajectory back, making the drift non-self-correcting and invisible to crash criteria.A stability-only evaluation would therefore have reported the run as a success.
- Stability is not fidelity: The measurement covered only the pre-bounce regime, so whether the drift remains linear, accelerates, or saturates after bounce is not established.The fixed benchmark ended well before bounce, and the gate’s deferral fraction was at its most conservative in this test.
10. Mechanism: a directed bias, not a variance blow-up
The surrogate’s failure is driven by a small but systematic directed bias that accumulates ballistically, carrying states off-manifold and forcing any correct gate to defer nearly every cell. Capacity, data expansion, conservation correction, and subcycling do not solve this regime, while pushforward training and timestep rescaling provide only limited survival gains.
- Mechanism: Directed bias accumulates linearly with rollout steps, progressively moving states off the training manifold until out-of-distribution gating defers almost every cell.The resulting feedback loop makes the surrogate less accurate as departure accelerates, preventing acceleration.
- Mechanism: The failure differs from variance-driven autoregressive instability: input-noise damping and pushforward stability losses target fluctuations that are sub-dominant to systematic drift here.The study therefore evaluates remedies against directed, ballistically accumulating error rather than zero-mean noise.
- Evidence for directed drift: |bias|/MAE = 0.249 ± 0.186 post-bounce, while directed electron-fraction bias remains 0.60 to 0.85 across six independent progenitors against a 0.60 trained-holdout baseline.The bias persists across progenitors spanning 12 to 40 M⊙, and the entropy-increment value is 0.40 ± 0.28.
- Alternative interventions: Rescaling the predicted increment extended post-bounce survival from 50 to 59 steps, an 18% gain that was of no practical use.The lever reduces per-step bias by scaling with dt/τε, but the loop’s failure mode remains insufficiently addressed.
- Alternative interventions: Pushforward training extended closed-loop survival from 31 to 65 steps, but the guard eventually fired at a low-density outer cell outside the dense-core training regime.Offline, the plain model’s radiation-channel error grew roughly fortyfold over eight steps, whereas the pushforward-trained model’s error stayed flat at a few-fold worse cost.
- Controls: Subcycling produced no discernible speedup, with measured wall clocks of 0.270, 0.279, 0.276, and 0.256 s per step, while every configuration reached an unphysical end state.The unmodified solver completed the full 20 ms window without incident, with central density flat within 0.1% of nuclear saturation over its final 3000 steps.
11. Discussion and transferable guidance
The discussion identifies runtime share, in-loop deployment behavior, and gating economics as measurable barriers that can determine surrogate acceleration before substantial modeling effort. It also distinguishes stability from fidelity and recommends matching remedies to the error’s character and validating them in the regime they target.
- Runtime economics: 16.9% of critical-rank wall clock was the target block’s runtime share, so a surrogate 5.8× cheaper per call only tied the solver.Validated critical-path share, not per-call cost, bounds achievable acceleration.
- Deployment evaluation: Fourteen-model offline metrics did not rank deployments: the pooled correlation was a between-family confound, with no within-family resolution.Model selection should therefore occur in-loop on deployment distributions.
- Gating economics: 1.6 percentage points was the survival change across a fourfold variation, indicating that deferral was governed by physics-driven drift rather than detector or surrogate quality.Equation (2) permits gating economics to be estimated from per-call cost ratio and deferral fraction before implementation.
- Acceptance criteria: 99.88% of cells were deferred in a never-crashing gated run, yet central density drifted −19.9% over 6000 steps.Stability and fidelity must be treated as separate acceptance criteria.
- Transferability and error character: A directed bias accumulates linearly, whereas variance-driven blow-up requires different remedies; applying variance-focused methods to bias only damps a sub-dominant term.The claims are bounded by measurements from one code, progenitor, and node configuration, while the favorable test case had an ideal interface, abundant exact labels, and R2 = 0.98.
- Validation regime: 5.5× horizon gain pre-bounce became a 3.9 to 19× degradation post-bounce across five paired seeds after dataset aggregation.Stabilizations should be validated in the regime they are intended to fix.
12. Conclusions
No tested neural-surrogate configuration accelerated the simulation end-to-end because runtime share, deployment correlations, gate economics, and closed-loop behavior impose structural barriers. The conclusions instead identify constructive stability techniques, exact-solver restructuring, and an on-trajectory evaluation methodology.
- No tested configuration accelerated the simulation end-to-end, despite targeting the code’s most expensive per-call routine with history-free interfaces and abundant exact labels.The reported barriers are structural rather than deficiencies of the trained network.
- 16.9% of critical-rank wall clock caps surrogate acceleration at ≈1.2×; a 5.8× cheaper call only ties the solver, while the stable surrogate reaches 0.96× end-to-end.The stable surrogate was 1.3× cheaper on the timed inference path and 3.6× cheaper when batched, yet achieved parity end-to-end (p = 0.41).
- −19.9% central-density bias accumulated linearly over 6000 steps in a gated run that deferred 99.88% of cells, showing crash-freedom does not imply fidelity.Stability and fidelity therefore remain separate acceptance criteria.
- 2 →102 steps resulted when predicting the learnable thermodynamic state and deriving remaining quantities through exact algebraic relations; dataset aggregation and contractivity penalties reached 1086 median closed-loop steps.The 1086-step result was measured across five independently seeded trainings with best-model retention and a recency window.
- A restructured exact solver, rather than a learned surrogate, is identified as the credible route to speedup; Laiu et al. (2020) report up to 100× on GPU for the same physics.Static-decomposition load imbalance also dominates the profile and is not a machine-learning problem.
- The transferable methodology is to measure runtime share first, validate in-loop across distributions, check gate economics against break-even, and evaluate stability separately from fidelity.The conclusions emphasize that per-step accuracy alone can be misleading and that trustworthiness requires on-trajectory validation.
CRediT authorship contribution statement
L. Thümmler led the study’s conceptualization, methodology, analysis, investigation, software, validation, data curation, writing, and visualization, while T. Kuroda contributed the original radiation-hydrodynamics code, resources, and manuscript review.
- Contributions: L. Thümmler handled the study’s core research, analysis, software, data, writing, and visualization contributions.T. Kuroda provided the original radiation-hydrodynamics code, resources, and writing review.
Funding
The research received no specific grant from public, commercial, or not-for-profit funding agencies.
- The research received no specific grant from public, commercial, or not-for-profit funding agencies.