PaperlayerView exact revision on arXiv

Source-linked AI summary

Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers

Kiwon Um, Robert Brand, Yun, Fei, Philipp Holl, Nils Thuerey

arXiv:2007.00016v2physics.comp-phcs.LG

TL;DR

The paper asks how to reduce numerical errors in iterative PDE solvers when discretization produces imperfect trajectories. It trains neural correction functions with solver interaction through differentiable physics and finds stronger accuracy, generalization, and long-rollout stability across diverse PDEs, while training cost increases and randomized forcing limits gains in one scenario.

  • Problem

    Iterative PDE solvers accumulate numerical errors, while supervised models trained on pre-computed trajectories can encounter shifted states during recurrent inference.

  • Method

    The paper compares learning approaches and trains neural correction functions by coupling recurrent solver rollouts with differentiable physics during training.

  • Results

    Across explicit and implicit solvers and diverse PDEs, differentiable-physics training outperforms pre-computed-data approaches, with stable recurrent rollouts and improved out-of-distribution generalization.

  • Takeaways & Limitations

    Long look-ahead differentiable-physics training can reduce PDE-solver errors without increasing inference cost, while hybrid ANN–solver evaluation can substantially accelerate simulations.

  • Takeaways & Limitations

    Training with differentiable physics costs more computationally, and randomized forcing can limit the future behavior that corrections accurately anticipate.

Abstract

from arXiv · show

Finding accurate solutions to partial differential equations (PDEs) is a crucial task in all scientific and engineering disciplines. It has recently been shown that machine learning methods can improve the solution accuracy by correcting for effects not captured by the discretized PDE. We target the problem of reducing numerical errors of iterative PDE solvers and compare different learning approaches for finding complex correction functions. We find that previously used learning approaches are significantly outperformed by methods that integrate the solver into the training loop and thereby allow the model to interact with the PDE during training. This provides the model with realistic input distributions that take previous corrections into account, yielding improvements in accuracy with stable rollouts of several hundred recurrent evaluation steps and surpassing even tailored supervised variants. We highlight the performance of the differentiable physics networks for a wide variety of PDEs, from non-linear advection-diffusion systems to three-dimensional Navier-Stokes flows.

1 Introduction

The paper targets numerical errors introduced when PDEs are discretized and develops neural corrections that interact with iterative solvers during training. It evaluates this approach across canonical fluid and PDE settings, including three-dimensional flows.

  • Discretizing PDEs introduces numerical errors in computational solutions, motivating learning methods that recover components not well captured by physical models.
  • The paper learns correction functions for discretization errors and evaluates them locally to improve PDE-solver solutions.
  • Small supervised-model inference errors can accumulate during iterative solver rollouts, causing distribution shift, deteriorated inference, or solver explosions.
  • Differentiable simulations let neural networks interact with PDE solvers during training and receive feedback throughout solver iterations.
  • The study covers advection-diffusion systems, multiple Navier-Stokes variants, three-dimensional flows, and the Poisson problem with conjugate-gradient solvers.

2 Learning to Reduce Numerical Errors

The framework compares source and reference discretizations and trains a neural correction operator to reduce their trajectory error. Its central distinction is whether corrected solver states are generated during training.

  • Reference and source manifolds represent more accurate and approximate discretizations of the same PDE, with trajectories evolving through their respective solver operators.
  • Discretization differences can grow exponentially in chaotic systems, so the correction operator minimizes the L2 distance between corrected source evolution and projected reference states.
  • Learning methods are categorized by solver interaction: no interaction, pre-computed interaction, or tight coupling through a differentiable solver in training.
  • Non-interacting (NON): Non-interacting training uses unaltered source trajectories, exposing models only to states from the original source distribution and potentially causing strong error accumulation.
  • The look-ahead horizon n determines how many recurrent solver steps are evaluated per training iteration and mini-batch.

3 Experiments

Experiments span constrained advection-diffusion and Navier-Stokes systems, including buoyancy-driven and three-dimensional flows. Models are evaluated by multi-step mean absolute error against projected reference solutions.

  • The advection-diffusion models impose constraints on velocity through a constraint matrix M.
  • The experiments cover four fluid scenarios: Burgers’ equation, two-dimensional Navier-Stokes, buoyancy-driven flow, and three-dimensional Navier-Stokes.
  • Reference solutions typically use the same numerical method with a 4x finer discretization, producing effective resolutions of 128^2 and higher.
  • Correction networks use fully convolutional architectures that are intentionally simple because the study focuses on PDE-model incorporation during training.
  • Performance is measured by mean absolute error over n consecutive simulation steps, alongside relative improvement over the unmodified source trajectory.

4 Results

Learned correction functions improve PDE-solver accuracy across several scenarios, with differentiable-physics training producing especially strong gains and stable correction of complex flows.

  • More than 70% further improvement over supervised and pre-computed approaches was achieved when correction functions were trained with differentiable physics.The experiments also found substantial gains over regular simulations.
  • Unsteady Wake Flow: MAE fell from 0.146 for SRC to 0.013 with SOL32 in unsteady wake flow, reducing source-simulation numerical errors by more than a factor of 10.PRE reached 0.031, while NON reached 0.049, despite identical architecture and weight count across the compared models.
  • Buoyancy-driven Flow: SOL128 reduced buoyancy-driven-flow error from 1.59 for SRC to 0.62, an improvement of more than 59%.Long differentiable-physics rollouts were particularly beneficial in this scenario; PRE and NON achieved errors of 1.37 and 1.07.
  • Forced Advection-Diffusion: In forced advection-diffusion, SOL models did not significantly outperform simpler baselines because randomized forcing limits accurate prediction of future behavior.The appendix reportedly shows increased interaction steps help when external disturbances are absent.
  • Conjugate Gradient Solver: The CG solver’s differentiable version required fewer steps to reach the desired accuracy than a physics-based loss model, which required 63% more steps.Differentiable solver iterations provide look-ahead about how local errors, including boundary errors, affect future states.
  • Three-dimensional Fluid Flow: SOL16 stably rolled out for several hundred time steps in 3D flow and improved SRC numerical accuracy by more than 22% across configurations.The NON version did not consistently stabilize the flow, whose chaotic dynamics form partially turbulent eddies.

5 Ablations and Discussion

Ablations show that longer differentiable-physics look-aheads improve deterministic-system performance and generalization, while training stability and computational cost remain practical considerations.

  • Future Look-Ahead: Average improvements reached 60% with 128 recurrent solver iterations, exceeding 54% with 64 iterations for the buoyancy-driven scenario.Longer rollouts provide gradients that help the network anticipate physical behavior without changing its architecture or number of weights.
  • Generalization: All buoyancy-flow test simulations used out-of-distribution initial conditions yet showed good generalization to substantially different structures and velocity ranges.
  • Training with Noise: Input-noise training had a sweet spot near σ = 10^-4, with improvements of 34.5%.
  • Training Stability: Early training can diverge when corrections amplify oscillations, motivating pre-training with small look-aheads before longer rollouts.
  • Runtime Performance: Differentiable-physics training increases training-time cost because each iteration evaluates the PDE for n steps and computes gradients of similar complexity.In buoyancy flow, average time rose from 0.21 seconds per iteration for SOL2 to 0.42s for SOL4 and 1.25s for SOL16; inference cost is unaffected.
  • Runtime Performance: The hybrid method ran 100 time steps in 13.3s versus 913.2s for a CPU-based reference simulation.The fully convolutional CNN has O(n) cost for n degrees of freedom, contrasting with many PDE solvers’ super-linear complexity.

6 Conclusions

The paper demonstrates that differentiable-physics training with long look-ahead rollouts can substantially reduce numerical errors in PDE solvers. It also finds that solver interaction improves generalization and stability, supporting learned networks as components of computational science.

  • Differentiable-physics training with long look-ahead rollouts achieves significant reductions in PDE-solver numerical errors.The resulting models yield substantially lower errors than models trained with pre-computed data.
  • The approach could help establish trained neural networks as components in the numerical toolbox of computational science.
  • Identical networks perform significantly better when the solver is included in the learning loop.
  • Solver-in-the-loop training improves out-of-distribution generalization and enables stable long-term recurrent predictions.

Broader Impact

PDE-based models support applications across science and engineering, while integrating learned corrections into established solver pipelines remains difficult. The supplied material also documents the method’s computational context, learning variants, and potential harmful uses.

  • PDE-based models are used in weather, climate, epidemics, civil engineering, manufacturing, and medical applications.
  • Integrating deep learning into established PDE-solving pipelines is difficult, although the method could eventually improve accuracy at fixed computational resources.
  • The approach can also be used in developing military equipment or other harmful systems, creating an ethical consideration for deployment.
  • The method alternates local neural-network inference with PDE-solver steps rather than replacing PDE-based time integration with a learned version.
  • The solver-in-the-loop objective trains recurrent corrected trajectories generated by alternating PDE steps and learned corrections.
  • The learning setups comprise non-interacting, pre-computed-interaction, and tightly coupled differentiable-solver approaches.
  • Pre-computed interaction modifies source trajectories before supervised training so training inputs are closer to those encountered at inference.

A.2.2 Pre-computed Spatiotemporal Regularization

Pre-computed spatiotemporal regularization addresses the difficulty of learning highly nonlinear correction fields by encouraging temporal smoothness. Solver-in-the-loop training instead couples recurrent corrected trajectories directly to a differentiable PDE solver, with reported accuracy and distribution-shift benefits.

  • Per-time-step pre-computation can produce correction fields that vary sharply under smooth source-state changes, making them highly nonlinear to learn.
  • Temporal regularization penalizes changes in the correction field over time while preserving constrained transfer between fine and coarse discretizations.
  • PRE models use a pre-computation correction Cpre to generate modified training trajectories before learning the correction C.
  • Differentiable physics aims to bridge inference-time input-distribution changes by training directly with the environment used at inference.
  • Solver-in-the-loop training unrolls corrected trajectories through the PDE solver and computes loss across intermediate states before back-propagating through solver steps.
  • The reference solver need not be differentiable when a differentiable source-manifold solver is available.
  • The experiments use source and reference velocity-field pairs from simulation sequences with varying initial conditions, including projected references obtained through spatial downsampling.

B.1 Unsteady Wake Flow in Two Dimensions

The two-dimensional unsteady wake-flow experiments evaluate learned corrections for incompressible Navier–Stokes simulations across Reynolds numbers and 500-step rollouts. Solver-in-the-loop training, especially SOL32, produces the strongest and most stable error reduction.

  • Setup: The setup uses incompressible Navier–Stokes flow through a domain containing a circular obstacle, with pressure solved implicitly using Chorin projection.The flow has open boundaries and a prescribed inflow; the divergence-free constraint restricts the velocity field.
  • Setup: Training spans Reynolds numbers from 97.7 to 3125.0, while five test Reynolds numbers are held out and evaluated over 500 time steps.The training range covers more than a factor of 30 in Reynolds number, and each dataset contains 500-step sequences.
  • Results: The source simulation has MAE 0.146, whereas the non-interacting model reduces it to 0.049 on the held-out test Reynolds numbers.These values are means across all five test Reynolds numbers.
  • Results: The pre-computed variants roughly halve the remaining error, with PRE outperforming PRESR through spatiotemporal regularization.PRESR performs better than NON but worse than PRE.
  • Results: SOL32 reaches a final MAE of 0.013, substantially outperforming shorter-look-ahead SOL variants and matching the reference flow behavior more closely.The larger SOL32 model reduces MAE to less than 9% of the source-simulation error on average.
  • Results: SOL performance remains largely uniform across the five Reynolds numbers, while the non-interacting model develops slightly larger errors at higher Reynolds numbers.Frequency evaluation also shows that SOL32 can achieve lower frequency error than PRE when trained with sufficient look-ahead.
  • Ablation: With a 57k-parameter network, the interaction ranking remains NON below PRE below differentiable-physics interaction, although overall accuracy deteriorates.SOL16 still outperforms the other reduced-size variants, while NON reduces the error by approximately 30%.

B.2 Buoyancy-driven Fluid Flow

The buoyancy-driven-flow experiments test learned velocity corrections in a coupled fluid–marker system with more complex swirling dynamics and mismatched reference/source discretizations. Solver-in-the-loop models yield substantial improvements and generalize to out-of-distribution initial conditions.

  • Setup: The scenario couples incompressible fluid motion with an advected marker field that generates buoyancy forces, producing complex swirling behavior.The marker is not directly corrected by the network; it changes through advection by the corrected velocity.
  • Setup: Reference trajectories use a 4× finer spatial discretization and a higher-order advection scheme than the source simulations.This setup tests correction learning when reference and source solvers differ.
  • Results: SOL variants yield very significant improvements over the NON and PRE learned baselines, with velocity and density gains increasing for larger look-ahead horizons.Kinetic-energy improvements are smaller than those for velocity fields.
  • Generalization: The models generalize well to out-of-distribution initial marker configurations that produce substantially different structures and velocity ranges.The test simulations use initial conditions outside the training parametrization.
  • Ablation: Increasing network size eventually saturates: a model exceeding 100k weights improves performance by 3.6%, while a fourfold further increase adds only 0.3%.The architecture ablation uses SOL2 interaction.
  • Comparison: Noise perturbations improve results by up to 34.6%, whereas the best SOL model achieves a 59.8% improvement.The authors attribute the difference to systematic solver-in-the-loop training rather than stochastic perturbations.
  • Conclusion: Overall, solver-in-the-loop training substantially reduces source-simulation errors despite complex dynamics and difficult reference trajectories.The conclusion covers the buoyancy-driven-flow scenario evaluated against higher-order reference data.

B.4 Inference of Initial Guesses for Conjugate Gradient Solvers

This section compares supervised, residual-based, and solver-in-the-loop methods for initializing conjugate-gradient pressure solves. Training through iterative solver steps improves convergence and captures large-scale solution structure.

  • Methods: Three approaches are compared: supervised NON, residual-based SOLDIV, and solver-in-the-loop SOLn models initialized before conjugate-gradient solving.SOLDIV incorporates the PDE in the loss, whereas SOLn minimizes the residual after n differentiable CG iterations.
  • Results: 2 iterations reach 10^-2 accuracy with SOL5, compared with 9 for NON, 28 for SOLDIV, and 78 from a zero guess.These averages are reported over test cases and show the strongest early convergence for SOL5.
  • Results: SOL5 overtakes the other methods within the first five to ten CG iterations and retains its advantage as solving continues.Its initial pointwise accuracy is only slightly better than the zero guess, but subsequent CG interaction improves convergence rapidly.
  • Interpretation: Solver-in-the-loop training supplies feedback about CG behavior, helping SOL5 match large-scale structures that SOLDIV leaves for later iterations.SOLDIV minimizes a pointwise PDE residual without iterating the solver during training, while SOLn back-propagates through n solver iterations.
  • Ablation: Look-ahead ablations show that too few iterations deteriorate performance, whereas more than 5 iterations slightly increase the iterations required.The authors attribute this pattern potentially to evaluating the loss only at the final output of the unrolled steps.
  • Discussion: The discussion emphasizes that solver-in-the-loop training provides gradient-based feedback unavailable to a loss-based residual alone, while hybrid solving preserves convergence guarantees.A traditional solver can continue improving a sub-optimal learned initial guess until the target accuracy is reached.

B.5 Three-dimensional Unsteady Wake Flow

The 3D cylindrical-obstacle wake flow increases complexity through additional rotational structure and chaotic, partially turbulent eddies. Solver-in-the-loop training produces stable corrections and better long-horizon behavior than the supervised variant.

  • Setup: The 3D setup uses incompressible Navier–Stokes flow around a cylindrical obstacle in a 1 × 1 × 2 domain.Reference and source resolutions differ substantially, and testing uses Reynolds numbers not seen during training over 300 time steps.
  • Training: SOL16 uses a differentiable Navier–Stokes solver unrolled for up to 16 simulation and inference steps during training.The model was first trained with SOL8 for 200k iterations and then with SOL16 for an additional 100k iterations.
  • Results: The source, NON, and SOL16 velocity-field MAEs are 0.167, 0.143, and 0.130, respectively.The SOL16 model also remains stable over several hundred simulation steps, unlike the unstable NON rollout.
  • Results: SOL16 closely matches reference kinetic-energy frequency distributions and corrects vortex shedding at Reynolds numbers absent from training.NON initially performs well but later oscillates and distorts the velocity field, while SOL16 retains stable long-horizon behavior.

C Performance

The paper evaluates runtime for hybrid PDE solvers that alternate source-solver evaluations with neural-network corrections. These measurements show substantial speedups, while the authors characterize them as preliminary.

  • Benchmark: The benchmark compares a CPU OpenMP reference simulator with a GPU TensorFlow implementation combining the differentiable PDE and trained model.The comparison uses an Intel Xeon E5-1650 CPU and NVIDIA GeForce GTX 1080 Ti GPU.
  • Results: For buoyancy-driven flow, 100 reference time steps take 5.79 seconds, while SOL128 inference takes 0.43 seconds.The source solver alone takes 0.476 seconds for the same 100 time steps.
  • Results: The 3D wake-flow reference requires 913.2 seconds for 100 steps, versus 13.3 seconds for SOL16, making the learned-correction source simulation more than 68 times faster.The paper notes that training iterations are substantially more expensive than forward inference, but training is a one-time preprocessing cost.
  • Caveat: The reported performance results are preliminary and may fall short of the speedup achievable in optimally engineered learning-augmented PDE solvers.The authors identify optimization of the source solver, gradient provision, and neural-network hardware as possible sources of additional gains.

D Neural Network Architectures

The paper uses varied neural-network architectures across scenarios rather than relying on one architecture. Most PDE interaction models use ResNets, while the CG scenario uses a different design.

  • Architecture choices: Architectures are intentionally varied across the five scenarios to demonstrate that solver-in-the-loop training does not depend on one specific network design.The architecture choices reflect differences among correction tasks and solver interactions.
  • Architecture choices: Most PDE interaction models use ResNets because correction learning resembles mapping phase-space quantities to localized correction fields.The CG solver scenario requires a different architecture.
  • Architecture overview: Figure 28 summarizes the two main neural-network architectures used for Sections B.1–B.3 and B.4.The figure presents one architecture grouping for the earlier PDE cases and another for the CG case.
Loading 2007.00016v2…