PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Neural Operators for robust and accurate learning of PDEs

Bogdan Raonić, Roberto Molinaro, Tim De Ryck, Tobias Rohner, Francesca Bartolucci, Rima Alaifari, Siddhartha Mishra, Emmanuel de Bézenac

arXiv:2302.01178v3cs.LG

TL;DR

Learning PDE solution operators requires surrogates that avoid the grid-resolution dependence and continuous-discrete inconsistencies affecting some existing approaches. The paper introduces CNOs, which modify CNN operations to preserve bandlimited function structure, and proves universality for PDE operators. Across representative PDE benchmarks, CNO is reported as on par with or outperforming tested baselines in both in-distribution and out-of-distribution testing.

  • Problem

    CNN-based operator learning has been limited by grid-resolution dependence, aliasing, and possible inconsistency in function spaces, despite the computational cost of conventional PDE solvers.

  • Method

    CNO modifies convolution, upsampling, downsampling, and nonlinear activation operations so a U-Net-like architecture maps bandlimited functions while preserving continuous-discrete equivalence.

  • Results

    CNO is on par with or outperforms tested baselines on all representative PDE benchmarks in both in-distribution and out-of-distribution testing.

  • Takeaways & Limitations

    CNO provides a convolution-based operator-learning framework with representation equivalence, a universality theorem, and reported empirical performance across diverse PDE benchmarks.

  • Takeaways & Limitations

    The presented CNO targets operators on two-dimensional Cartesian domains; three-dimensional extensions are computationally demanding and non-Cartesian domains require transformation maps.

Abstract

from arXiv · show

Although very successfully used in conventional machine learning, convolution based neural network architectures -- believed to be inconsistent in function space -- have been largely ignored in the context of learning solution operators of PDEs. Here, we present novel adaptations for convolutional neural networks to demonstrate that they are indeed able to process functions as inputs and outputs. The resulting architecture, termed as convolutional neural operators (CNOs), is designed specifically to preserve its underlying continuous nature, even when implemented in a discretized form on a computer. We prove a universality theorem to show that CNOs can approximate operators arising in PDEs to desired accuracy. CNOs are tested on a novel suite of benchmarks, encompassing a diverse set of PDEs with possibly multi-scale solutions and are observed to significantly outperform baselines, paving the way for an alternative framework for robust and accurate operator learning. Our code is publicly available at https://github.com/bogdanraonic3/ConvolutionalNeuralOperator

1 Introduction.

PDE operator learning seeks fast surrogates because conventional numerical solvers can be prohibitively expensive for high-dimensional and repeated-query problems. CNO addresses unresolved expressivity, aliasing, and continuous-discrete consistency issues while retaining convolutional architectures for operator learning.

  • Motivation: PDE solution operators map function-space inputs, such as initial and boundary conditions, to solutions in suitable function spaces.These operators are the objects learned in operator-learning approaches.
  • Motivation: Conventional numerical methods can be prohibitively expensive in high dimensions and repeated-query applications, motivating fast, robust, and accurate data-driven surrogates.The cited applications include uncertainty quantification, inverse problems, and PDE-constrained control and optimization.
  • Open challenges: Existing operator-learning methods face issues including limited expressivity, aliasing errors, and possible inconsistency across function spaces and resolutions.Continuous-discrete equivalence is identified as necessary for learning the underlying operator rather than only a discrete representation.
  • Open challenges: Naive CNN-based operator learning can depend heavily on grid resolution, despite CNNs offering locality, computational and data efficiency, and multiscale processing.The paper motivates adapting CNNs because they are widely successful in other machine-learning settings and resemble finite-difference discretizations.
  • Proposed approach: CNO modifies CNNs to enforce structure-preserving continuous-discrete equivalence and enable alias-free operator learning through an operator-adapted U-Net.The architecture is designed to process functions as inputs and outputs.
  • Evaluation: CNO is tested on representative PDE benchmarks spanning linear elliptic and hyperbolic, nonlinear parabolic and hyperbolic PDEs with possibly multiscale solutions.The benchmarks evaluate both in-distribution and out-of-distribution performance.
  • Evaluation: CNO is reported as on par with or outperforming tested baselines across all benchmarks in both in-distribution and out-of-distribution testing.The paper presents this empirical performance alongside desirable theoretical properties.

2 Convolutional Neural Operators.

CNO restricts computation to bandlimited function spaces and composes structure-preserving convolution, resampling, activation, and U-Net-style blocks. This construction yields resolution-invariant discrete representations and a representation-equivalent neural operator.

  • Bandlimited formulation: CNO approximates continuous operators between bandlimited spaces, chosen because general Sobolev spaces are too large to support the required continuous-discrete equivalence.Bandlimited approximation permits discrete representations based on the sinc basis.
  • Architecture: The architecture lifts an input function into a latent bandlimited space, processes it through function-to-function layers, and projects the final function into the output space.The lifting and projection operations are convolutional.
  • Architecture: CNO layers combine upsampling or downsampling, convolution, and activation operators, with modified U-Net blocks adding ResNet and invariant components.The resulting architecture uses downsampling, upsampling, ResNet, and invariant blocks.
  • Convolution: CNO convolutions use discrete physical-space kernels on grids satisfying the Whittaker-Shannon-Kotelnikov sampling condition.Unlike FNO, the kernel is directly parametrized in physical space, giving the parametrization a local nature.
  • Activation: The activation layer upsamples before applying the nonlinearity and downsamples afterward so outputs return to the desired bandlimit.This modification addresses the high-frequency features and aliasing that naive pointwise nonlinearities can generate.
  • Theoretical property: CNO is proven to be a representation-equivalent neural operator, so it can be evaluated on multiple grid resolutions without aliasing errors.Representation equivalence provides a form of resolution invariance.

3 Universal Approximation by CNOs.

The paper establishes that CNOs can approximate solution operators for a broad class of PDEs under continuity and regularity assumptions. The theorem also extends to PDEs with additional source terms.

  • Universality theorem: The universality result covers a large class of operators arising from PDEs under the specified abstract setting.The paper proves a more general version that includes additional source terms.
  • PDE setting: The target PDE has a differential operator depending on spatial coordinates through a coefficient function, with a continuous solution operator mapping coefficients to solutions.The setting is posed on the two-dimensional torus and assumes a modulus of continuity.
  • Universality theorem: Under stated regularity, continuity, and boundedness assumptions, for any ε > 0 there exists a CNO approximating the PDE solution operator to the theorem’s prescribed accuracy.The theorem requires σ ∈ N0, p ∈ {2, ∞}, r > max{σ, 2/p}, and B > 0.

4 Experiments.

The experiments introduce diverse, multiscale PDE benchmarks and compare CNO with established operator-learning and neural-network baselines. CNO performs competitively or better across tested tasks, generalizes to unseen inputs, and offers advantages in resolution robustness, computational efficiency, and data scaling.

  • Representative PDE Benchmarks: The Representative PDE Benchmarks provide diverse, computationally demanding problems with accessible data for rapid prototyping and reproducibility.The suite emphasizes PDEs with sufficiently many spatial and temporal scales, where fast surrogates are worthwhile.
  • Results: CNO is the best-performing architecture on every RPB task except Allen-Cahn, outperforming FNO, U-Net, and ResNet across the considered benchmarks.On the Poisson test case, CNO outperforms FNO by almost a factor of 20.
  • Out-of-Distribution Testing: CNO generalizes well in zero-shot out-of-distribution testing, with errors increasing by approximately a factor of 2 at most, except for Allen-Cahn.CNO still significantly outperforms the baselines in all other cases; FNO generalizes best on Allen-Cahn.
  • Resolution Invariance: For Navier-Stokes, CNO’s Fourier spectrum decays more accurately than FNO’s, while FNO exhibits resolution-dependent errors of up to 25% on lower resolutions and 10% above training resolution 64^2.The observed spectral differences are discussed in relation to incorrect frequency content and aliasing errors.
  • Efficiency: CNO achieves smaller validation errors than FNO at equal model size and training time, with a best-performing model that is almost twice as fast to train.The comparison uses model size and per-epoch training time on the Navier-Stokes benchmark.
  • Scaling laws: CNO reaches 1% error with approximately 14.3K samples, compared with 60.1K for FNO and 133.2K for GT, while its convergence rate is r = 0.37.The corresponding rates are r = 0.28 for FNO and r = 0.27 for GT; a smaller CNO model scales worse with data.

5 Discussion.

The paper frames CNO as a structure-preserving operator-learning architecture, supports its approximation capabilities theoretically, and identifies extensions beyond its current two-dimensional Cartesian setting.

  • Discussion: CNO enforces continuous-discrete equivalence so it learns underlying operators rather than merely discrete representations.The construction modifies convolution, upsampling, downsampling, and nonlinear activations to obtain a representation equivalent neural operator.
  • Discussion: The paper proves that CNO can approximate a large class of operators arising in PDEs to arbitrary accuracy.The approximation is obtained by discarding sufficiently high-frequency components through Fourier projection and selecting an appropriate cutoff.
  • Discussion: Experiments use representative PDE benchmarks with diverse, potentially multi-scale solutions and report that CNO significantly outperforms baselines on most benchmarks and out-of-distribution tasks.The out-of-distribution evaluations assess zero-shot generalization to unseen data.
  • Limitations and Future Work: The reported framework currently targets operators on two-dimensional Cartesian domains; three-dimensional extensions are conceptually straightforward but computationally demanding.Non-Cartesian domains require transformation maps, while time-dependent PDE trajectories and operators beyond forward PDE solution maps remain future directions.
  • Discussion: The continuous-discrete construction is defined through bandlimited function spaces, where sampling at or above the Nyquist rate permits unique and stable recovery.Below the Nyquist rate, the aliasing error function is nonzero, motivating the equivalence requirement.

A.5 Proof of Proposition 2.1 of Main Text

CNO layers are built from convolution, nonlinear activation, and projection operators whose discretizations preserve continuous-discrete correspondence, establishing CNO as a representation equivalent neural operator.

  • A.5 Proof of Proposition 2.1: Each CNO layer consists of a convolution operator, a nonlinear activation operator, and a projection operator.The activation depends on a chosen function σ, while the three mappings operate between bandlimited function spaces.
  • A.5 Proof of Proposition 2.1: The activation layer is defined using projection and embedding between bandlimited spaces so its discretization preserves continuous-discrete equivalence.The corresponding projections and embeddings are discretized through the same equivalence construction.
  • A.5 Proof of Proposition 2.1: The resulting constituent correspondence establishes CNO as a representation equivalent neural operator and proves Proposition 2.1.The proof also notes that the activation layer may be followed by an additional projection operator.

B Proof of Theorem 3.1 of Main Text

The universality proof constructs a continuous approximation on bandlimited function spaces, realizes it with a convolutional network, and controls the approximation error through resolution and frequency choices.

  • Setting and theorem: The theorem assumes a stable continuous PDE solution operator on Sobolev spaces, with regularity and non-polynomial activation conditions.The setting allows coordinate dependence through coefficient and source functions and requires the solution map to satisfy a continuity bound.
  • Step 1: construction of G: The proof first constructs a continuous operator G by encoding grid samples together with Fourier features and approximating the target on bandlimited representations.Trigonometric interpolation and the sampling theorem establish correspondence between discrete samples and bandlimited functions.
  • Setting and theorem: For any ε > 0, the theorem provides a CNO approximating the target operator on inputs bounded in the specified Sobolev norm.The result introduces integers K and N controlling the construction and approximation.
  • Error control and correspondence: The approximation error can be made arbitrarily small by choosing ε sufficiently small and M, N sufficiently large in the stated order.The proof identifies the exact CNO correspondence through interpolation and projection on the fine grid.
  • Step 2: G corresponds to a CNO: Translation equivariance enables a shallow CNN representation, which is lifted to a finer grid using stride N/M and sparse convolution kernels.The lifted network produces approximations across translated subgrids of the fine grid.
  • Proof limitation: The construction crucially requires choosing M independently of N; setting N = M may prevent the error bound from converging to zero.The alternative proof also yields sparse kernels because the convolution stride is N/M.

B.1 Auxiliary results

This section states auxiliary approximation results and introduces baseline neural-network architectures used for comparison. It also describes how sampled functions are encoded and processed by these baselines.

  • Universal approximation: Theorem B.6 states that finite neural-network sums are dense in C(R^d) when the activation is not an algebraic polynomial.The theorem assumes a locally essentially bounded activation whose discontinuity set has closure of zero Lebesgue measure.
  • Interpolation equivalence: Trigonometric polynomial interpolation is shown to be equivalent to interpolation using a periodic sinc filter.The proof is given first in one dimension for odd N, with even and multidimensional cases following identically and by tensorisation.
  • Baseline architectures: A feedforward neural network maps inputs through alternating affine-linear layers and scalar nonlinear activations.The network is represented as a composition of L layers, with tunable weights and a scalar activation function.
  • Baseline architectures: The residual-network baseline encodes a sampled function on an s × s Cartesian grid and outputs the operator result on the same grid.Most experiments use s = 64, while compressible Euler uses s = 128; Leaky ReLU activations and residual blocks of depth k = 2 are used.
  • Baseline architectures: The ResNet baseline uses convolutional blocks with skip connections, downsampling, residual processing, and transposed-convolution upsampling.Its initial block uses a 7 × 7 convolution, while later downsampling blocks use 3 × 3 convolutions with stride 2.
  • Baseline architectures: The UNet baseline follows the proposed architecture while varying the first layer’s output channels through cross-validation.Subsequent channel counts preserve the original architecture’s progressive increases and decreases.

C.1.4 Convolutional Neural Operator

This section explains CNO filter design, architecture, and parameter choices. CNO uses band-limited operations and filtered activations to approximate continuous operators while controlling computational cost.

  • Filter design: CNO uses windowed-sinc filters because ideal sinc filters have infinite impulse responses and can produce Gibbs ringing near high-gradient points.Windowing gives finite impulse response filters with weakened ringing effects.
  • Filter design: The filters use cutoff frequency wc = s/2.0001, half-width wh = ch · s, and a prescribed compact support of Ntap taps.The implementation permits manual control of cutoff frequency and transition-band half-width.
  • Activation and architecture: Pointwise nonlinear activations can generate arbitrarily high frequencies, so CNO upsamples before activation to keep outputs within desired band limits.This modulation is intended to preserve the continuous-discrete equivalence and reduce aliasing errors.
  • Parameter choices: Experiments fix Nσ = 2 and ch = 0.8 because larger ch values show no significant test-error difference and Nσ has no strong error correlation.The selected Nσ is kept as low as possible, while ch is set to 0.8.
  • Parameter choices: Increasing Ntap from 12 to 20 improves test error by approximately 1.5% but raises per-epoch training time from 4.37s to 5.26s.The training-time increase exceeds 20%, motivating the fixed choice Ntap = 12.
  • Caveat: Exact activation equivariance is only approximate in practice, although sufficiently high upsampling appears to make the approximation adequate empirically.Polynomial or rational activations can realize exact equivariance, but this may have limited practical utility.

C.2 Training Details

The training protocol uses optimizer, loss, scheduling, early-stopping, and random-search procedures shared across the benchmark comparisons. Robustness is additionally assessed across ten random initializations.

  • Training protocol: Most models use ADAM for 1000 epochs, minimize relative L1 loss, and train with a step learning-rate scheduler.FFNN, UNet, and DeepONet use batches of 10, while FNO and CNO use batches of 32.
  • Training protocol: Galerkin Transformer models use ADAM with weighted L2 loss and a OneCycleLR schedule.The schedule uses max_lr = 5 · 10^-4, div_fac = 10^4, and pct_start = 0.3.
  • Model selection: The best validation checkpoint is saved, and training stops when validation error fails to improve for 50 epochs.At every epoch, relative L1 error is computed on the validation set.
  • Model selection: Random search selects the configuration with the lowest validation relative L1 error for each model and benchmark.The search evaluates 24–48 configurations for CNO and 36–72 for FNO, alongside the stated ranges for other models.
  • Initialization robustness: CNO is reported as robust to random initialization, with a very low standard-deviation-to-mean ratio across all RPB benchmarks.The comparison retrains CNO, FNO, and UNet ten times per experiment.

C.3.1 Poisson Equation

The Poisson benchmark tests operator learning on multiscale solutions, including higher-frequency out-of-distribution inputs. CNO is reported to outperform UNet and FNO and to capture the solution spectrum more accurately.

  • Problem setup: The Poisson experiment approximates the operator G† mapping source terms f to exact solutions u, with training samples generated using K = 16 and r = 0.5.Testing uses 256 samples for each distribution, sampled at 64 × 64 resolution; training uses 1024 samples.
  • Problem setup: Out-of-distribution testing increases the mode count to K = 20 to evaluate generalization to frequencies higher than those seen during training.The experiment explicitly targets inputs with higher frequencies than the training distribution.
  • Model comparison: CNO is reported as the best-performing model on the Poisson benchmark, with UNet a distant second and FNO errors more than an order of magnitude higher than CNO.The comparison covers both in-distribution and out-of-distribution testing.
  • Spectral behavior: CNO captures the complete multiscale solution spectrum with high accuracy, whereas FNO and UNet resolve the underlying spectrum less successfully.The ground-truth solution contains multiple scales corresponding to a range of frequencies.
  • Additional benchmark: The wave-equation figures also report CNO as the best-performing model in both in-distribution and out-of-distribution testing.Figure 8 presents empirical error distributions for UNet, FNO, and CNO.

C.3.3 Transport Equation

The transport and Allen-Cahn experiments evaluate CNO, FNO, UNet, and additional baselines on in-distribution and out-of-distribution PDE solution-operator tasks. Results vary by problem: CNO is advantageous for smooth transport, while FNO or UNet lead on other listed experiments.

  • Transport Equation: Transport experiments map initial conditions f to the solution u at T = 1 using fixed velocity v = (0.2, −0.2).Both smooth and discontinuous transport use 512 training, 256 validation, and 256 in-distribution and out-of-distribution testing samples.
  • Smooth Transport: Smooth Transport uses normalized radially symmetric Gaussian inputs, with out-of-distribution centers shifted from (0.2, 0.4)2 to (0.4, 0.6)2.Data are generated at 64×64 resolution.
  • Smooth Transport: CNO is slightly superior to UNet and FNO in-distribution and has a significant advantage over UNet out-of-distribution on Smooth Transport.FNO, DeepONet, and FFNN generalize poorly out-of-distribution in the reported comparisons.
  • Discontinuous Transport: Discontinuous Transport uses indicator functions of disks, and sampling discontinuous data introduces unavoidable aliasing because the data have infinite spectral content.Samples are generated at 128 × 128 and downsampled in the frequency domain to 64 × 64.
  • Discontinuous Transport: On the reported Discontinuous Transport comparison, UNet performs as well as CNO out-of-distribution, while FNO, DeepONet, FFNN, and GT generalize poorly.The conclusion is tied to failure to be translation equivariant for the latter models.
  • Allen-Cahn Equation: For Allen-Cahn, FNO is marginally superior to CNO and UNet in-distribution, whereas UNet is the best model out-of-distribution.The operator maps initial conditions to u(·, T = 0.0002) and data are generated by finite differences at 64 × 64 resolution.

C.3.8 On the Choice of the RPB benchmarks.

The benchmark suite is designed to cover diverse PDE types while remaining reproducible and computationally demanding enough to motivate fast surrogates. The Navier-Stokes analyses contrast an easy benchmark with a multiscale problem and connect CNO's resolution stability to continuous-discrete equivalence.

  • Benchmark Design: The RPB benchmarks span linear elliptic, linear hyperbolic, nonlinear parabolic, nonlinear hyperbolic, and non-local advection-diffusion PDEs.The suite includes Poisson, wave, transport, Allen-Cahn, compressible Euler, and incompressible Navier-Stokes problems.
  • Benchmark Design: Benchmarks are selected for reproducibility and sufficient computational complexity so traditional numerical methods are expensive enough to justify machine-learning surrogates.This excludes one-dimensional problems where numerical methods are very fast.
  • Navier-Stokes Comparisons: In the older Navier-Stokes experiment, at most 20 Fourier modes per direction approximate the solution, while FNO and CNO achieve 1.15% and 0.96% test errors.Both learned models have inference times of order 10−4 seconds on a NVIDIA qaudro t2000 GPU.
  • Navier-Stokes Comparisons: The thin shear layer requires almost 100 Fourier modes for 2% L1-error, making it more numerically challenging than the older Navier-Stokes benchmark.Its multiscale structure motivates including it as a benchmark.
  • Navier-Stokes Comparisons: CNO inference takes 10−4 seconds for approximately 3% error on the thin shear layer, yielding three orders of magnitude or more speedup over the numerical solver.The comparison uses a numerical runtime of 10−1 seconds for a 100 × 100 grid.
  • Resolution Stability: CNO is approximately invariant to resolution, whereas UNet and FNO show notable fluctuations when the resolution changes.The reported behavior is presented as evidence that CNO respects continuous-discrete equivalence.

C.5 Ablation Studies.

The ablation studies test CNO's modified sampling and activation operations and its encoder-decoder ResNets. The original CNO generally performs best, while removing filters or ResNets can accelerate training but reduce performance or generalization.

  • Ablation Design: The ablations examine modified upsampling, downsampling, and activation operations, as well as ResNets connecting the Operator U-Net encoder and decoder.Both performance and training time are evaluated across the benchmark suite.
  • Ablation Design: CNO w/o Filters replaces filtered operations with nearest-neighbor upsampling, average pooling, and pointwise activation.This produces a structure resembling a regular U-Net with additional encoder-decoder ResNets.
  • Performance: The original CNO outperforms the ablation models in 12 of 16 tests, while the remaining four testing errors are close to those of the best-performing models.The first ablation is inferior in almost all tests.
  • Performance: The first ablation's regular CNN operations introduce aliasing errors and resolution dependence similar to UNet.It does not adhere to the continuous-discrete equivalence property.
  • Performance: The second ablation is slightly better in one out-of-distribution test, but the original CNO is significantly better in many cases.The performance gap ranges from less than 10% on Compressible Euler and Darcy Flow to almost 300% on Poisson.
  • Training Time: Removing filters accelerates training by approximately 1.75 times on Poisson and 1.35 times on Darcy Flow, while other benchmarks show smaller gains or similar time.Removing middle ResNets gives approximately 1.75 times faster Poisson training and 1.25 times faster Navier-Stokes and Darcy training.

D Depiction of the Datasets.

The dataset depiction section illustrates input and output samples for the PDE forward problems and identifies the reported ablation-error table. The figures cover Poisson, wave, transport, Allen-Cahn, Navier-Stokes, Darcy flow, and compressible Euler.

  • Ablation Results: Table 13 reports relative median L1 test errors for CNO and two ablation models under in-distribution and out-of-distribution testing.The table summarizes the ablation comparison across the two testing settings.
  • Dataset Illustrations: The dataset illustrations show input samples on the left and output samples on the right for the Poisson and wave equations.These are presented in Figures 28 and 29.
  • Dataset Illustrations: Figures 30 and 31 illustrate input and output samples for continuous and discontinuous transport.The discontinuous transport figure is labeled with the misspelled title “discontinous transport problem.”
  • Dataset Illustrations: Figure 32 depicts input and output samples for the Allen-Cahn equation, while Figure 33 depicts samples for Navier-Stokes equations.The Navier-Stokes illustration is listed as Figure 33.
  • Dataset Illustrations: Figure 34 illustrates input and output samples for Darcy flow, and Figure 35 does so for the compressible Euler equations.Both figures use the same left-input, right-output layout.
Loading 2302.01178v3…