Source-linked AI summary
Neural Operator: Graph Kernel Network for Partial Differential Equations
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar
TL;DR
The paper addresses the challenge of learning mappings between infinite-dimensional spaces while sharing parameters across finite-dimensional discretizations. It introduces graph kernel networks that approximate these mappings with integral operators evaluated by graph message passing. Experiments show discretization generalization and competitive performance, while runtime and storage become costly on large grids.
Problem
Existing neural networks are primarily designed for finite-dimensional mappings, while PDE applications require learning mappings between function spaces across different discretizations.
Method
The paper introduces Neural Operators instantiated as graph kernel networks, using integral operators, Nyström approximations, and message passing on spatial graphs.
Results
The method generalizes across meshes, resolutions, and geometries, achieves competitive performance, and can learn from few samples or sparse point observations.
Takeaways & Limitations
A single trained network can serve different finite-dimensional approximations and query new locations without interpolation.
Takeaways & Limitations
Runtime and storage scale as O(K^2) with the number of graph edges, making full-grid training and evaluation costly for large grids.
Abstract
from arXiv · showhide
The classical development of neural networks has been primarily for mappings between a finite-dimensional Euclidean space and a set of classes, or between two finite-dimensional Euclidean spaces. The purpose of this work is to generalize neural networks so that they can learn mappings between infinite-dimensional spaces (operators). The key innovation in our work is that a single set of network parameters, within a carefully designed network architecture, may be used to describe mappings between infinite-dimensional spaces and between different finite-dimensional approximations of those spaces. We formulate approximation of the infinite-dimensional mapping by composing nonlinear activation functions and a class of integral operators. The kernel integration is computed by message passing on graph networks. This approach has substantial practical consequences which we will illustrate in the context of mappings between input data to partial differential equations (PDEs) and their solutions. In this context, such learned networks can generalize among different approximation methods for the PDE (such as finite difference or finite element methods) and among approximations corresponding to different underlying levels of resolution and discretization. Experiments confirm that the proposed graph kernel network does have the desired properties and show competitive performance compared to the state of the art solvers.
1 INTRODUCTION
The paper introduces Neural Operators for learning mappings between function spaces, targeting PDE solution operators whose behavior can span discretizations and resolutions. Graph kernel networks implement this idea with mesh-free message passing and demonstrate competitive approximation, cross-grid generalization, few-shot learning, and semi-supervised learning.
- 1 INTRODUCTION: A single trained parameter set can remain invariant across different discretizations, resolutions, and grid representations.The Nyström formulation connects the function-space network to finite-dimensional approximations and arbitrary grids.
- 1 INTRODUCTION: Unlike mesh-dependent convolutional approaches, the method can transfer solutions between meshes and requires only a forward pass for a new coefficient function after training.The underlying PDE may be treated as a black-box mapping learned from experimental data or costly simulations.
- 1 INTRODUCTION: Graph kernel networks approximate PDE solution operators through integral operators evaluated by message passing on spatial graphs.The approach directly constructs graphs on output-function locations and uses Nyström-style connections between grids.
- 1 INTRODUCTION: Neural Operators generalize neural networks to mappings between infinite-dimensional function spaces rather than only finite-dimensional Euclidean spaces.The paper focuses on real-valued functions defined on subsets of R^d and applies the framework to PDE inputs and solutions.
- 1 INTRODUCTION: Experiments report competitive approximation accuracy, generalization from few samples, and semi-supervised learning from observations at only a few points.The method is illustrated on elliptic PDEs and can incorporate data on unstructured grids.
2 PROBLEM SETTING
The paper formulates supervised learning of typically nonlinear mappings between infinite-dimensional function spaces from finite observations, focusing on PDE solution operators. It seeks discretization-independent approximations while acknowledging that the current implementation relies on pointwise function evaluations.
- Problem formulation: The central problem is to approximate a typically nonlinear map F† between separable Banach spaces using observed input-output function pairs.The observations may contain noise, and the approximation is parameterized by a finite-dimensional space Θ.
- Discretization independence: Unlike standard finite-dimensional neural architectures, the intended model uses common parameters across finite-dimensional approximations and different discretizations.This targets consistent approximation across resolutions and grids rather than an architecture tied to one value of K.
- PDE application: The PDE application learns mappings from coefficient functions to solutions, defining a solution operator for a second-order elliptic equation.For fixed f and suitable coefficient a, the equation has a unique weak solution u, although the solution operator is nonlinear even when the PDE is linear.
- Discrete observations: Finite-dimensional training data consist of pointwise evaluations of input and output functions on a K-point discretization of the domain.The proposed network can then produce u(x) for any x in the physical domain given a new input function.
- Scope boundary: The current application assumes pointwise function evaluations, while extensions to truncated basis coefficients are described as possible but left for future work.Representation invariance would then concern the size of the coefficient set.
3 GRAPH KERNEL NETWORK
The graph kernel network approximates PDE solution operators with integral kernels implemented through message passing on spatial graphs. Its shared kernel parameters support mesh-independent discretization, while experiments show accurate kernel learning and strong data efficiency on Poisson problems.
- 3 GRAPH KERNEL NETWORK: The architecture represents PDE solutions through iterative neural updates combining pointwise nonlinearities with an integral operator parameterized by a learned kernel.The kernel network maps edge features to matrix-valued kernels, while W and the kernel parameters are learned from data.
- 3 GRAPH KERNEL NETWORK: The discretized kernel shares one set of parameters across all K × K blocks, making the network independent of the discretization used.This is the mechanism connecting the continuum formulation to different finite-dimensional grids.
- Experiments: The 1D proof-of-concept compares learned and true Green’s-function kernels, while the broader method also permits mappings from f or boundary data to u.The f-to-u example uses a one-step linearized specialization with known analytic structure.
- Experiments: In the 2D Poisson experiment, the graph kernel network reached its minimum error with approximately 5 training samples and had lower test error than a dense network trained with 100 samples.The reported explanation is that the graph kernel network learns the true Green’s function, as also demonstrated in the 1D case.
- Message passing: Kernel integration is approximated by Monte Carlo message passing, with local neighborhoods chosen both for computational efficiency and to exploit decay of the Green’s function.A more informative measure could incorporate known kernel structure, such as a Gaussian density.
- Graph construction: Graph construction connects each spatial node to neighbors within a physical-radius ball and uses coordinates and coefficient values as edge features.The radius is fixed in physical space, so neighborhoods grow under refinement, preserving mesh independence while reducing computation relative to a fully connected graph.
4 EXPERIMENTS
Experiments evaluate graph kernel networks on PDE solution mappings across resolutions, sampling regimes, training-set sizes, graph sparsity, and kernel-network architectures. The results show resolution generalization, competitive accuracy with limited data, and trade-offs involving kernel support, sampling, and model design.
- Experimental setup: The experiments use coefficient-to-solution mappings on D = [0, 1]^2, with piecewise-constant random coefficients generated from a Gaussian-process construction and solutions computed on a 241 × 241 finite-difference grid.Different resolutions are obtained by downsampling the generated dataset.
- Generalization of resolutions on full grids: Test errors remain on the same scale across full-grid resolutions, showing that graph kernel networks can train at one resolution and generalize to another.Performance is best when training and testing use the same grid; larger test grids can be easier than smaller ones because smaller grids truncate the learned kernel support.
- Expressiveness and overfitting: The kernel network achieves reasonable accuracy with N = 10 training pairs and competitive accuracy with N = 100, while its nonzero training error indicates expressiveness without severe overfitting.The experiment compares N = 10, 100, and 1000 training pairs on a 31 × 31 grid.
- Generalization of resolutions on sampled grids: In sampled-grid experiments, test errors are similar across resolutions, while finer training grids and matching training and test resolutions generally improve performance.Finer grids provide more edges because the kernel support is larger.
- Sampling repetitions: Increasing the number of Nyström sampling repetitions improves accuracy, although l = 2 already provides reasonable performance and larger l is especially helpful with limited training data.The comparison relates the total sampled training pairs Nl to approximation error.
- Graph sparsity and node sampling: Kernel-support truncation hurts more than Nyström approximation at a fixed edge budget, so larger radius with fewer sampled nodes is preferable.Increasing the numbers of sampled nodes generally improves performance, but matching training and test node counts is typically best.
- Inner kernel network architecture: A three-layer inner kernel network with width 256 is reported as a good configuration, whereas a very wide shallow network can become unstable under the 1e-4 learning rate.The compared architectures vary in width and depth; the width-4096, depth-2 configuration deteriorated after initially reaching lower error.
- Comparison with benchmarks: GKN shows competitive performance against the benchmark methods while generalizing to different mesh geometries, unlike mesh-dependent fully convolutional networks.The comparison identifies FCN as strong on small grids but increasingly erroneous on larger grids.
5 DISCUSSION AND FUTURE WORK
The graph kernel network is designed to generalize across meshes, geometries, resolutions, and irregular grids, while its quadratic edge complexity limits scalability on large grids.
- 5 DISCUSSION AND FUTURE WORK: The network trains and generalizes across different meshes because it learns mappings between infinite-dimensional function spaces.The same approach can share approximations across different discretization levels.
- 5 DISCUSSION AND FUTURE WORK: Different graphs can be constructed and trained together for problems involving different geometries, such as flow around many airfoils.
- 5 DISCUSSION AND FUTURE WORK: Irregular-grid data can be queried at new locations without interpolation, avoiding subsequent interpolation error.
- 5 DISCUSSION AND FUTURE WORK: Runtime and storage scale as E = O(K^2), whereas PCA+NN and RBM scale as O(K), making full-grid training and evaluation costly.Subsampling reduces cost but loses information and causes errors that make the method less competitive than PCA+NN and RBM.
- 5 DISCUSSION AND FUTURE WORK: Future work proposes multigrid and fast multipole methods to reduce complexity, with multigraphs connecting nearest neighbors so edges scale as O(K).
A.1 Table of Notations
Table 9 provides the paper’s notation reference.
- A.1 Table of Notations: Table 9 is a table of notations.
A.2 Nystr¨om Approximation
The Nyström approximation analysis constructs empirical and population operators from sampled kernel features and establishes a Hilbert-Schmidt error bound.
- A.2 Nystr¨om Approximation: The proof samples independent points y_j from ν and defines kernel-based operators T^(j) and T_y whose expectation equals T.
- A.2 Nystr¨om Approximation: The empirical operator T_m is finite rank, while the population operator T is shown to be Hilbert-Schmidt.
- A.2 Nystr¨om Approximation: Jensen’s inequality is applied to control the approximation error between T_m and T.
- A.2 Nystr¨om Approximation: C√m bounds the expected Hilbert-Schmidt error E∥T_m − T∥_HS.
A.3 Figures of Table 8
Figures 4–6 evaluate graph kernel network predictions across increasing resolutions, while Figure 7 visualizes the learned and true Poisson kernel.
- A.3 Figures of Table 8: Figures 4, 5, and 6 show performance on the first five testing examples at s = 141, s = 211, and s = 421, respectively.
- A.3 Figures of Table 8: Most error occurs around the singularity of coefficient a, where the two colors cross.
- A.3 Figures of Table 8: Figure 7 presents contour plots of the learned and true kernel for Poisson’s equation on a unit disk.The plotted Green’s function is G(0, 0; e^ρ, e^θ).
A.4 Approximating the 2D Green’s function
The graph kernel network approximates the 2D Poisson Green’s function on a unit disk. It learns the true kernel, with larger error near the singularity at zero.
- The graph kernel network captures the truth Green’s function for the 2D Poisson equation.
- The experiment uses a unit-disk domain because a closed-form Green’s function exists there.
- The network approximates the mapping f(x) 7→u(x) and compares the learned kernel G with the truth kernel Gt.
- The learned Green’s function matches the true function, although approximation error increases near the singularity at zero.