PaperlayerView exact revision on arXiv

Source-linked AI summary

FTD-GNO: Memory-Efficient Graph Neural Operators through Functional Tensor Decomposition of the Kernel

Xiaomin Zhang, Boyue Wang, Junbin Gao, Yongli Hu anbd Baocai Yin

arXiv:2610.04212v1cs.LGcs.AI

TL;DR

Standard GNOs incur substantial memory and computational costs from monolithic kernel parameterization and edge-wise tensor construction. FTD-GNO factorizes the kernel into low-dimensional functions and algebraically reconstructs the integral operator. The paper reports competitive PDE-benchmark performance and lower peak memory than corresponding unfactorized baselines.

  • Problem

    Standard GNOs construct and store complete edge-wise kernel tensors and use large networks, creating substantial memory and parameter costs as dimensionality or grid density increases.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    FTD-GNO decomposes the continuous kernel into low-dimensional factor networks and uses tensor-decomposition associativity to reconstruct the integral operator.

    Source: §1 · p. 2, §3.2.1 · p. 5

  • Results

    FTD-GNO reports competitive performance against GNOs on PDE benchmarks and lower peak memory than corresponding unfactorized baselines across regular-grid and irregular-mesh tasks.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The factorization reduces memory growth to linear scale while retaining continuous spatial expressive capacity and addressing parameter growth with dimensionality.

    Source: §1 · p. 2

    The authors identify remaining OOM risks for fully connected graphs or exceptionally large receptive fields.

    Source: F Limitations · p. 19

Abstract

from arXiv · show

Graph Neural Operators (GNOs) provide flexible surrogate models for learning solution operators of partial differential equations (PDEs). However, standard GNOs typically parameterize the integral kernel with a monolithic neural network and evaluate kernel interactions over graph edges, leading to substantial computational and memory overhead at high resolutions or with large neighborhoods. To address these limitations, we propose Functional Tensor Decomposition Graph Neural Operator (FTD-GNO), a memory-efficient GNO framework that decouples the high-dimensional continuous integral kernel into low-dimensional mode-wise functions. By instantiating the kernel with classical tensor decomposition formats, including CP, Tensor-Train, and Tucker decompositions, FTD-GNO enables algebraic reconstruction of the integral operator without explicitly materializing full edge-wise kernel tensors. This factorized formulation reduces the memory footprint of kernel evaluation and aggregation while retaining the continuous operator-learning structure of GNOs. Theoretical complexity analysis shows that FTD-GNO substantially lowers parameter and activation-memory costs associated with high-dimensional kernel construction. Experiments show lower peak memory than the corresponding unfactorized graph-integral baselines, with shorter recorded training times. Fourier-graph experiments further demonstrate that FTD can improve the efficiency of a graph-integral layer within a hybrid operator and has good scalability.

1 Introduction

Standard GNOs face memory and parameter growth as kernel inputs become higher-dimensional or grids denser. FTD-GNO decouples the kernel into low-dimensional functions and reconstructs the integral operator algebraically.

1 Introduction

  • Standard GNOs store complete edge-wise kernel tensors, driving exponential memory growth with dimensionality or grid density and forcing larger networks with more parameters.These costs can trigger out-of-memory errors and constrain network depth.Source: §1 · p. 2
  • FTD-GNO decomposes the high-dimensional kernel into low-dimensional, dimension-specific subnetworks and uses tensor-decomposition associativity to reconstruct the spatial integral operator.Source-dependent factors are aggregated before combination with query-dependent factors.Source: §1 · p. 2
  • The decomposition avoids explicit full-size tensors, reducing memory complexity from exponential to linear growth while addressing dimensional parameter explosion without sacrificing continuous spatial expressive capacity.The framework considers CP, TT, and Tucker decomposition formats.Source: §1 · p. 2
  • The paper reports competitive performance against GNOs in PDE benchmarks as empirical validation of its theoretical analysis.Source: §1 · p. 3

2 Related works

Related work covers neural-operator architectures, functional tensor decomposition in continuous-domain learning, and tensor-based dimensionality reduction for operator learning.

Neural Operators

  • Neural-operator approaches include DeepONet’s branch–trunk architecture, Fourier-domain convolution in FNO [18], graph-based integration in GNO [11], and attention-based GK-Transformer [19] and GNOT [20].Source: §2 · p. 3

Functional Tensor Decomposition in Deep Learning

  • Functional tensor decomposition has been applied to implicit neural representations, where F-INR [23] separates coordinate-wise subnetworks, and to PINNs [24] to separate spatial and temporal variables.Source: §2 · p. 3

Tensor Decomposition for Operator Learning

  • Operator-learning compression includes frequency-domain factorization in F-FNO [25] and D-FNO, low-rank kernel constraints in Tucker-FNO and MG-TFNO [28], and spatial-basis or optimizer methods in POD-DeepONet [29] and TensorGRaD [30].Source: §2 · p. 3

3 Proposed Methods

The methods section defines the GNO integral-operator setting, then factorizes its kernel into mode-wise functions and reconstructs the operator within the standard architecture. Its complexity analysis describes linear scaling for key costs.

3.1 The Setting of GNOs

  • Neural operators map input functions to output functions; standard GNO layers update node features through kernel integration, approximated on graphs using domain truncation and Nyström subsampling.A standard NO architecture comprises lifting, iterative operator layers, and projection.Source: §3.1 · pp. 3–4
  • Standard GNO kernel parameterization uses a monolithic network and stores full high-dimensional kernel tensors on edges, causing exponential parameter and memory growth.The paper introduces FTD-GNO to address this challenge by representing the kernel as low-dimensional, low-rank component functions.Source: §3.1 · p. 4, §3.2 · p. 4

3.2 Functional Tensor Decomposition for GNO

  • FTD-GNO represents the kernel over d input-variable blocks with separate factor networks, replacing a dense d-dimensional tensor with a decomposition-based composition.The standard kernel input comprises query point, neighboring point, and their associated field values.Source: §3.2.1 · pp. 4–5

3.2.1 Functional Tensor Decomposition of the kernel

  • The framework instantiates the factorization with CP, TT, or Tucker formats, using rank-one outer products, linked core tensors, or a core tensor with factor matrices, respectively.Source: §3.2.1 · p. 5

3.2.2 Model Architecture

  • FTD-GNO retains the lifting–operator-layer–projection architecture, parameterizes the kernel with the factor networks and core tensor, and uses domain truncation and Nyström approximation.Figure 2 presents the full architecture and an example of four-variable kernel parameterization.Source: §3.2.2 · p. 5, Figure 2 · p. 6

3.2.3 Complexity Analysis

  • The complexity analysis reports linear-scale input-memory occupancy and single-kernel evaluation cost for FTD-GNO, alongside reduced parameter scale compared with standard GNO.Table 1 summarizes the complexity comparison; standard GNO’s single-kernel evaluation and parameter count are O(W^2L).Source: §3.2.3 · pp. 6–7, Table 1 · p. 6

4 Experiments

Across regular-grid and irregular-mesh benchmarks, FTD reduces peak memory relative to unfactorized graph-integral baselines. The experiments also examine memory scaling and zero-shot super-resolution.

4.1 Main Comparisons

  • With closely matched parameter counts, LR-GNO has substantially different memory costs from FTD-GNO, and its slightly lower error shows the models do not behave identically.The comparison indicates that parameter reduction alone does not explain FTD-GNO’s memory advantage.Source: §4.1 · p. 8
  • Across regular-grid and irregular-mesh benchmarks, FTD reduces peak memory versus corresponding unfactorized graph-integral baselines in both regular-grid architecture groups.At higher NS resolution, factorized variants remain trainable when FNOGNO encounters OOM.Source: §4.1 · pp. 7–8
  • Regular-grid experiments report training speedups and parameter reductions, while graph-only configurations have relatively high NS errors and Fourier layers improve predictive performance.FNO provides more accurate predictions than graph-only configurations in NS.Source: §4.1 · p. 8
  • On irregular meshes, all three FTD-GNO variants reduce peak memory and training time versus GNO; additional 300-epoch runs improve error beyond 100-epoch GNO while retaining lower memory and runtime.The authors report that reduced training cost can support additional optimization.Source: §4.1 · p. 8

4.2 Memory Footprint

  • 64 × 64: GNO reaches OOM as graph density rises, while all FTD-GNO variants show stable linear memory growth proportional to node count.The comparisons vary resolution and neighborhood radius.Source: §4.2 · p. 8, Table 4 · p. 8, Table 5 · p. 8

4.3 Zero-Shot Super-Resolution

  • After training at 16 × 16, all models’ relative test errors at 32 × 32 remain within the same order of magnitude, suggesting zero-shot super-resolution generalization.The models are evaluated at both the training resolution and the unseen higher resolution.Source: §4.3 · p. 9

5 Conclusion

The conclusion presents FTD-GNO as a kernel-reconstruction approach for reducing memory in graph-integral models. The accompanying background describes parametric PDEs and standard GNO discretization.

5 Conclusion

  • FTD-GNO reconstructs the integral kernel through functional tensor decomposition, and the paper reports lower peak memory across regular-grid and irregular-mesh tasks and in a Fourier–graph hybrid.The conclusion describes memory requirements as scaling linearly with node size.Source: §5 · p. 9

A General Framework of Parametric PDEs

  • Parametric PDEs are parameter-driven equations; in Darcy flow, the input field represents permeability and the target solution represents fluid pressure.For fixed source and boundary conditions, each parameter instance is associated with a unique solution.Source: A General Framework of Parametric PDEs · p. 11

B Extended Background on Standard GNOs

  • Domain truncation connects each node to a radius-based neighborhood, reducing computation while remaining invariant to mesh refinement.The neighborhood is defined by the spatial ball intersected with the domain.Source: B Extended Background on Standard GNOs · p. 11
  • O(J'²), rather than O(J²): Nyström subsamples nodes, and standard GNO treats the kernel integral as message aggregation.The sampled node count J' is much smaller than the original discretization size J.Source: B Extended Background on Standard GNOs · pp. 11–12

C Mathematical Foundations of FTD-GNO: Decomposition

The section describes CP, TT, and Tucker functional decompositions of the integral kernel, then states an increasing-capacity approximation result for continuous kernels.

C Mathematical Foundations of FTD-GNO: Decomposition

  • CP, TT, and Tucker express the continuous kernel through low-rank factor functions; Tucker additionally uses a learnable core tensor to capture coupling across rank dimensions.The factor functions are parameterized by neural networks.Source: C Mathematical Foundations of FTD-GNO:… · p. 12
  • On compact product domains, continuous kernels admit finite-rank expansions in products of continuous block functions, with approximation error arbitrarily small as rank increases.The proposition also extends component-wise to finite-dimensional channel-wise kernels.Source: C Mathematical Foundations of FTD-GNO:… · p. 12
  • If each factor-network family is uniformly dense in continuous functions on its block, sufficiently capable networks retain the approximation result; TT and Tucker inherit it from CP representations.TT can realize a CP sum using diagonal intermediate cores, and Tucker using a superdiagonal core.Source: C Mathematical Foundations of FTD-GNO:… · p. 12

D Detailed Complexity and Memory Analysis

The appendix analyzes how CP, TT, and Tucker factorize kernel computation and parameters, then describes memory savings from decoupled input storage and algebraic reconstruction of neighborhood aggregation.

D.1 Computational and Parameter Complexity

  • CP-GNO uses d rank-R factor networks, with parameter count O(dw2lR).The supplied derivation also states outer-product composition costs O(dR).Source: §D.1 · p. 13
  • TT-GNO has single-kernel computation O(d(w2l+R2)) and parameter count O(dw2lR2), assuming intermediate TT-ranks Ri = R.The computation combines factor-network propagation O(dw2l) with tensor contraction O(dR2).Source: §D.1 · p. 13
  • Tucker-GNO has single-kernel computation O(dw2l + Rd) and parameter count O(Rd + dw2lR), with uniform Tucker ranks Ri = R.The core tensor contributes O(Rd) to both contraction computation and parameter count.Source: §D.1 · pp. 13–14

D.2 Memory Occupancy

  • Input construction stores d independent vectors of length I rather than an explicit full-scale tensor, changing the stated memory scaling from exponential to linear.Each branch network processes only the objects in its corresponding dimension.Source: §D.2 · p. 14
  • Algebraic reconstruction turns neighborhood aggregation into local aggregation over y followed by feature combination at x, shifting execution memory from edge-dependent to node-dependent.This avoids constructing and storing high-dimensional concatenated tensors along graph edges; the analysis assumes uniform rank R.Source: §D.2 · p. 14

E.1 Experimental Settings

The experiments compare factorized and baseline graph operators across regular grids and irregular meshes, with controlled data splits and specified architectures, training budgets, and resource-measurement procedures.

E.1 Experimental Settings

  • Darcy flow and Navier–Stokes use regular grids, while elasticity uses sample-dependent irregular meshes with 972 nodes.Darcy flow and Navier–Stokes each use 1,000 training and 200 test samples; elasticity uses 200 training, 50 validation, and 200 test samples.Source: §E.1 · pp. 14–15
  • Baseline graph operators use four layers and 32 hidden channels; GNO uses a three-layer kernel MLP, while FTD-GNO tests CP, TT, and Tucker kernels.The stated ranks are R = 8 for CP and TT and R = 4 for Tucker; FNOGNO comparisons replace graph layers with Fourier layers while retaining the final graph integral.Source: §E.1 · pp. 15–16
  • Training uses AdamW with batch size 32, initial learning rate 1e −3, weight decay 1e −4, and cosine annealing, with epoch budgets varying by experiment.The primary Darcy and resolution-transfer studies use 300 epochs; additional Darcy/Navier–Stokes architectural comparisons use 100 epochs, as do the elasticity reports before additional 300-epoch factorized runs.Source: §E.1 · p. 16
  • Relative l2 error is used for training and evaluation, and peak GPU memory and total training time are measured on a single NVIDIA GeForce RTX 5090 GPU.Total time includes loading, training-time preprocessing, forward/backward computation, neighborhood operations, optimization, and scheduled validation; dataset download and model construction are excluded.Source: §E.1 · p. 16

E.2.1 Detailed analysis in Section 4.1

Across the reported comparisons, factorized graph-integral models reduce peak memory, including in Fourier–graph hybrids and irregular-mesh elasticity; accuracy remains dependent on training budget and architecture.

E.2.1 Detailed analysis in Section 4.1

  • With nearly identical parameter counts, LR-GNO and CP/TT-GNO have substantially different peak-memory costs, so weight-count reduction alone does not explain FTD's memory advantage.LR-GNO retains edge-wise kernel evaluation, whereas FTD reorganizes kernel evaluation and aggregation through reusable node-wise factors; LR-GNO errors are close but slightly better.Source: §E.2.1 · p. 16, Table 9 · p. 16
  • FTD variants reduce peak memory versus corresponding unfactorized baselines across Darcy flow and Navier–Stokes graph-only and Fourier–graph comparisons.At low resolution, FTD-FNOGNO variants retain accuracy close to FNOGNO; the comparison uses the 100-epoch protocol.Source: §E.2.1 · pp. 16–17
  • At higher Navier–Stokes resolution, GNO and FNOGNO encounter OOM while factorized variants remain trainable, but FNO is more accurate with lower memory than the tested hybrids.Graph-only configurations have relatively high errors, and Fourier layers improve graph-based predictive performance without eliminating the gap to FNO.Source: §E.2.1 · p. 17
  • On irregular-mesh elasticity, all three factorization variants reduce peak memory and recorded total training time versus GNO at 100 epochs.At 300 epochs, factorized variants improve beyond the 100-epoch GNO error while retaining lower memory and recorded total runtime.Source: §E.2.1 · p. 17, Table 10 · p. 17

E.2.2 Supplementary experiments

Supplementary studies examine training duration, model capacity, kernel inputs, and memory stability, finding that additional optimization can help while extra capacity does not guarantee better finite-budget accuracy.

E.2.2 Supplementary experiments

  • For Darcy flow, longer training progressively narrows the accuracy gap between CP-GNO and GNO without changing CP-GNO’s architecture; at the longest tested budget, a small error gap remains and recorded total training times become comparable.Source: §E.2.2 · p. 17
  • Increasing width yielded little accuracy improvement while increasing parameters and memory, and the shallowest tested configuration achieved lower error and lower measured resources than deeper alternatives.Rank response was non-monotonic: CP and TT had their lowest tested error at R = 8, while Tucker performed best at R = 4.Source: §E.2.2 · p. 17, Table 12 · p. 18, Table 13 · p. 18
  • Additional optimization can improve a fixed-capacity factorized model, whereas increasing capacity alone does not guarantee better finite-budget performance.The paper recommends selecting rank, width, depth, and training budget jointly using validation performance and available memory and runtime budgets.Source: §E.2.2 · p. 17
  • For full-state inputs, Tucker-GNO memory rose to 861.5 MB, while CP and TT maintained highly compressed footprints with only marginal increases.With identical factor subnetworks and two spatial inputs, CP and rank-R TT represent the same function class and have identical memory footprints.Source: §E.2.2 · p. 18, Table 14 · p. 18
  • Across tested resolutions and radii, FTD-GNO variants show substantial memory reductions versus standard GNO, with standard deviations generally below 1.0 MB.Measurements are reported as mean ± standard deviation across independent runs.Source: §E.2.2 · p. 18, Table 15 · p. 19

F Limitations

FTD-GNO has three stated limitations: Tucker-GNO can face memory growth beyond four input variables, fixed ranks may constrain complex dynamics, and dense neighborhoods may still risk OOM. Future work targets adaptive ranks and coordinate-based spatial decomposition.

F Limitations

  • Tucker-GNO’s core tensor remains susceptible to combinatorial memory growth when inputs exceed four variables.The paper notes that CP and TT avoid this issue, while further Tucker optimization remains open.Source: F Limitations · p. 19
  • A pre-defined decomposition rank may constrain kernel expressivity for highly dynamic processes such as strong turbulence.The paper reports R = 8 is sufficient for most tasks, but identifies complex physical processes as a potential exception.Source: F Limitations · p. 19
  • FTD-GNO may still face OOM risks when the number of neighboring nodes increases drastically.The authors identify adaptive rank allocation and coordinate-based spatial decomposition as future directions, aiming to address risks in fully connected graphs or exceptionally large receptive fields.Source: F Limitations · p. 19
  • Future work will investigate dynamic rank allocation and spatial decomposition based directly on grid coordinates.The stated goal is to adapt to varying physical fields and address OOM risks in fully connected graphs or exceptionally large receptive fields.Source: F Limitations · p. 19
Loading 2610.04212v1…