PaperlayerView exact revision on arXiv

Source-linked AI summary

FAST-DeepONet: Factor-Augmented Branch Representations for High-Dimensional PDE Inputs in the Small-Sample Regime

Jiyong Kwon, Bongseok Kim, Guang Lin

arXiv:2608.15408v1cs.LG

TL;DR

DeepONet branches can be unstable with thousands of correlated PDE sensors but few training operators. FAST-DeepONet combines fixed spectral coordinates with a regularized low-dimensional encoding of the orthogonal residual, lowering error across three benchmarks while using fewer parameters.

  • Problem

    DeepONet must learn from thousands of strongly dependent sensor covariates despite only tens of independent PDE solutions for training.

  • Method

    FAST-DeepONet combines fixed spectral coordinates with a learned low-dimensional encoding of the orthogonal residual and regularizes its normalized effective residual map.

  • Results

    4.7% to 37.0% lower error than a plain branch across three benchmarks, while training three to seven times fewer parameters.

  • Takeaways & Limitations

    On these benchmarks, the gains increase with branch dimension and disappear when the branch is small, while severe sample scarcity remains uncompensated.

  • Takeaways & Limitations

    The evaluation covers three generated PDE families, fixed sensor layouts, and only 40 to 60 training operators.

Abstract

from arXiv · show

Deep operator networks can become statistically unstable when partial differential equation inputs are observed at thousands of strongly correlated sensors but only a small number of operator samples is available. We introduce FAST-DeepONet, a branch representation combining a fixed spectral path with a regularized projection of the orthogonal residual, in which the directional penalty acts on the effective residual map after each of its rows is normalized. On Navier--Stokes flow a plain DeepONet degrades from $0.0394$ to $0.1556$ mean relative $L_2$ error as the branch grows from $129$ to $8193$ coordinates, while FAST-DeepONet stays near $0.04$, so the sensor grid can be refined without a statistical penalty. Across independent test sets for Navier--Stokes flow, Darcy flow, and signed terminal wavefield prediction it lowers mean relative $L_2$ error by $4.7\%$ to $37.0\%$ with three to seven times fewer trainable parameters. A spectral-only branch sharing the same basis separates the two paths: the fixed spectral path carries the improvement on Navier--Stokes and Darcy, while terminal wave prediction requires the residual path together with its directional penalty. FAST-DeepONet targets coordinate-query architectures and trains on solution values alone.

1. Introduction

FAST-DeepONet adapts factor augmentation to DeepONet branches by combining fixed spectral factors with a learned projection of the orthogonal residual. Its effective residual map is row-normalized before directional regularization, reducing branch size while preserving coordinate-query comparisons.

  • Regularization: The effective residual map, rather than raw projection parameters, is regularized after each row is normalized to remove layer-scaling and projected-away-component effects.This parameterization makes the directional penalty depend on the represented residual map instead of adjacent trainable-layer scaling.
  • Method: FAST-DeepONet combines a fixed spectral path, a numerical rank safeguard, and a trainable orthogonal-residual projection into at most 32 branch features.The representation converts 4,096 to 8,193 sensor values into a compact branch, and the first branch layer shrinks accordingly.
  • Method: 3–7 times fewer parameters result because the reduced branch representation also contracts the first branch layer relative to a plain branch.The comparison changes only the branch representation while keeping the DeepONet trunk and output construction fixed.
  • Experimental framing: A spectral-only branch sharing the same basis provides the input-side reduced-basis reference, while the learned residual path is interpreted as residual representation rather than sparse variable selection.Existing alternatives compress the output side, both input and output, or both sides without a coordinate-query head.

2. Preliminaries

The preliminaries define operator learning for parametric PDEs and describe DeepONet’s sensor-based branch and coordinate-based trunk representation. They then motivate FAST-DeepONet through factor-adjusted modeling, combining fixed spectral features with a trainable path on orthogonal residuals.

  • Operator learning: Operator learning approximates mappings between input and output function spaces induced by parametric PDE solution processes.Inputs may include spatially varying fields and physical parameters, while outputs include stationary or terminal solutions.
  • DeepONet: DeepONet samples input functions at fixed sensors, encodes them with a branch network, and combines that representation with a trunk network encoding output coordinates.Sensor locations and ordering remain fixed across samples, while output coordinates need not coincide with sensors.
  • Factor-adjusted modeling: Approximate factor models decompose high-dimensional covariates into common factors and idiosyncratic components, motivating sparse use of residual coordinates.FAST-NN combines projected factor features with a trainable throughput path and penalizes additional coordinate selection.
  • FAST-DeepONet: FAST-DeepONet adapts this two-path structure by using spectral features from training inputs and applying the trainable branch path explicitly to the orthogonal residual.The scalar regression network is replaced by a DeepONet branch combined with a coordinate-dependent trunk.
  • FAST-DeepONet: FAST-DeepONet retains the throughput path and clipped penalty, regularizing them through an effective map that yields a distributed residual representation.The effective map concentrates on a small fraction of branch coordinates, though the selected coordinates can differ across fits.

3. FAST-DeepONet

FAST-DeepONet replaces raw high-dimensional branch inputs with fixed spectral coordinates and a learned low-dimensional encoding of the orthogonal residual. Its residual regularization acts on the effective residual map after row normalization, making the penalty independent of arbitrary feature factorizations and positive layer rescalings, but discontinuous at zero rows.

  • Architecture: FAST-DeepONet combines a fixed spectral path with a learned low-dimensional encoding of the orthogonal residual, concatenating both before the branch output.The trunk maps an output query to trunk features, and the branch–trunk inner product plus a bias predicts the field value.
  • Representation: Auxiliary physical parameters enter the fixed representation directly and are excluded from the residual path.When no auxiliary parameters are present, the auxiliary blocks are omitted and the field rank equals the requested total rank.
  • Residual regularization: The directional penalty is applied to G, the effective map from original branch coordinates to first-layer residual contributions, rather than directly to a factor such as Θ.Directly penalizing Θ depends on arbitrary factorizations and on components that do not affect predictions.
  • Limitations: Exact row normalization makes the penalty discontinuous and nondifferentiable at zero rows, although the zero-row convention keeps it well defined.Replacing the denominator with the row norm plus a positive epsilon would restore continuity, but the passage is truncated before stating the full consequence.
  • Residual regularization: The row-normalized penalty is invariant to positive rescalings of first-layer ReLU units and depends on realized residual-row directions rather than adjacent-layer factorization or scaling.This invariance follows from exact normalization and the corresponding transformation of G under positive diagonal rescaling.

4. Numerical experiments

Numerical experiments on Navier–Stokes, Darcy, and signed terminal wavefield prediction show that FAST-DeepONet improves accuracy and remains stable as input dimension grows. Ablations indicate that the fixed spectral path drives gains on Navier–Stokes and Darcy, whereas wave prediction benefits from the regularized residual path.

  • Main benchmark comparison: 37.0%, 10.1%, and 4.7% are FAST-DeepONet’s reductions in mean relative L2 error for Navier–Stokes, Darcy, and signed terminal wavefield prediction, respectively.The comparisons use independent test sets, and both intervals in Table 3 exclude zero for each benchmark.
  • High-dimensional small-sample scaling: 0.1556 is Plain DeepONet’s mean error at p=8193 versus 0.0394 at p=129, while FAST changes from 0.0422 to 0.0408.At n=240, this grid sweep shows Plain degrading as the Navier–Stokes sensor dimension increases while FAST remains stable.
  • High-dimensional small-sample scaling: At n=20, both methods remain near 0.2, so the representation does not compensate for severe sample scarcity.FAST’s test-mean spread across paired seeds is narrower at three of four corners, by factors of 4.4, 9.9, and 15.1 in reported settings.
  • Path ablations: For terminal wave prediction, FAST lowers the reference error from 0.2557 to 0.2063, a 19.3% reduction, whereas the residual path offers little or negative benefit on Navier–Stokes and Darcy.The reference reaches 0.1108 versus 0.1119 for Navier–Stokes and 0.0876 versus 0.0925 for Darcy.
  • Path ablations: With λ=0, the residual path is indistinguishable from the spectral-only reference on terminal wave and actively harmful on Darcy, implicating regularization rather than path addition.The ablation changes only λ, leaving the other conditions unchanged.

5. Conclusion

FAST-DeepONet stabilizes high-dimensional DeepONet branches by combining fixed spectral factors with a regularized projection of the orthogonal residual. Across three benchmarks, it reduces error by 4.7% to 37.0% while using three to seven times fewer parameters, but the results are limited to generated PDE families, fixed sensor layouts, and 40 to 60 training operators.

  • Method: FAST-DeepONet combines fixed spectral factors with a regularized projection of the orthogonal residual to stabilize high-dimensional branches.The regularizer acts on the normalized effective residual map, removing dependence on nonidentifiable parameter scalings.
  • Efficiency: 3–7 times fewer parameters are required than for a plain branch.This reduction accompanies the combined spectral-factor and residual-projection representation.
  • Results: 4.7% to 37.0% lower error is achieved across the three benchmarks relative to a plain branch.The margin grows with branch dimension, reaching nearly a factor of four on Navier–Stokes at p=8193, but disappears when the branch is small.
  • Results: The fixed spectral path carries the gain on Navier–Stokes and Darcy, while the two paths do not compensate for severe sample scarcity.The supplied conclusion passage also states that the two paths do not contribute equally.
  • Limitations: The results cover three generated PDE families with fixed sensor layouts and 40 to 60 training operators.Future work will consider varying sensor layouts, broader PDE distributions, and adaptive residual capacity.

CRediT authorship contribution statement

The paper credits Jiyong Kwon with leading conceptual, methodological, software, analytical, investigative, and original-draft contributions, with BongSeok Kim and Guang Lin contributing to methodology, investigation, supervision, and manuscript review.

  • Jiyong Kwon contributed to conceptualization, methodology, software, formal analysis, investigation, and writing the original draft.
  • BongSeok Kim contributed to data curation, methodology, investigation, and writing review and editing.
  • Guang Lin contributed to supervision, conceptualization, and writing review and editing.

A. Data separation and reproducibility … H. Fixed optimization-budget diagnostic

The paper separates development, tuning, and independent evaluation, then uses paired controls and sensitivity studies to identify when the spectral and residual paths, scale, and penalty drive FAST-DeepONet’s behavior. Diagnostics further provide representation-selection guidance and show that the main conclusions largely persist under a shorter optimization budget.

  • A. Data separation and reproducibility: All configurations were fixed on development data before independent accuracy streams were opened, with paired seeds, test paths, targets, checkpoints, and metadata archived for audit.The spectral-only reference shared the same seed vectors, and preprocessing, query schedules, and paired seeds were frozen before its added Navier–Stokes fits.
  • B. Spectral-only reference: The fixed spectral path suffices for low-rank Navier–Stokes and is more accurate for Darcy, whereas the residual path provides its clearest benefit for terminal wave.At retained rank six, spectral-only wave error is 0.2557 versus 0.2165 for a plain branch, while FAST reaches a pooled 90th-percentile error of 0.303 versus 0.291 for Plain.
  • C. Two controls on the source of the gain: Random-basis and parameter-matched plain controls were worse than the reported representations, showing that the gain depends on the training-input spectral subspace rather than compression or parameter count alone.The random basis loses to the spectral one by factors of 2.5, 1.7, and 1.5 across the three benchmarks, while the narrower parameter-matched plain branch underperforms the full-width plain branch.
  • D. Sensitivity to the directional penalty: 33.2% and 19.8% are the directional-penalty reductions in mean error on Darcy and terminal wave, respectively, while the Navier–Stokes paired interval includes zero.The selected penalty wins at every fit, with fieldwise advantages on 94 and 100 of 100 test fields; without it, wave reaches 0.2572 versus 0.2063 for FAST, and Darcy rises from 0.0876 to 0.1385.
  • E. Sensitivity to the factor scale: Terminal-wave tuning MSE falls from 0.7331 at c_f=1 to 0.2168 at c_f=8, while Darcy varies only from 0.0900 to 0.0925 across the scale grid.Wave test error follows the same ordering, from 0.3980 to 0.2063; Darcy’s tuning-MSE choice is c_f=1 although the lowest test error occurs at c_f=2.
  • F. Diagnostics of the effective residual map; F.1. Directional concentration: The row-normalized effective residual map has no zero rows, is heavy-tailed and far from uniform, yet its participation ratio indicates tens of active directions rather than a handful.The smallest nonzero row norm is 0.265, and concentration is therefore reported using counts rather than energy shares dominated by a few columns.
  • F.2. Relation to the generative support; F.3. Residual information after the spectral path: 5.3 and 2.4 are the enrichment factors of the largest residual-map columns on true scatterer support for Darcy and wave, while 69.2 and 43.9 are the corresponding absolute residual-variance factors.Roughly two thirds of the largest columns remain outside the disks, so the residual path reflects support-related information without recovering the support directly.

I. Training curves

Training curves show that all reported fits retain their own lowest tuning-MSE checkpoint within the training budget. FAST reaches its optimum later and maintains descent better than Plain, especially on Navier–Stokes.

  • Training curves: Figure I.1 reports per-epoch tuning MSE for five paired seeds per representation, with lower values preferred and vertical scales comparable only within each benchmark.The curves describe the fits behind Table 2; no model is refitted.
  • Checkpoint selection: Every fit retains the epoch of lowest tuning MSE, and each retained epoch falls within the training budget.The reported Plain errors therefore reflect the branch’s best result under this checkpoint schedule.
  • Optimization dynamics: 37 to 71 versus 222 to 297 retained epochs on Navier–Stokes distinguish Plain from FAST, respectively.The two representations leave the descent at different times, with FAST improving later.
  • Optimization dynamics: 6.3% versus 1.1% final-epoch tuning-MSE excess over the minimum shows greater late drift for Plain than FAST on Navier–Stokes.The same ordering holds on the two smaller benchmarks, but with a smaller gap.

J. Parameter count and training cost

FAST-DeepONet sharply reduces parameter counts by replacing the raw sensor-reading branch layer with a factor-and-residual representation. This structural saving comes with a modest training-time increase from forming the effective map at each optimizer step.

  • Parameter count: 6.8 times fewer parameters occur on Navier–Stokes, while each smaller benchmark uses 3.7 times fewer than Plain.The reduction is measured from retained checkpoints producing the reported errors.
  • Parameter count: The structural reduction replaces Plain’s first layer of size w × p with FAST-DeepONet’s w × (r_eff + s).The factor and residual representation means the first branch layer no longer reads every sensor.
  • Training cost: 1.3 times the wall-clock is required for one FAST-DeepONet fit compared with a plain branch.FAST-DeepONet forms the w × p effective map at every optimizer step; this cost grows linearly in p and is independent of the number of training fields.

K. Sensor resolution and sample size … L. Terminal wave propagation

The appendix validates FAST-DeepONet across sensor-resolution and sample-size settings, explains limitations of coarse point sampling, and confirms the terminal-wave result on a fresh bank. It also documents signed wavefield propagation through a heterogeneous medium to the terminal learning target.

  • K. Sensor resolution and sample size: The Navier–Stokes corner study uses paired model seeds and a fixed 240-field bank, with a 128 × 128 target grid throughout.Interior grid cells use one prespecified seed and are descriptive.
  • K.1. The designed benchmarks: Fresh per-sample seeds are used for the terminal-wave confirmatory generator because the earlier stream predates zero-velocity half-step initialization.The Darcy generator stream instead continues from index 610 after indices 0 to 609 were allocated to the reported study.
  • K.1. The designed benchmarks: The benchmark sweep uses training banks disjoint from reported splits, making cells comparable with each other but not with Table 2.Table K.1 reports FAST reduction in mean relative L2 error against a plain branch at matched sensor grids and training-set sizes.
  • K.1. The designed benchmarks: The wave column remains below the other benchmark columns, consistent with spectral-only loss at retained rank six and recovery by the complete representation.This finding matches Appendix B’s interpretation of the wave benchmark.
  • K.2. What a coarse sensor grid measures: At 8 × 8, sensors fall inside only 30.9% of active Darcy disks and 8.5% of wave disks, whereas every active disk contains a sensor from 16 × 16 onward.Coarsening uses point sampling rather than averaging, so structures narrower than sensor spacing can fall between sensors.
  • K.3. The reported wave configuration on a fresh bank: 6.4%: On a fresh bank, FAST-DeepONet reaches 0.2206 ± 0.0029 versus 0.2357 ± 0.0037 for a plain branch across five paired seeds.FAST has lower error at every seed, with paired interval [−0.0211, −0.0091]; the confirmatory bank reports 4.7%.
  • L. Terminal wave propagation: The terminal-wave snapshots show a centered pulse propagating through a heterogeneous medium, developing distorted fronts from t = 0.08 to terminal time T = 0.4.All panels share one symmetric color scale, and the final panel is the learning target.

M. Monte Carlo dropout and operator-level conformal calibration

FAST’s dropout variant uses ten-pass Monte Carlo means and operator-level split conformal calibration to produce uncertainty intervals with a grid-point coverage guarantee under exchangeability. Empirically, FAST has lower point error across all three benchmarks, with shorter intervals on Navier–Stokes and Darcy but slightly wider terminal-wave intervals.

  • Monte Carlo dropout: FAST and Plain are retrained with dropout rate 0.1 after every ReLU hidden layer in both branch and trunk networks.The dropout fits are separate from the deterministic point predictors.
  • Monte Carlo dropout: Ten Monte Carlo dropout passes are averaged at inference to form the point prediction used for calibration.The passes follow a paired random-number schedule, and the resulting intervals are centered at the ten-pass mean.
  • Operator-level conformal calibration: Split conformal calibration constructs intervals around the ten-pass mean using fixed choices and fresh calibration and uncertainty-quantification test fields.The calibration protocol is fixed before generating the fresh fields.
  • Operator-level conformal calibration: The conformal guarantee concerns the fraction of covered grid points for a new operator, marginal over the random calibration sample, assuming exchangeability.This is an operator-level rather than pointwise statement.
  • Empirical evaluation: FAST has lower MC-mean point error on all three benchmarks, shorter intervals on Navier–Stokes and Darcy, and slightly wider intervals on terminal wave.Field-success flags differ by benchmark: all five FAST seeds are flagged on Navier–Stokes, both methods on Darcy, and neither on terminal wave.

N. Implementation checks

Implementation checks verified that FAST-DeepONet’s parameterization preserves its effective map and penalty under key transformations, including degenerate zero-row cases. Separate float64 replay, refinement, serialization, and regeneration audits checked solver consistency across Navier–Stokes, Darcy, and wave inputs.

  • Parameterization checks: The effective map and directional penalty remain unchanged to floating-point tolerance under row normalization, residual-coordinate, factor-null-space, and positive-ReLU row-scaling transformations.Zero-row backward evaluations also remain finite.
  • Solver audit: A solver-only audit evaluated five fixed accuracy-test indices per stream without loading or evaluating a learned model.The reported refinement error is the relative L2 difference between production and refined solutions restricted to the production grid.
  • Solver audit: Float64 replay, refinement, serialization, and regeneration checks addressed Navier–Stokes, Darcy, and wave solver representations.Wave regeneration from the saved seed reproduced stored arrays exactly, while the table’s wave check re-solved the stored medium and sampled the refined periodic field at coarse cell centers.
Loading 2608.15408v1…