PaperlayerView exact revision on arXiv

Source-linked AI summary

Pretrain Finite Element Method: A Pretraining and Warm-start Framework for PDEs via Physics-Informed Neural Operators

Yizheng Wang, Zhongkai Hao, Mohammad Sadegh Eshaghi, Cosmin Anitescu, Xiaoying Zhuang, Timon Rabczuk, Yinghua Liu

arXiv:2601.03086v3math.NA

TL;DR

Existing PDE-solving approaches face efficiency, data-dependence, or accuracy limitations, and transformer-based PINO pretraining with warm-starting remains underexplored. PFEM combines physics-only Transolver pretraining on point clouds with FEM warm-start refinement, yielding broad generalization while retaining classical solver accuracy and reducing computational effort.

  • Problem

    Transformer-based physics-informed neural operators with a pretraining and warm-start mechanism remain underexplored, while PDE solvers face efficiency, retraining, data, or accuracy limitations.

  • Method

    PFEM trains a Transolver-based neural operator solely from governing PDEs on point-cloud inputs, then uses its predictions as initial guesses for conventional FEM solvers.

  • Results

    PFEM demonstrates generalization across geometries, boundary conditions, materials, and linear or nonlinear solvers while combining neural-operator efficiency with traditional numerical accuracy.

  • Takeaways & Limitations

    PFEM provides a pretraining and warm-start framework that reduces iterative-solver effort while supporting flexible encoding-based generalization across PDE settings.

  • Takeaways & Limitations

    PFEM currently focuses on quasi-static problems and remains constrained by high-dimensionality challenges and sensitivity to input-point sampling distributions.

Abstract

from arXiv · show

We propose a Pretrained Finite Element Method (PFEM),a physics driven framework that bridges the efficiency of neural operator learning with the accuracy and robustness of classical finite element methods (FEM). PFEM consists of a physics informed pretraining stage and an optional finetuning stage. In the pretraining stage, a neural operator based on the Transolver architecture is trained solely from governing partial differential equations, without relying on labeled solution data. The model operates directly on unstructured point clouds, jointly encoding geometric information, material properties, and boundary conditions, and produces physically consistent initial solutions with extremely high computational efficiency. PDE constraints are enforced through explicit finite element, based differentiation, avoiding the overhead associated with automatic differentiation. In the fine-tuning stage, the pretrained prediction is used as an initial guess for conventional FEM solvers, preserving their accuracy, convergence guarantees, and extrapolation capability while substantially reducing the number of iterations required to reach a prescribed tolerance. PFEM is validated on a broad range of benchmark problems, including linear elasticity and nonlinear hyperelasticity with complex geometries, heterogeneous materials, and arbitrary boundary conditions. Numerical results demonstrate strong generalization in the pretraining stage with relative errors on the order of 1\%, and speedups of up to one order of magnitude in the fine-tuning stage compared to FEM with zero initial guesses.

1. Introduction

PFEM addresses the gap in transformer-based physics-informed neural operators with pretraining and warm-starting, combining physics-only learning with classical FEM refinement. It targets flexible generalization across PDE settings while retaining neural-operator efficiency and numerical-solver accuracy.

  • Motivation: Traditional PDE solvers trade computational efficiency for accuracy and often require rerunning when geometry, materials, or boundary conditions change.
  • Related paradigms: PINNs typically solve individual PDE instances and require retraining when initial or boundary conditions change.
  • Related paradigms: PINO reduces data dependence by incorporating governing equations, but its accuracy remains below traditional FEM, motivating hybrid approaches.
  • PFEM framework: PFEM pretrains a Transolver-based neural operator solely from governing equations, using point-cloud inputs that encode geometry, boundary conditions, and material properties.
  • Results: Experiments cover varied geometries, boundary conditions, materials, and linear or nonlinear solvers, with progressively stronger generalization as training samples increase.
  • PFEM framework: PFEM uses pretrained predictions as FEM initial guesses, reducing iterative-solver work while preserving arbitrarily high accuracy.

2. Prerequisite knowledge

Neural operators learn mappings between function spaces for PDE solution prediction, while PINO incorporates governing equations to reduce dependence on labeled data. FNO uses structured-grid representations, whereas complex geometries and physics-informed training motivate alternative formulations.

  • Operator learning: Neural operators learn mappings between function spaces, such as boundary-condition functions and PDE solution fields.In practice, these functions are represented by discrete spatial samples.
  • Operator learning: Purely data-driven operator learning requires large amounts of high-quality training data that may be expensive and noisy.This limitation motivates physics-informed alternatives.
  • FNO limitations: FNO typically represents inputs and outputs on structured grids, which restricts its ability to describe complex geometries accurately.The text identifies approximation errors for complex domains and persistent challenges even after geometric extensions.
  • FNO limitations: FFT-based efficiency makes abandoning structured grids difficult within the original FNO framework.The fast Fourier transform requires regularly gridded data.
  • Physics-informed neural operators: PINO embeds governing PDEs into neural-operator training, reducing data requirements and sometimes enabling training without labeled data.PINO losses use residual operators for the governing PDE and boundary conditions.
  • Physics-informed neural operators: PINO losses can use strong or variational formulations, with the choice depending on whether the governing PDE admits an energy formulation.Variational formulations generally involve lower-order derivatives and can improve efficiency and accuracy empirically.

3. Method

PFEM combines a Transolver-based physics-informed pretraining stage with an optional warm-start stage for classical iterative solvers. It uses point-cloud inputs and explicit finite-element differentiation to produce initial solutions that can be refined to the required accuracy.

  • 3.1. Transolver: Transolver reduces attention complexity from O(N^2) to O(N + S^2) by operating on S physics-aware tokens instead of all N input points.Because S is typically much smaller than N, the token-based design improves scalability for large, complex problems.
  • 3.1. Transolver: Transolver encodes spatial coordinates, material information, and boundary conditions from irregular point clouds.The representation does not require points to lie on a structured grid.
  • 3.2. Pretrained Finite Element Method: PFEM trains a Transolver-based neural operator solely with governing physical equations, without labeled solution data.The framework has separate pretraining and warm-start stages, both relying exclusively on physical equations.
  • 3.2.1. Pretraining: PFEM converts finite-element nodal information into point-cloud inputs and predicts quantities of interest at the same spatial locations.The prediction is treated as a low-fidelity solution before physical refinement.
  • 3.2.1. Pretraining: Explicit differentiation through finite-element shape functions constructs spatial derivatives from the predicted low-fidelity fields.This approach is described as less computationally costly than automatic differentiation.
  • 3.2.2. Warm-start: The warm-start stage initializes iterative finite-element solvers with the pretrained prediction and then refines it through classical iterations.The iterative update uses the operator prediction as U^(0), with geometry and material properties entering through the stiffness matrix.
  • 3.2.2. Warm-start: Warm-start refinement is optional: if the initial residual norm is below the user-defined threshold, further refinement can be skipped.With a sufficiently small convergence tolerance, the warm-start solution can achieve arbitrary accuracy.

5. Results

PFEM generalizes across geometry, material properties, boundary conditions, and input resolutions, producing low-error predictions for linear and nonlinear mechanics problems. Its warm-start predictions substantially reduce classical solver iterations while preserving the convergence behavior and accuracy of the underlying FEM algorithms.

  • Linear elasticity: Approximately 1% relative error is achieved across irregular hole geometries during PFEM pretraining.PFEM converges within approximately 100 training epochs.
  • Linear elasticity: PFEM predictions remain nearly identical across different point densities, with consistently low displacement errors despite different training and testing resolutions.This demonstrates robustness to the number of input points and discretization.
  • Warm-start refinement: PFEM’s warm-start reduces iterations because its improved initial guess accelerates convergence, while the convergence rate remains determined by the selected iterative algorithm.For linear elasticity, the same CG algorithm gives identical convergence rates for PFEM and FEM; the iteration reduction comes from initialization.
  • Linear elasticity: 6.24 speedup over FEM is achieved at tol = 10^-3, requiring 83.32 PFEM iterations versus 519.88 FEM iterations.At tol = 10^-6, the resulting relative error reaches the order of 10^-7.
  • Linear elasticity: 9.18 speedup over conventional FEM is achieved at tol = 10^-3 while maintaining a relative error of approximately 10^-7 at tol = 10^-6.The result demonstrates accelerated convergence without sacrificing high accuracy.
  • Nonlinear hyperelasticity: PFEM generalizes across geometry, heterogeneous materials, and boundary conditions for nonlinear problems, while substantially reducing Newton iterations during warm-start refinement.These results require only point-cloud inputs and cover irregular geometries and strong nonlinearities.

6. Discussion

PFEM is evaluated under distribution shifts, diverse geometries, and sampling variations, while its patch test provides a low-cost screening criterion for pretraining models. The framework combines physics-only training, explicit differentiation, and FEM warm starts to retain accuracy and improve solver efficiency.

  • Out-of-distribution generalization: PFEM accurately predicts displacement fields under shifted geometry, material, and boundary-condition distributions, with only a moderate increase in error relative to in-distribution cases.The evaluated plate uses a circular hole at (2, 2) with radius 0.5 and distributions differing from training.
  • Patch tests for PFEM: A PFEM patch test evaluates candidate pretraining models using only a very small amount of data, potentially a single sample, and PDE constraints.The procedure trains on one known analytical or high-fidelity reference case without a data-driven loss, then compares the prediction with the reference.
  • Patch tests for PFEM: N × M bounds the typical maximum training iterations when convergence requires N iterations for one sample and the dataset contains M samples with batch size one.Practical convergence often requires substantially fewer iterations than this bound.
  • Patch tests for PFEM: The patch test is a necessary model-selection condition: failure on single-sample PDE-constrained training suggests discarding the candidate before larger-scale training.Its limited data and computational cost make it a rapid screening tool for neural operator architectures.
  • Sampling sensitivity: Changes in the input point-distribution PDF can introduce approximation errors, so uniformly distributed sampling is recommended despite observed robustness to moderate deviations.Sensitivity to the sampling PDF remains a stated limitation and future-research direction.
  • Physics-only training: PFEM training requires no labeled data and avoids discretization errors from numerically generated fixed-mesh datasets by enforcing governing equations directly.This zero-shot formulation trains the neural operator through physical equations rather than precomputed solver outputs.
  • Physics-only training: Explicit finite-element-based differentiation is advocated because it offers lower cost and stronger numerical robustness than automatic differentiation for repeated PDE derivative evaluation.Its computational cost remains essentially independent of derivative order, supporting efficiency comparable to data-driven training.
  • Warm-start behavior: PFEM’s FEM warm-start stage preserves extrapolation capability beyond training distributions, while higher pretraining accuracy reduces the iterations needed for refinement.The framework is intended to become more efficient as it encounters increasingly diverse geometries, materials, and boundary conditions.

7. Conclusion

PFEM combines a Transolver-based, PDE-only pretraining stage with a FEM warm-start stage that uses predictions as initial guesses for conventional iterative solvers. The framework is validated across linear and nonlinear mechanics problems, while remaining limited by its current quasi-static scope, high-dimensional challenges, training speed, and sampling-strategy design.

  • Framework: PFEM integrates neural-operator efficiency with classical FEM accuracy and robustness while eliminating labeled training data.Its two stages are pretraining through PDEs and warm starting a conventional iterative solver with the pretrained prediction.
  • Validation: The framework is validated on elasticity and hyperelasticity benchmarks spanning variations in geometry, material properties, and boundary conditions.The point-cloud input encodes spatial coordinates, material properties, and boundary conditions.
  • Limitations: PFEM currently focuses on quasi-static problems, leaving transient and time-dependent extensions for future research.The proposed direction is to incorporate temporal derivatives using finite difference schemes.
  • Limitations: PFEM faces challenges from the curse of dimensionality, slower Transolver training, and the need for effective input-point sampling strategies.These are identified as limitations and extension directions alongside transient-problem support.
  • Outlook: The authors position PFEM as a potential new paradigm for computational mechanics combining artificial intelligence with finite element methods.This is presented as the paper’s forward-looking conclusion.

CRediT authorship contribution statement

The paper’s authors divide contributions across conceptualization, methodology, analysis, investigation, validation, visualization, writing, supervision, funding, and data curation.

  • Contributions: Yizheng Wang covers conceptualization, methodology, formal analysis, investigation, data curation, validation, visualization, and writing.The contribution statement also credits Wang with original drafting and review and editing.
  • Contributions: Zhongkai Hao and Mohammad Sadegh Eshaghi are credited with investigation.
  • Contributions: Cosmin Anitescu and Xiaoying Zhuang contribute supervision and writing review and editing.
  • Contributions: Timon Rabczuk contributes supervision, writing review and editing, and funding acquisition.
  • Contributions: Yinghua Liu contributes supervision and funding acquisition.

Appendix A. Comparison of computational graphs for automatic and explicit differentiation

The appendix contrasts automatic differentiation’s derivative-order-dependent computational growth with explicit finite-element-based differentiation. Explicit differentiation reconstructs the target field with shape functions, enabling analytical derivatives without additional computational graphs.

  • Pipeline role: The appendix frames the comparison around learning the mapping k(x) 7→T(x), whose spatial derivatives are needed to evaluate the differential operator.
  • Automatic differentiation: Automatic differentiation recursively constructs computational graphs through the chain rule, increasing computational cost and memory usage during repeated PDE-operator evaluation.The graph begins from the neural-operator mapping and expands for higher-order derivatives.
  • Automatic differentiation: When the highest PDE derivative order is M, automatic-differentiation complexity grows exponentially with M as O(2MCT ).This makes automatic differentiation increasingly inefficient for PDE training involving higher-order derivatives.
  • Explicit differentiation: Explicit differentiation uses finite-element shape functions to derive spatial derivatives analytically in closed form without adding computational graphs.Once the shape-function representation is established, derivatives of any order can be evaluated explicitly.
  • Complexity comparison: Explicit differentiation’s cost remains constant with respect to derivative order, distinguishing it from automatic differentiation in PDE-based operator learning.
  • Complexity comparison: For an M-th order derivative at N query points, automatic differentiation scales as O(2MCT N), whereas explicit differentiation scales as O(N).N denotes the total number of query points, and CT is the base neural-network mapping cost.

Appendix B. Iterative algorithms of the finite element method

The appendix formulates linear and nonlinear FEM as equilibrium problems solved iteratively, with convergence controlled by residual tolerances. PFEM bypasses early nonlinear load steps by supplying a pretrained final-step initial guess, reducing Newton iterations while retaining iterative refinement.

  • Linear FEM: Linear FEM solves a system involving the global stiffness matrix, external force vector, and nodal unknowns.
  • Linear FEM: Large-scale linear systems are commonly solved iteratively, terminating when the relative residual reaches a prescribed tolerance.
  • Nonlinear FEM: Nonlinear FEM solves a nonlinear equilibrium system in which internal force depends nonlinearly on displacement; quasi-static problems set inertial force to zero.
  • Nonlinear FEM: Nonlinear FEM uses material and geometric stiffness matrices within a total Lagrangian formulation for hyperelastic problems.
  • PFEM warm-start: PFEM bypasses the outer iteration over the first Nf −1 load steps and supplies a pretrained neural-operator prediction directly at the final load step.
  • PFEM warm-start: A pretrained initial guess close to the final solution significantly reduces the Newton iterations required by nonlinear FEM.

Appendix C. Benefit of the warm-start stage in PFEM

The appendix explains why PFEM warm starts reduce iterative-solver work: a better initial guess lowers the initial error and therefore the iteration bound for a fixed target accuracy. The benefit diminishes at very tight tolerances because tolerance-dependent work dominates.

  • A better initial guess reduces the initial error norm and lowers the iteration bound needed to reach the same target accuracy.
  • PFEM’s more accurate pretrained solution can reduce warm-start iterations, especially when conventional solvers lack a good initial guess.
  • A good initial guess has less pronounced benefit when the convergence tolerance is very small.
  • The convergence rate depends on the iteration matrix but is independent of the iteration index and initial guess.
  • As tolerance tightens, the relative contribution of the initial guess to total iteration count becomes smaller, diminishing speedup benefits at extremely small tolerances.

Appendix D. Numerical homogenization theory

Numerical homogenization computes an effective elastic tensor by solving fluctuation fields for six macroscopic unit-strain load cases. PFEM targets the displacement-field solves, whose acceleration can improve homogenization efficiency across varying microstructures.

  • Loading and homogenization: The effective elastic tensor is computed from local material tensors and the fluctuation strain fields induced by the applied unit strains.
  • Loading and homogenization: Homogenization applies six independent unit strains—three normal and three shear—to the periodic unit cell.
  • Field solution: Fluctuation displacements satisfy a weak equilibrium form and are typically solved with FEM before their strains are substituted into the homogenization formula.
  • Boundary conditions: Periodic boundary conditions enforce matching fluctuation displacements on opposite unit-cell faces and can be imposed by constraining corresponding nodal degrees of freedom.
  • Boundary conditions: PFEM imposes periodic constraints through ring-type coupling using virtual elements whose strain energy is equivalent to a penalty formulation.
  • Computational motivation: Because traditional homogenization repeatedly solves displacement fields for all six load cases, accelerating those solves can improve overall efficiency.

Appendix E. Generation of TPMS geometries

TPMS geometries are represented through level-set fields that define periodic minimal-surface topologies and separate solid from void. Varying the level-set threshold controls volume fraction, while solid- and sheet-network constructions provide distinct architectures.

  • TPMS representation: TPMS are periodic minimal surfaces characterized by zero mean curvature and represented using level-set formulations.
  • TPMS representation: The level-set value partitions a TPMS cell into solid and void regions, and varying it adjusts porosity or volume fraction.
  • TPMS architectures: Solid-network cells treat regions with ϕ(x, y, z) > c as solid, whereas sheet networks retain a thin layer around the minimal surface.
  • Material parameters: Volume fraction is the solid-to-total-volume ratio and directly influences the effective elastic tensor and unit-cell stiffness.
  • Geometric dataset: The study generates TPMS cells from the Schoen Gyroid, Schwarz Diamond, and Fischer Koch S families for training and validation.
Loading 2601.03086v3…