PaperlayerView exact revision on arXiv

Source-linked AI summary

An overview of machine learning-enhanced iterative methods for systems of linear and nonlinear equations

Yuhuang Meng, Jing Zhao, Alexander Heinlein

arXiv:2610.07211v1math.NAcs.LGcs.NA

TL;DR

Choosing effective iterative solvers remains challenging, and existing reviews had not systematically covered ML-enhanced classical methods for both linear and nonlinear systems. This review organizes those hybrid approaches by how ML is incorporated, and finds that existing work is concentrated on linear systems while nonlinear methods remain less studied.

  • Problem

    Existing reviews addressed specific topics or broader scientific machine learning, while a systematic review of ML-enhanced classical iterative methods for equation systems was lacking.

    Source: §1.3 · p. 5

  • Method

    The paper synthesizes ML-enhanced iterative methods for linear and nonlinear systems by organizing approaches according to how ML is incorporated into classical iterative frameworks.

    Source: §1.3 · p. 5

  • Results

    Most existing work concerns linear systems, whereas nonlinear hybrid iterative methods are less extensively studied.

    Source: §6 · p. 53

  • Takeaways & Limitations

    The review identifies nonlinear hybrid iterative methods as an important direction for future research.

    Source: §6 · p. 53

    Research on nonlinear update-function learning and nonlinear preconditioning remains relatively limited, reflecting difficulties in stabilization, analysis, feature selection, and training-data generation.

    Source: Discussion · p. 49

Abstract

from arXiv · show

Systems of equations arise in a wide range of scientific and engineering applications. The present work focuses on solvers for general systems of equations, including but not limited to those arising from partial differential equations. These systems can be broadly categorized into linear and nonlinear problems. For large linear systems, iterative solvers are generally preferred over direct methods due to the latter's superlinear growth of computational costs. Although convergence theory is well-developed under certain assumptions on the coefficient matrix, many classes of systems still pose open challenges. These difficulties become even more severe for systems of nonlinear equations, where nonlinear solvers typically rely on repeated linearization. For example, Newton's method may even converge quadratically near the solution; it can also converge slowly or diverge when the initial guess is not chosen appropriately. A wide range of solvers with diverse variants and hyperparameter settings exists, and the development of efficient and robust iterative methods remains an active area of research. Recently, machine learning (ML) techniques have been applied to enhance the efficiency of classical iterative methods while preserving their interpretability and reliability. We refer to these ML-enhanced iterative methods as hybrid iterative methods, in the sense that they combine classical iterative methods with ML. This paper provides a comprehensive overview of state-of-the-art approaches to constructing hybrid iterative methods for systems of both linear and nonlinear equations, while also discussing open challenges and outlining potential directions for future research.

1 Introduction

The introduction motivates hybrid iterative methods (HIMs) as a way to combine machine learning with classical solvers for linear and nonlinear systems. It establishes the review’s terminology and scope, then organizes HIMs by how ML enters classical iterative frameworks.

1.1 Background and motivation

  • Large linear systems make direct methods computationally demanding, while classical iterative methods depend on problem-specific choices that can be costly to construct; nonlinear convergence is more delicate and sensitive to initialization.Newton-type methods may converge quadratically near a solution, but can be slow or diverge from a poor initial guess.Source: §1.1 · pp. 2–3
  • The review uses “hybrid iterative methods” for approaches in which ML enhances and complements, rather than entirely replaces, a classical numerical method.This hybridization is presented as combining classical methods’ robustness and, when available, convergence guarantees with ML’s flexibility.Source: §1.1 · p. 3, §1.3 · p. 4

General iterative framework for linear systems

  • A linear iterative method starts from an initial guess and generates successive approximations through an update function and its associated parameters.The review focuses, for simplicity, on square systems and describes the update direction and relaxation parameter as one common formulation.Source: General iterative framework for linear… · p. 3

1.2 Iterative methods and terminology

  • The review organizes iterative methods around three core components: the initial guess, the update function, and algorithmic parameters.For linear systems, preconditioning can be part of the update function, but constructing an effective, inexpensive preconditioner is often challenging and problem-dependent.Source: General iterative framework for linear… · p. 3

General iterative framework for nonlinear systems

  • Nonlinear solvers retain the same core components, but are generally more sensitive to the initial guess; nonlinear preconditioning and extensions of multigrid and domain decomposition are also considered.Solver selection has received relatively little attention for nonlinear systems, in part because fewer solvers are commonly used and linearization-based methods rely on linear solvers at each iteration.Source: General iterative framework for… · p. 4

1.3 Scope and organization of this paper

  • The review classifies linear HIMs into core-component learning, preconditioner learning, subspace-correction enhancements, and automated solver selection.These categories cover learning initial guesses, update functions, or parameters; learning or improving preconditioners; designing components such as smoothers or coarse spaces; and selecting a solver configuration from problem features.Source: §1.3 · p. 4
  • A systematic review of ML-enhanced classical iterative methods for systems of equations was lacking; this paper organizes approaches by how ML enters the framework and covers both linear and nonlinear systems.The paper also discusses benefits, limitations, and open challenges, with later sections covering ML techniques, linear HIMs, nonlinear HIMs, related topics, and future perspectives.Source: §1.3 · p. 5

2 Machine learning preliminaries for hybrid iterative methods

The section introduces ML tools commonly used in hybrid iterative methods and explains their relevant characteristics and connections to classical iterative methods. It covers dimensionality reduction and neural architectures suited to different data structures.

2 Machine learning preliminaries for hybrid iterative methods

  • The review introduces data, models, and learning as the ingredients of ML, with training adjusting model parameters to capture patterns in observed data.It also outlines supervised, unsupervised, and reinforcement-learning paradigms.Source: §2 · p. 6

Proper orthogonal decomposition

  • POD reduces high-dimensional data by projecting centered samples onto a lower-dimensional subspace, reducing memory requirements and computational cost.The basis can be chosen from the leading left singular vectors of the centered data matrix; the review notes its use in ROM-based preconditioners and nonlinear preconditioning.Source: Proper orthogonal decomposition · p. 6

Feedforward neural networks

  • FNNs are nonlinear function approximators, but vectorized inputs do not explicitly encode relationships among data elements, limiting flexibility on structured or sequential data.The review connects FNN spectral bias toward low-frequency components with stationary relaxation schemes that damp high-frequency errors.Source: Feedforward neural networks · pp. 7–8

Convolutional neural networks

  • CNNs exploit local spatial structure on regular grids; in linear-system applications, they can process matrix images or grid-based PDE solutions.Convolution extracts local features, while pooling downsamples and reduces computational cost; the review relates pooling to multigrid restriction.Source: Convolutional neural networks · pp. 8–9

Graph neural networks

  • GNNs aggregate information over graph neighborhoods, making them suited to irregular data such as PDE solutions on unstructured meshes.They can also process coefficient matrices represented as graphs, using matrix sparsity to inform graph structure.Source: Graph neural networks · pp. 9–10

Recurrent neural networks

  • RNNs use recurrent hidden states to carry information across sequential inputs, and their recurrence resembles iterative processes used to learn update functions.Standard RNNs may struggle with long-range dependencies because of vanishing gradients; LSTMs, GRUs, and transformers are among the architectures developed to address this issue.Source: Recurrent neural networks · p. 10

Meta-learning

  • Meta-learning aims to improve adaptation to new tasks by learning the learning process, and has been used to select parameters for iterative solvers.The review also identifies few-shot learning and neural architecture search as typical applications.Source: Meta-learning · pp. 10–11

3.2 Preconditioner learning

Preconditioner learning addresses the problem dependence and construction cost of effective preconditioners by using ML to approximate or enhance inverse actions. The review organizes these methods into inverse-action learning, enhancement of classical preconditioners, and alternating hybrid iterations.

3.2 Preconditioner learning

3.2.1 Preliminaries and motivation

  • Preconditioning transforms a linear system to improve iterative convergence; for nonsymmetric or nonnormal systems, GMRES convergence generally cannot be inferred from eigenvalues or condition number alone.Standard preconditioned Krylov methods use a fixed linear preconditioner, while nonlinear or varying preconditioners generally call for flexible methods such as FCG or FGMRES.Source: §3.2.1 · p. 19, Preconditioned Krylov methods · p. 19

Classical choice of preconditioners

  • Classical choices include matrix-splitting, incomplete-factorization, sparse approximate-inverse, and subspace-correction preconditioners, as well as other specialized approaches.Examples include Jacobi and Gauss-Seidel, IC and ILU, SPAI, multigrid, domain decomposition, deflation, and augmentation.Source: Classical choice of preconditioners · pp. 19–20

Motivation for preconditioner learning

  • Effective preconditioners are problem-dependent and costly to construct, especially across related systems; ML may help learn problem-adapted preconditioners or improve classical construction.More sophisticated preconditioners can reduce iterations while increasing construction or application costs.Source: §3.2 · p. 19, Motivation for preconditioner learning · p. 20

3.2.2 Learning the action of the inverse

  • Training objectives for inverse-action learning include direct inverse-action error, residual-based losses, and hybrid losses that combine objectives simultaneously or across training stages.Residual losses assess the predicted action through the residual of Az = q; hybrid training can pretrain with residual loss and fine-tune using a Krylov-subspace angle loss.Source: §3.2.2 · p. 22

Inverse-action error losses

The reviewed inverse-action methods vary in their learned models, training losses, and solver integration. The section also discusses training-data choices and computational trade-offs, then introduces ML enhancements to classical preconditioners, including IC-based approaches.

Inverse-action error losses

  • Local FNNs learn stencil-based corrections for shallow-water systems, while discretization-invariant neural operators can transfer nonlinear preconditioners trained on coarse grids to finer elliptic-PDE grids.The latter are trained from residual-error pairs collected during unpreconditioned CG and used with FCG.Source: Inverse-action error losses · pp. 22–23

Residual losses

  • Residual-trained learned inverse actions appear in methods including CNN-based DCDM, geometry-conditioned DLSM, GCN-based GNPs, and U-Net preconditioners for parametric PDE systems.DLSM omits Gram-Schmidt orthogonalization, which is reported to provide negligible numerical benefit in that method.Source: Residual losses · p. 23

Hybrid losses

  • A two-stage hybrid loss pretrains with residual loss and fine-tunes using a principal-angle loss derived from Arnoldi-process Givens rotations, designed to promote faster FGMRES convergence.The angle loss is differentiated with respect to the neural preconditioner parameters.Source: Hybrid losses · pp. 23–24

Discussion

  • Nonlinear learned preconditioners generally require flexible Krylov methods, which can add computation and memory overhead compared with standard preconditioned methods.The review also notes sampling trade-offs: standard-normal vectors are inexpensive but may miss difficult directions, whereas low-spectrum sampling costs more to construct.Source: Discussion · p. 24
  • For small systems, inference cost may make learned preconditioners uncompetitive in iteration counts or runtime; larger-system advantages may instead entail training-data generation and model-training costs.Flexible Krylov methods used with nonlinear learned preconditioners can add further computational and memory overhead.Source: Discussion · p. 24
  • Training-data generation remains an open challenge because emphasizing low-frequency components through approximate spectral subspaces can be computationally expensive.The review also identifies multistage hybrid-loss training and preservation of structural properties such as locality, symmetry, and sparsity as directions for further work.Source: Discussion · p. 24

3.2.3 Enhancing classical preconditioners

  • ML-enhanced classical preconditioners learn or improve selected components to reduce setup costs, improve adaptivity or preconditioning quality, while retaining structural advantages of established frameworks.For sparse IC, ILU, and SPAI matrices, GNNs can predict nonzero entries using the matrix sparsity pattern as graph connectivity.Source: §3.2.3 · p. 25

IC-based preconditioners

  • IC-based methods learn or correct the lower-triangular factor while preserving the IC structure, so the resulting preconditioner remains SPD and can be used with PCG when the learned factor has nonzero diagonal entries.Reviewed approaches include GNN-learned factors, unsupervised NeuralIF, learned corrections to classical IC, and PEARL’s actor-critic formulation.Source: IC-based preconditioners · pp. 25–26

ILU-based preconditioners

The review describes ML approaches that augment classical ILU and SPAI constructions and discusses Green’s-function-based preconditioning. It also locates ML enhancement of multigrid, domain decomposition, and deflation methods in a separate section.

ILU-based preconditioners

  • For ILU preconditioners, ML can predict fill-in patterns or learn sparse triangular factors for non-SPD systems, with resulting preconditioners used in GMRES-family methods.Examples include U-Net prediction of factor sparsity, GraphPAN for advection-dominated nonsymmetric problems, and learned factors trained with losses tied to sampled solutions or singular values.Source: ILU-based preconditioners · p. 26

SPAI-based preconditioners

  • SPAI approaches explicitly represent a sparse approximation of A−1; reviewed methods learn inverse matrices or parameterized sparse factors, and some target GPU-friendly sparse matrix-vector application.An early learned SPAI preconditioner outperformed Jacobi and Gauss-Seidel in iteration counts on its test problem; other approaches learn sparsity patterns or optimize Frobenius- or condition-number-based objectives.Source: SPAI-based preconditioners · p. 27

MG-, DD-, and deflation-based preconditioners

  • ML-enhanced multigrid, domain-decomposition, and deflation components include smoothers, subdomain solvers, transfer operators, and coarse spaces; the review treats these methods separately in Section 3.3.Source: MG-, DD-, and deflation-based… · p. 27

Green’s function-based preconditioners

  • Neural Green’s functions approximate inverse operators for preconditioning, addressing the high-dimensional domain and singular behavior that make large-scale numerical approximation challenging.The adaptive MSNN represents multiscale structure with branches for near-, middle-, and far-field behavior; its learned Green’s function yields SPAI or hierarchical-matrix preconditioners for GMRES, while dense constructions are limited to small systems unless used as a coarse-grid solver.Source: Green’s function-based preconditioners · pp. 28–29

Other preconditioners

These approaches use ML to predict or refine components of classical preconditioners, often preserving compatibility with standard Krylov methods. Their promise is balanced by training, generalization, loss-design, sparsity, and stability challenges.

Other preconditioners

  • POD snapshots build a reduced-space preconditioner, with a smoother preserving correction outside the reduced space and POD projections serving as transfer operators.The method can also be interpreted as a two-grid algebraic multigrid scheme.Source: Other preconditioners · p. 29
  • A CNN predicts diagonal block starts from a matrix’s sparsity pattern, supplying the partition used to construct a block-Jacobi preconditioner.The sparsity pattern is treated as an image and block detection as multi-label classification.Source: Other preconditioners · p. 29
  • A U-Net predicts leading eigenvectors and eigenvalues from the current state for a spectral preconditioner in variational data assimilation.QR enforces orthonormality, and a scaled sigmoid bounds the predicted eigenvalues.Source: Other preconditioners · p. 29

Discussion

  • ML-enhanced classical preconditioners can retain a linear operator, enabling use with standard preconditioned Krylov methods and facilitating spectral or condition-number analysis.Analysis is generally more difficult for nonlinear learned preconditioners and remains relatively limited.Source: Discussion · p. 30
  • Prescribed sparsity patterns control memory and computational costs but may restrict preconditioner quality, motivating adaptive pattern selection that balances approximation, fill-in, and cost.ML-predicted IC or ILU factors can be constructed in parallel, but triangular solves remain difficult to parallelize during application; learned SPAI is applied through sparse matrix-vector products.Source: Discussion · p. 30
  • Learning corrections to standard IC factors retains a classical baseline while reducing the learning model’s burden; similar corrections could target ILU or SPAI.Possible targets include their factors, entries, or sparsity patterns.Source: Discussion · p. 30
  • Training costs are most compelling to amortize over sufficiently similar sequences of linear systems; generalization across sizes, discretizations, and parameters remains challenging.Performance may depend on how closely test matrices resemble the training distribution.Source: Discussion · p. 30
  • For SPD systems, condition-number objectives connect to standard convergence bounds, but costly spectral quantities lead many methods to use proxies that may poorly reflect solver convergence.Designing loss functions that more directly target spectral properties remains a challenge.Source: Discussion · p. 30

3.2.4 Alternating hybrid iterative methods

Alternating hybrid iterative methods interleave classical iterations with ML-based iterative schemes, combining their different error-reduction strengths. The review covers HINTS, FNS, and other approaches, while identifying training, parameter-choice, and stability constraints.

3.2.4 Alternating hybrid iterative methods

  • Alternating HIMs pair classical stationary or Krylov methods with ML-preconditioned Richardson or Krylov iterations, addressing slow classical convergence and neural-network difficulty learning high-frequency features.One example alternates nc classical relaxation iterations with nm ML-preconditioned Richardson iterations.Source: §3.2.4 · p. 31

HINTS and related variants

  • HINTS alternates a DeepONet-preconditioned Richardson iteration with classical relaxation for parametric linear PDE systems, mapping the current residual and PDE parameters to an update direction.Alternation may produce nearly uniform convergence across eigenmodes; coarse-grid training can transfer to larger systems, though distribution shifts may degrade performance.Source: HINTS and related variants · p. 31
  • HINTS variants address geometry transfer through source-to-target fine-tuning or masked self-attention, which filters points outside the computational domain and supports transfer without additional fine-tuning.The geometry-aware model is trained on simple reference geometries.Source: HINTS and related variants · pp. 31–32
  • Other HINTS-related approaches include label-free FNO training with physics-informed loss, MIONet for multiple PDE inputs with theoretical convergence analysis, and DP/TB preconditioners combined with classical solvers.A related meta-solver applies DP or TB approaches to 3D Helmholtz scattering problems using a DeepONet trained on non-scattering problems.Source: HINTS and related variants · p. 32

FNS and related variants

  • FNS combines stationary iterations with a learned Fourier-space correction, interpreted as a linear ML-preconditioned Richardson method that approximates inverse action using Fourier modes and learned inverse eigenvalues.Later work adds a learnable transition operator, while another extension uses GNNs for unstructured grids.Source: FNS and related variants · p. 32

Other approaches

  • The review also describes alternating Krylov iterations with CNN corrections for Helmholtz problems and combinations of neural Green’s function preconditioners with damped Jacobi or geometric multigrid.The CNN refines the intermediate Krylov solution before subsequent Krylov iterations.Source: Other approaches · pp. 32–33
  • Update-based losses can yield small updates despite large residuals, so physics-based residual losses can improve performance; loss choice is crucial and approach-dependent.The study also considers unrolling and Anderson acceleration for alternating fixed-point schemes.Source: Other approaches · p. 32

Discussion

  • Alternating HIMs suit multi-query and large-scale settings where training costs can be amortized; ML inference overhead may offset benefits on small systems, and nc strongly affects convergence.Too few classical iterations may cause divergence, while increasing nc initially improves acceleration but eventually diminishes it; nonlinear learned preconditioners may also stagnate or produce later residual growth.Source: Discussion · p. 33

3.4 Automated solver selection

Automated solver selection uses matrix properties to choose iterative methods, preconditioners, or their combinations from a large configuration space. The review covers feature-based classification and online learning, and identifies extensions to more complex settings as future work.

3.4 Automated solver selection

  • Automated selection can avoid manual choices among many solver and preconditioner configurations, including linear solvers needed at nonlinear iterations.The review notes that the approach is non-intrusive and often does not require existing-code modifications.Source: §3.4 · p. 42, Discussion of automated solver selection · p. 45

3.4.1 Preliminaries

  • Feature-based workflows extract global numerical or local structural matrix features, then classify them into solver configurations; the feature paths can be used separately or fused.Numerical examples include rank and nonzero counts; CNNs or GNNs extract structural features from image- or graph-based representations.Source: §3.4.1 · pp. 42–43, Figure 10 · p. 43

Numerical features only

  • Cost-aware feature selection removes low-information or redundant features, evaluates inexpensive features first, and adds features until the target prediction accuracy is reached.Sun et al. extend this approach with genetic algorithms and information gain; tested classifiers include RF, XGBoost, and GBDT.Source: Numerical features only · p. 44

Structural features only

  • CNN image classification uses a three-channel representation of sparse matrices to predict a GMRES preconditioner.The channels encode normalized matrix size, local nonzero density, and average nonzero magnitude in regions.Source: Structural features only · pp. 44–45

Numerical and structural features

  • Combined-feature models fuse numerical and structural representations to select solver–preconditioner combinations, using GNN/FNN branches or multimodal learning.The GNN-based model’s training score jointly considers runtime and residual error.Source: Numerical and structural features · p. 45

3.4.3 Other approaches

  • Online methods adapt choices during sequences of systems: PCGBandit minimizes cumulative solution time, while another approach updates solver selection from observed performance.The latter uses Bayesian optimization with Gaussian processes or a heuristic approach.Source: §3.4.3 · p. 45

Discussion of automated solver selection

  • The review presents block preconditioners and meta-learning as directions for extending solver selection to more complex systems and algorithm-selection frameworks.For coupled systems, block-wise matrix features or coupling information could guide preconditioner choices.Source: Discussion of automated solver selection · p. 45

4 Machine learning-enhanced nonlinear iterative methods

Nonlinear iterative solvers can be sensitive to initialization, and Newton steps require costly linear solves. The review surveys ML methods for solver components and nonlinear preconditioning, while noting that these areas remain limited and problem-dependent.

4.1 Preliminaries

  • Newton convergence is sensitive to initialization and often locally guaranteed; each Newton step also requires a linear solve that can dominate computational cost.Inexact Newton methods reduce cost by allowing approximate solutions of the linearized systems.Source: §4.1 · p. 46

4.2.1 Initial guess learning

  • Hybrid initial-guess methods use learned approximate solutions to start classical nonlinear solvers, including Newton and Picard iterations, across PDEs and engineering problems.Approaches include supervised networks, Fourier neural operators, and unsupervised CNN training; Int-Deep combines a mesh-free DNN approximation with finite-element solver refinement [335].Source: §4.2.1 · p. 47

Discussion

  • Learning initial guesses is widely studied because initialization strongly affects nonlinear convergence, but useful inputs are problem-dependent and closeness to the exact solution may matter less than basin membership.Nonlinear iterations can stagnate even from a close initial guess, while a less accurate guess within the basin of attraction may converge rapidly.Source: §4.2.1 · p. 46, Discussion · p. 47

4.2.2 Update function learning

  • The Newton-informed neural operator learns the Newton update mapping using supervised update matching and a physics-informed loss enforcing the linearized equation [351].The trained model can generate multiple solutions from different initial guesses and discover solutions absent from its training data.Source: §4.2.2 · p. 48

Discussion

  • Update-function learning remains very limited beyond, though broader rules using previous iterates or residuals connect to learning-to-optimize methods [352].The review identifies these as a possible direction for extending learned iterative updates.Source: Discussion · p. 48

4.2.3 Parameter learning

  • Random-forest regression predicts relaxation parameters for nonlinear iterations from dimensionless numbers and simulation properties, with an online extension refining the model during simulation.A separate approach learns spatially varying pseudo-time steps from local information for finite-element problems.Source: §4.2.3 · p. 48

4.3 Nonlinear preconditioner learning

  • ML nonlinear preconditioners address unbalanced nonlinearities that can stall Newton iterations, including POD-based PINL and its online stagnation-phase extension SSIN.PINL uses snapshots from a training problem; SSIN constructs its preconditioner online from snapshots collected during stagnation.Source: §4.3 · pp. 48–49

Discussion

  • ML-enhanced nonlinear methods remain limited, especially for update learning and preconditioning, because stabilization, analysis, feature choice, and training-data generation are difficult and problem-dependent.Nonlinear left-preconditioning is relatively underexplored compared with right-preconditioning; extensions of alternating methods and subspace correction are suggested.Source: Discussion · p. 49

5 Related topics

Section 5 briefly surveys work adjacent to hybrid iterative methods, including ML-enhanced time integration, iterative-method-inspired ML, and ML-enhanced direct solvers. It closes by identifying open questions about preserving numerical safeguards, establishing theory, and evaluating practical performance.

5.1 Machine learning-enhanced time integration methods

  • ML-enhanced time integration learns solver corrections, time-stepping rules, or step sizes, with related approaches also using pseudo-time steps to aid nonlinear convergence.Examples include solver-in-the-loop correction, multiscale neural time stepping, coarse-grid scheme learning, neural-vector correction, and feature-based step-size selection.Source: §5.1 · p. 50

5.2.1 Training data sampling

  • Multilevel training-data strategies use coarse and fine datasets or grid samples to address spectral bias and reduce the cost of estimating neural-operator training gradients.MLD-PINNs switch among datasets with different collocation-point counts, while MLMC uses coarse grids for most variance and a few fine samples to refine gradient estimates.Source: §5.2.1 · pp. 50–51

5.2.2 Neural network architectures

  • Iterative-method-inspired neural architectures draw on domain decomposition, multigrid, residual updates, and recurrent subspace construction to encode locality, hierarchy, or repeated correction.Examples include cPINNs and FBPINNs, MgNet, and R2N2, which learns an information subspace and an update within it.Source: §5.2.2 · pp. 51–52

5.2.3 Training algorithms

  • Training algorithms borrow preconditioning, multigrid, and domain decomposition ideas to scale or improve optimization of neural networks.Preconditioned gradient updates encompass adaptive scaling and curvature-based methods; other approaches apply MGRIT across network layers or Schwarz-style methods to parameters or training data.Source: §5.2.3 · pp. 52–53

5.3 Machine learning-enhanced direct solvers for linear systems

  • ML-enhanced direct-solver studies mainly learn matrix reorderings or graph partitions that reduce fill-in in sparse factorization.Examples select among ordering strategies using matrix features or recursively refine separators for nested dissection.Source: §5.3 · p. 53

5.4 Discussion of related topics

  • For ML-enhanced time integration, a key open question is whether learned predictions preserve stability, error control, and long-time reliability when embedded in classical schemes.Evaluation should also examine robustness outside the training regime and interaction with classical adaptivity mechanisms.Source: §5.4 · p. 53
  • Connections between iterative methods and neural-network structure remain often heuristic because classical analysis assumptions may not hold for neural models.The paper calls for theory explaining when structural choices improve training, generalization, or convergence, including at large scale.Source: §5.4 · p. 53
  • ML-enhanced direct-solver research has focused mainly on fill-in, while practical factorization-time and memory evaluation and systematic comparisons with iterative methods remain lacking.The proposed comparisons include efficiency, memory use, robustness, and scalability across problem classes.Source: §5.4 · p. 53

6 Conclusions and future directions

The review finds that hybrid iterative methods combine ML components with classical solvers and that linear systems dominate the existing literature, leaving nonlinear methods less studied. It highlights complementary solver capabilities, potential efficiency gains and difficult problem classes, alongside challenges in generalization, evaluation, and real-world validation.

6 Conclusions and future directions

  • The review covers hybrid iterative methods for linear and nonlinear systems, finding that existing work concentrates on linear systems while nonlinear HIMs remain less studied.It also briefly surveys ML-enhanced time stepping, iterative-method-inspired ML, and ML-enhanced direct solvers.Source: §6 · pp. 53–54

Key observations from existing studies

  • HIMs augment classical solvers with ML components, aiming to improve efficiency while preserving classical methods’ structure, interpretability, robustness, and convergence guarantees whenever possible.Classical methods and ML can also complement one another: local methods often damp high-frequency error, while neural networks can represent global or low-frequency features.Source: Key observations from existing studies · p. 54

Main future directions

  • HIMs may reduce computational cost when savings across repeated or parametric solves outweigh data-generation, training, and inference costs.The review also points to combining ML approximations with classical correction, relaxation, projection, or Krylov mechanisms to retain numerical safeguards.Source: Main future directions · p. 54
  • Data-driven solver construction or selection may be especially useful for nonsymmetric, indefinite, saddle-point, or highly nonlinear systems where effective solver choices are less established.The review contrasts these settings with standard discretizations of the Poisson equation, for which classical methods are comparatively mature.Source: Main future directions · p. 54

Challenges and practical applicability

  • Generalization, justified training costs, structure-aware integration, common-setting benchmarks, and validation on complex real-world problems remain important challenges.Existing gains may depend on the problem, and systematic investigation is still needed for complicated geometries, heterogeneous or multiscale coefficients, large discretizations, and coupled multiphysics systems.Source: Challenges and practical applicability · pp. 54–55

A.1 Classical iterative solvers and matrix datasets

Appendix A.1 lists open-source packages for classical sparse iterative solvers and datasets of sparse matrices.

A.1 Classical iterative solvers and matrix datasets

A.2 ML-enhanced linear iterative methods

Appendix A.2–A.5 list open-source code for four areas of ML-enhanced linear iterative methods: core-component learning, preconditioner learning, subspace-correction enhancements, and automated solver selection.

A.2 ML-enhanced linear iterative methods

Related papers

Loading 2610.07211v1…