RINS: Residual-Image Neural Subspace Solvers for Large Sparse Linear Systems

1 INTRODUCTION

• Classical sparse-system solvers rely on useful correction directions, but their effectiveness is problem-dependent and often requires substantial algorithmic design.The passage situates this challenge in PDE simulation, inverse modeling, and scientific computing workflows.Source: §1 · p. 1
• Residual reduction depends on the distance from the current residual to range(AQ_t), and a neural-generated image space can yield a smaller projected residual than an Arnoldi-generated one.The opportunity is to enlarge the admissible family of expansion directions without increasing iteration count.Source: §1 · p. 1, Figure 1 · p. 2
• Neural-iterative solvers retain projected numerical closure while learning search directions, but graph message passing is costly and may favor early residual states.The passage notes that this can leave later-stage difficult modes less effectively served.Source: §1 · p. 2
• RINS uses residual-image polynomial probes, learnable combinations, and pointwise gating to add directions while tying dominant operator-probe cost to probe order L.This balances a richer control family against evaluation cost.Source: §1 · pp. 2–3
• A hybrid schedule uses GRANS for the first neural subspace block, then switches to RINS under the same projected minimum-residual update.The schedule combines graph attention's initial fitting capacity with cheaper polynomial probes tied to the operator.Source: §1 · p. 2
• Experiments report that RINS reaches fixed residual thresholds faster than GMRES and a graph-only neural baseline in most settings, with further gains from hybrid schedules.The comparison is in synchronized wall-clock time.Source: §1 · p. 3

2.1 SCIENTIFIC MACHINE LEARNING FOR PDES

• PDE neural surrogates learn fields or solution maps, whereas RINS learns a residual-conditioned basis policy within projected linear-algebra closure.The passage characterizes RINS as complementary to physics-informed and operator-learning approaches rather than an end-to-end replacement for the PDE solver.Source: §2.1 · p. 3

2.2 NEURAL-ITERATIVE SOLVERS

• G-RANS generates residual-aware directions through graph attention, while RINS uses polynomial matrix actions and cached sparse matrix-vector products to emit multiple directions per order-L call.The passage identifies this generator distinction as central to runtime and subspace-quality comparisons.Source: §2.2 · p. 3

3.1 EXTENDING THE SUBSPACE EXPANDING DYNAMICS

• GMRES searches a Krylov space, while neural controllers generate blocks whose orthogonalized increments expand the search space toward one with a smaller projected residual.Residual reduction is measured by projection onto the residual-image space, range(AQ_t), rather than by the search basis alone.Source: §3.1 · p. 4

3.2 RINS POLYNOMIAL BASIS GENERATOR

• RINS builds candidate correction directions from cached polynomial probes of the current residual, requiring L sparse matrix-vector products to form an order-L probe dictionary.Static-RINS selects emitted columns through learnable polynomial filters over the dictionary.Source: §3.2 · p. 4
• Gate-RINS applies a residual- and coordinate-dependent pointwise MLP to modulate polynomial probes, expanding the admissible block beyond a fixed polynomial span.The gate adds pointwise computation and diagonal modulation without increasing the dominant SpMV count for the probe dictionary.Source: §3.2 · p. 5

3.3 HYBRID MODELS BOOTSTRAP TRAINING AND INFERENCE

• Hybrid training freezes previously trained controllers and trains the next controller on the residual induced by their prefix; short unrolled horizons detach bases and residuals between steps for stability.The next controller's objective is to improve the next residual-image projector.Source: §3.3 · p. 5
• Hybrid schedules assign controllers by solver stage, using a GRANS block first and a shared Gate-RINS controller thereafter while retaining the same projected residual-image closure.Hybridization changes the control sequence, not the numerical closure that evaluates each generated search space.Source: §3.3 · pp. 5–6

3.4 THEORETICAL ROLE OF FIXED-STEP GMRES EMULATION

• Under compact-family, continuity, and constant-rank assumptions, a continuous neural generator with one correction direction can uniformly match m-step GMRES residual quality to within any ε.The passage frames this as an expressivity statement for parameterized nonsingular systems.Source: §3.4 · p. 6

4.1 MAIN BENCHMARK

• The benchmark covers six PDE tasks at n ≈1000 and n ≈10000, with 200 neural iterations generating 20 directions each and no preconditioner.GMRES uses CPU-optimized implementations at these scales; residual curves summarize 10 held-out samples, while timing statistics pool valid seed/sample observations.Source: §4.1 · p. 6, Figure 2 · p. 7
• At n ≈1000, RINS reduces residuals faster than GRANS and GMRES across the benchmark systems; at n ≈10000, it reaches threshold earlier than GMRES on all six tasks and than GRANS on most.Even when Static-RINS generates one direction per iteration, it reaches the target residual faster than GMRES on the reported n ≈1000 tasks.Source: §4.1 · pp. 6–8, Table 1 · p. 7

4.2 LARGE SCALE COMPARISON

• At n ≈10^5, Figure 3a compares Gate-RINS, GRANS, Block-GRANS, and GPU GMRES on Helmholtz using synchronized H100 wall-clock time.The comparison includes basis generation, orthogonalization, and projected least squares in the measured solve time.Source: §4.2 · p. 8, Figure 3 · p. 8

4.3 TRAJECTORY FOR DIFFERENT MODELS

• In a shared PCA projection, Gate-RINS follows a path close to GMRES, while GRANS and Block-GRANS follow visibly different correction trajectories.The paper attributes the distinct learned-solver paths to their basis-generation policies; Gate-RINS can add a richer block of directions at each neural step.Source: §4.3 · p. 8

4.4 DIFFICULT-MODE SPECTRAL DIAGNOSTICS

• After the first few iterations, Gate-RINS maintains stronger overlap with difficult Poisson modes, and this pattern motivates switching from an initial graph correction to Gate-RINS.The diagnostic uses the eight eigenvectors with lowest eigenvalue magnitudes and reports overlap statistics across held-out samples.Source: §4.4 · pp. 8–9

4.5 HYBRID INFERENCE ACCELERATION

• On Poisson systems, hybrid schedules use a frozen GRANS first block followed by Gate-RINS or Static-RINS, retaining the same projected least-squares closure.Because only the first step uses graph message passing, the remaining steps retain the RINS cost profile while benefiting from a better initialized subspace trajectory.Source: Figure 4 · p. 9, §4.5 · p. 9

• Gate-RINS combines polynomial residual-image probes with a lightweight pointwise gate while preserving the projected minimum-residual update.The gate increases expansion flexibility without increasing the probe dictionary’s dominant SpMV cost.Source: §5 · p. 9
• Lower time-to-threshold is associated with better alignment between range(AQ_t) and the residual, especially its difficult spectral components.Although one RINS neural block iteration is more expensive than one GMRES Arnoldi step, the generator typically achieves greater residual reduction per block, leading to lower end-to-end runtime in the evaluated systems.Source: §5 · pp. 9–10
• Future work will pursue preconditioner-aware adaptation for difficult tasks, including high-wavenumber Helmholtz systems, and extend RINS toward matrix-free deployment.The authors also identify new neural-solver variants and multiphysics systems as future directions.Source: §5 · p. 10

• For every ε > 0, a continuous generator can produce a one-dimensional projected update whose residual is at most ε worse than m-step GMRES, uniformly over the family.The result assumes a compact parameter set, continuous system data, uniform nonsingularity, and constant rank of the m-step Krylov action matrix.Source: A UNIFORM EMULATION OF FIXED-STEP GMRES · p. 13
• Constant rank makes the GMRES correction continuous over the compact system-data set because the action matrix pseudoinverse is continuous.The action matrix is built from the residual and its successive operator images.Source: A UNIFORM EMULATION OF FIXED-STEP GMRES · p. 13
• A continuous neural network can uniformly approximate the continuous GMRES correction map on the compact data set.The proof invokes the universal approximation theorem.Source: A UNIFORM EMULATION OF FIXED-STEP GMRES · p. 13
• The projected one-dimensional update can choose α = 1, translating correction-vector approximation into a residual bound with operator-norm error.The proof uses this admissible scalar choice to show the projected residual is no worse than the approximated correction’s residual.Source: A UNIFORM EMULATION OF FIXED-STEP GMRES · pp. 13–14

• All benchmark systems use continuous piecewise-linear triangular finite elements on a randomized quadrilateral domain, with no preconditioner.The stored systems are homogeneous-boundary correction systems formed using lifting and identity rows on boundary degrees of freedom.Source: B EXPERIMENTAL DETAILS AND BENCHMARK… · p. 14
• Synchronized CUDA timing barriers make the reported neural-solver times conservative relative to asynchronous deployment.The protocol fixes CPU linear-algebra threading and measures wall-clock intervals around explicitly synchronized GPU work.Source: B EXPERIMENTAL DETAILS AND BENCHMARK… · p. 14
• Scalar parameters vary by relative variation ρ = 0.3 around baseline values unless a smaller geometric perturbation range is specified.For the common coefficient/source pattern, κin is sampled from [0.77, 1.43], κout from [0.70, 1.30], sf from [0.07, 0.13], and cg from [0.70, 1.30].Source: B EXPERIMENTAL DETAILS AND BENCHMARK… · p. 14
• Table 2 identifies the PDE-derived benchmark systems, while Tables 3 and 4 report actual mesh-size distributions at nominal n ≈1000 and n ≈10000.The table captions specify mean±standard deviation and observed min–max ranges for the mesh-size distributions.Source: Table 2 · p. 15, Table 3 · p. 15, Table 4 · p. 15

B.1 POISSON

• Poisson uses a P1 finite-element stiffness system with the common diffusion/source inclusion and task-specific perturbations to inclusion radius and center.Local stiffness entries use κ-weighted basis-function gradient products, and the source load uses triangle-centroid values.Source: §B.1 · p. 15

B.2 HELMHOLTZ

• Helmholtz adds a positive reaction term to the common diffusion/source pattern, yielding K + Mα rather than an indefinite high-frequency Helmholtz system.The reaction parameters are sampled as αin ∈[1.40, 2.60] and αout ∈[0.70, 1.30].Source: §B.2 · p. 15

B.3 HEAT

• Heat represents one backward-Euler implicit step, with correction matrix A = M + ∆tK and right-hand side Mun + ∆tb.The previous state is made consistent with the prescribed bottom boundary before forming the step.Source: §B.3 · p. 16

B.4 ADVECTION–DIFFUSION

• Advection–diffusion adds the first-order transport field v(x, y) = β(−y, x), producing the nonsymmetric correction matrix K + C.This is the only main benchmark branch whose nonsymmetry comes from a first-order differential operator.Source: §B.4 · p. 16

B.5 KLEIN–GORDON

• Klein–Gordon represents one implicit step of a second-order equation, with correction matrix ∆t−2M + c2K + Mm.Its right-hand side includes f n+1 + ∆t−2M(2un −un−1).Source: §B.5 · p. 16

B.6 WAVE

• The wave task models one implicit step with correction matrix Δt^-2M + c^2K and right-hand side f_n+1 + Δt^-2M(2u_n − u_n−1).Wave speed c ranges from 0.70 to 1.30 and time step Δt from 0.35 to 0.65.Source: §B.6 · p. 16
• Each PDE and scale uses 200 training samples and 10 held-out samples, with randomized geometry and triangular refinement varying the realized node counts.The nominal scales are n ≈1000 and n ≈10000.Source: §B.6 · p. 16
• The benchmark trains RINS and GRANS for 100, 100, and 150 epochs across three stages and evaluates both neural solvers for 200 iterations.Timing uses one CPU thread for OMP and MKL; GPU measurements synchronize around each solver step.Source: §B.6 · p. 16

C LARGE-SCALE n ≈10000 RESULTS

• At n ≈10000, 200 neural iterations yield at most 4000 directions, keeping the accumulated basis substantially smaller than the ambient dimension.This makes the setting a more stringent test of learned subspace expansion than the n ≈1000 setting.Source: C LARGE-SCALE n ≈10000 RESULTS · p. 17
• Table 5 reports pooled mean and standard deviation times to relative residual 10^-5 at n ≈10000, alongside RINS speedups over GRANS and GMRES.Figure 5 presents the corresponding main benchmark relative-residual curves.Source: Table 5 · p. 17, Figure 5 · p. 17

D FROM GRANS TO BLOCK-GRANS

• Block-GRANS retains GRANS’s input construction, edge attention, basis heads, concatenation, and QR step, but splits source feature channels into contiguous blocks.Its source projection is feature-block-diagonal; the large-scale summary uses the source-block-projected variant.Source: D FROM GRANS TO BLOCK-GRANS · p. 18
• Block-GRANS is a structured GRANS subfamily: increasing the number of projection blocks restricts cross-channel mixing, while its generated directions still enter the same residual-image least-squares projection.The graph basis generator changes, not the algebraic update.Source: D FROM GRANS TO BLOCK-GRANS · p. 18

E PRECONDITIONING INTEGRATION EXPERIMENTS

• Preconditioned RINS and GRANS use 20 directions per iteration and the same 50/50/100 training schedule on 200 training systems per PDE.Inference evaluates the methods on the same 10 held-out systems.Source: E PRECONDITIONING INTEGRATION… · p. 18
• GMRES stops when its relative residual reaches the better final residual from RINS and GRANS for that PDE, or when its iteration budget is exhausted.Source: E PRECONDITIONING INTEGRATION… · p. 18
• All three methods are evaluated on the same left-Jacobi-preconditioned systems, with Figure 6 plotting relative residual against cumulative inner-solve time.The experiment covers six PDE tasks at n ≈1000.Source: E PRECONDITIONING INTEGRATION… · p. 18, Figure 6 · p. 19

F ONE-BASIS LIMIT TEST

• The test uses Static-RINS with one filter and polynomial order four, while retaining the main benchmark’s samples and three-stage training schedule.Source: F ONE-BASIS LIMIT TEST · p. 19
• The limit test gives Static-RINS and GRANS one new direction per iteration, so the accumulated neural subspace grows at the same rate as unrestarted GMRES.Source: F ONE-BASIS LIMIT TEST · p. 20
• Static-RINS decreases residual faster than GRANS across the six tasks and often reaches the target before its accumulated subspace reaches 100 dimensions.Under the one-basis restriction, Static-RINS also reaches the target with fewer iterations than GMRES, while GRANS fails to converge in these runs.Source: Figure 7 · p. 19, F ONE-BASIS LIMIT TEST · p. 20