Source-linked AI summary
Sparsely connected rank-inspired neural network
Yunqing Huang, Wu Su, Nianyu Yi, Peimeng Yin
TL;DR
Function approximation and PDE solving need accurate numerical methods, while RINN’s fully connected design is costly and does not ensure progressive basis-rank growth. SCRINN uses sparse connectivity and a revised orthogonality- and normalization-guided loss in a two-stage procedure. Experiments report comparable or higher accuracy than RINN-es at substantially lower training cost, often with a lower-rank basis.
Problem
RINN’s fully connected architecture can be costly, and its training does not guarantee progressive growth of the neural-basis matrix rank.
Source: §2.3 · p. 6Method
SCRINN uses sparse connectivity and an orthogonality- and normalization-guided loss to train neural bases before determining output weights through least squares.
Source: §1 · p. 2, §3.2.2 · p. 9Results
SCRINN achieves comparable or higher accuracy than RINN-es with substantially lower training cost across function approximation and varied PDEs, often using a lower-rank basis.
Source: §5 · p. 28Takeaways & Limitations
The experiments emphasize basis quality and effective utilization rather than simply increasing the number of basis functions.
Source: §5 · p. 28The loss constructs approximately orthonormal bases independently of the target, so it does not incorporate problem-specific features and enforces orthogonality only on finitely sampled points.
Source: §3.2.2 · p. 11
Abstract
from arXiv · showhide
We propose a Sparsely Connected Rank-Inspired Neural Network (SCRINN) for function approximation and the numerical solution of partial differential equations (PDEs). By introducing a structured sparse connectivity architecture and an orthogonality- and normalization-guided loss function, SCRINN constructs an approximately orthonormal neural basis while promoting progressive rank growth of its evaluation matrix and reducing the number of trainable parameters. A two-stage framework is developed, in which an approximately orthonormal neural basis is first constructed, and the output weights are subsequently determined using least-squares methods. Numerical experiments on function approximation and a variety of steady-state, time-dependent, and nonlinear PDEs demonstrate that SCRINN consistently achieves high approximation accuracy with substantially lower training costs. The code is publicly available on GitHub.
1. Introduction.
The paper develops SCRINN for function approximation and PDE computation, addressing the computational cost and rank-growth limitations of RINN with sparse connectivity and a revised loss. Its experiments report a favorable balance of accuracy, efficiency, and robustness across function-approximation problems and several PDE classes.
1. Introduction.
- Most neural-network approaches described rely on iterative parameter optimization, while ELM-type methods use least squares for output weights but remain sensitive to hidden-layer initialization.RINN uses covariance-based orthogonalization, but its fully connected architecture is costly and its performance is sensitive to a loss penalty parameter.Source: §1 · pp. 1–2
- SCRINN combines sparse hidden-layer connectivity with an orthogonality- and normalization-guided loss to reduce parameters and redundancy among neural basis functions.The revised mean-square-type normalization also reduces sensitivity to the normalization parameter and yields the RINN+ variant.Source: §1 · p. 2
- Across function approximation and steady-state, time-dependent, linear, and nonlinear PDEs, SCRINN achieves comparable or higher accuracy than RINN-es with substantially lower training cost, often using a lower-rank basis.The conclusion emphasizes basis quality and effective utilization rather than simply increasing the number of basis functions.Source: §1 · p. 2
- SCRINN first trains neural basis functions to promote orthogonality and normalization, then determines output weights using least squares.This two-stage procedure is similar to RINN’s framework.Source: §1 · p. 2
- The paper studies function approximation and numerical solution of linear and nonlinear PDEs.It situates these problems among established scientific-computing tasks and neural-network methods.Source: §2 · p. 2, §1 · p. 2
2. Governing problems and RINN.
The paper formulates function approximation and PDE solving as target approximation problems, then reviews RINN’s neural-basis construction and two-stage solution procedure. It motivates SCRINN by RINN’s computational, parameter-sensitivity, and rank-growth limitations.
2.1. Governing problems.
- Function approximation seeks an accurate approximation of a target function, while the PDE problems include time-dependent and elliptic cases with source and boundary data.The domain may be spatial or space–time; time-dependent cases additionally impose initial conditions, while elliptic cases are spatial boundary-value problems.Source: §2.1 · p. 3
2.2.1. Neural networks and neural basis functions.
- The outputs of the final hidden-layer neurons form neural basis functions, and the rank of their sampled evaluation matrix measures their linear independence.Higher rank indicates a richer basis, whereas rank deficiency indicates redundancy that may limit approximation capability.Source: §2.2.1 · p. 4
2.2. Overview of RINN.
- RINN uses a two-stage framework: it first optimizes hidden-layer parameters for approximately orthogonal neural basis functions, then determines output weights by least squares.For PDEs, the second stage approximates the solution at selected collocation points.Source: §2.2.2 · pp. 4, 6
2.2.2. RINN framework.
- RINN uses a sample covariance matrix to enforce discrete orthogonality and normalization by driving its off-diagonal entries toward zero and diagonal entries toward one.Its tunable ε weights the normalization term, and the second stage fixes the hidden-layer parameters while solving for output weights.Source: §2.2.2 · pp. 5–6
2.3. Limitations of RINN.
- RINN’s fully connected architecture is costly, its normalization penalty is problem-sensitive, and its basis-matrix rank is not guaranteed to grow during training.Rank may decrease, potentially degrading approximation capability; SCRINN is proposed to mitigate rank saturation through sparse connectivity and reduced parameter coupling.Source: §2.3 · p. 6, §3 · p. 6
3. SCRINN.
SCRINN combines structured hidden-layer sparsity with an orthogonality- and normalization-guided loss to construct neural bases. It then freezes the basis and determines output weights through least-squares procedures, selecting training epochs by downstream-task criteria.
3.1. Sparsely connected neural networks.
- Input and output layers remain fully connected, while equal-width intermediate layers use channel-wise sparse connectivity, equivalent to diagonal intermediate weight matrices.More general sparse architectures with varying layer widths and neighboring-neuron connections are left for future work.Source: §3 · p. 6, §3.1 · p. 7
3. SCRINN.
- Finite-element-inspired hidden-layer sparsity reduces parameter coupling and basis redundancy while promoting growth of the neural basis matrix rank.Source: §3.2 · p. 7
3.2.1. Stage I: Orthogonality and normalization of neural basis func-
- Stage I penalizes off-diagonal Gram-matrix entries for discrete orthogonality and drives diagonal entries toward one for discrete normalization.The discrete inner product directly approximates the L2 inner product.Source: §3.2.1 · pp. 8–9
3.2.2. Stage II: Least-squares problem for function fitting and PDE solving.
- Stage II freezes the trained basis and solves for output weights, using linear least squares for linear systems and nonlinear least squares for nonlinear PDEs.For nonlinear PDEs, the authors use an iterative Trust Region Reflective solver.Source: §3.2.2 · pp. 9–11
3.2.3. Optimal neural network approximation.
- SCRINN's stopping indicators typically reach minima within a few epochs, while RINN often requires substantially more; Stage I selection uses downstream-task error.RINN+ keeps RINN-es's architecture and parameter count unchanged, isolating the effect of the loss formulation.Source: §3.2.3 · pp. 12, 14
4. Numerical examples.
The numerical examples evaluate SCRINN on nonlinear function approximation and steady-state and time-dependent PDEs, examining accuracy, rank behavior, robustness, and computational cost. Reported comparisons show lower errors and training costs than RINN-es across time-dependent and nonlinear PDEs, alongside associations between sparse connectivity, progressive basis-rank growth, and reduced dependence on parameter initialization.
4.1. SCRINN for Function Approximation.
- For the Drop-Wave function, SCRINN's relative L2 error decreases from 1.3524 × 10−6 to 1.0479 × 10−10 as basis functions rise from 2000 to 6000, with 4.8 seconds training at 6000 versus RINN-es's 61.9 seconds.At 6000 basis functions, RINN-es's error rises to 8.5361×10−3, while SCRINN's effective rank increases from 1030 to 3444 across the stated basis-size range.Source: §4.1 · p. 15, Table 1 · p. 16
- For SCRINN on the Drop-Wave function, the proposed loss gives relative L2 errors 1.1415 × 10−9, 1.1976 × 10−9, and 1.1133 × 10−9 at ε = 0.01, 0.5, and 1.0, respectively.RINN-es with the original loss is strongly ε-sensitive, whereas RINN+ errors remain close to 2×10−8 across those values; ε = 1.0 is used subsequently.Source: §4.1 · pp. 15–16, Table 2 · p. 16
- For the Rosenbrock function, SCRINN's relative L2 error decreases from 1.7898 × 10−5 to 1.8341 × 10−10 as basis functions rise from 2000 to 6000, while rank grows from 2000 to 3991.ELM errors are close to 1 and RINN-es errors are on the order of 10−2 across the tested sizes.Source: §4.1 · p. 18, Table 3 · p. 19
- At 4000 and 6000 basis functions, SCRINN's relative L2 errors are 1.4124 × 10−3 and 3.6651 × 10−4, versus RINN-es's 1.3985 × 10−2 and 3.7126 × 10−3.At 6000 basis functions, training takes 18.2 seconds for SCRINN versus 126.3 seconds for RINN-es.Source: Table 4 · p. 19, §4.1 · p. 19
- For the Michalewicz function, SCRINN's relative L2 error drops from 1.7179 × 10−3 to 5.9261 × 10−5 as basis functions increase from 2000 to 6000.At 6000 basis functions, SCRINN trains in 4.4 seconds, versus 7.7 seconds for ELM and 19.6 seconds for RINN-es; both comparison models have errors at the level of 101–102.Source: §4.1 · p. 17, Table 5 · p. 20
4.2.1. SCRINN for steady-state PDEs.
- For the two-dimensional Poisson equation, SCRINN reports relative L2 error 8.0784×10−9 at rank 1597, versus RINN-es error 8.3797×10−3 at rank 4010 in the comparison.These reported errors differ by approximately six orders of magnitude; SCRINN training times are only a few seconds for all tested configurations.Source: Table 6 · p. 21, §4.2.1 · p. 21
- During Poisson-problem Stage I training, SCRINN rank generally rises as EMSE decreases and stays low, whereas RINN-es rank overall declines with larger EMSE variations.For this test, the authors attribute a more effective approximation space to SCRINN's sparse connectivity, local support, and near-orthogonality constraints.Source: Figure 7 · p. 22, §4.2.1 · p. 22
4.2.2. SCRINN for time-dependent PDEs.
- At k = 6, SCRINN’s relative L2 error was 3.25×10−11 versus RINN-es’s 3.65×10−7, with training times of 7.72 versus 13.00 seconds.At k = 12, SCRINN’s relative L2 error was 1.86 × 10−4 versus RINN-es’s 2.06 × 10−3, with training times of 4.78 versus 13.44 seconds.Source: Table 9 · p. 24, §4.2.2 · p. 24
- For the wave equation, SCRINN’s relative L2 error reached 3.1039 × 10−11 while its training time was reduced by more than a factor of three versus RINN-es.Source: Table 10 · p. 25, §4.2.2 · p. 25
- For the finest 3D Poisson configuration, SCRINN’s relative L2 error was 1.5351×10−9 in 20.10 seconds, versus RINN-es’s 2.8185 × 10−8 in 801.11 seconds.SCRINN’s training time across tested configurations ranged from 6.23 to 20.10 seconds, compared with 608.51 to 801.11 seconds for RINN-es.Source: Table 11 · p. 26, §4.2.2 · p. 26
- For 3D Poisson, widening SCRINN from 4000 to 6000 basis functions at 5832 interior points reduced relative L2 error from 8.8023 × 10−8 to 8.2511 × 10−9.With 6000 basis functions, increasing interior points from 5832 to 8000 reduced error to 1.5351 × 10−9; RINN-es error rose from 1.4759 × 10−8 to 2.8185 × 10−8 under the two finest configurations.Source: Table 11 · p. 26, §4.2.2 · p. 26
- In 3D Poisson training, both methods’ effective ranks stayed near full rank, but RINN-es’s lower residual still left a larger relative L2 error than SCRINN.Source: §4.2.2 · p. 26, Figure 12 · p. 27
4.2.3. SCRINN for nonlinear PDEs.
- For the nonlinear problem, basis-rank growth synchronized with decreasing EMSE, while sparse local support reduced initialization dependence by limiting basis-function correlation.SCRINN reduced error to a low level after the first optimization stage; local support was described as producing a well-conditioned, low-redundancy initial representation.Source: §4.2.3 · pp. 27–28, Figure 13 · p. 28
- For the nonlinear equation, SCRINN achieved substantially higher accuracy, over 70% lower basis rank, and shorter training time than RINN-es.Across the analytical solutions, the relative L2 error was also several orders of magnitude lower for SCRINN; the results are summarized in Table 12.Source: Table 12 · p. 28, §4.2.3 · p. 28
- SCRINN combines channel-wise sparse connectivity with a Gram-matrix orthogonality- and normalization-guided loss to sustain rank growth, reduce full-connectivity costs, and alleviate sensitivity to RINN’s ε.Source: §5 · p. 28
5. Conclusion.
Across function approximation and varied PDEs, SCRINN achieves comparable or higher accuracy at substantially lower training cost, often with a lower-rank basis. The proposed loss also improves robustness in the original RINN architecture, while future work targets more flexible sparse connectivity and harder, higher-dimensional PDEs.
5. Conclusion.
- Across function approximation and varied PDEs, SCRINN achieves comparable or higher accuracy than RINN-es at substantially lower training cost, often using a lower-rank neural basis.The results emphasize basis quality and effective utilization rather than simply increasing the number of basis functions.Source: §5 · p. 28
- Figure 14 shows the exact function, SCRINN approximation, and pointwise absolute error for nonlinear equation (4.13).Source: Figure 14 · p. 29
- The proposed loss improves the robustness of the original RINN architecture, yielding the RINN+ variant.Future work will focus on more flexible sparse connectivity and problem-adaptive basis construction for higher-dimensional and more challenging PDEs.Source: §5 · p. 29