Multilevel Training for Kolmogorov Arnold Networks

1 Introduction

• Neural-network training lacks the approximation hierarchies and complementary relaxation routines that enable algorithmic multilevel speedups in numerical PDE solvers.Existing machine-learning multigrid efforts generally focus on architecture design, parallelization, or modest convergence improvements rather than algorithmic speedups.Source: §1 · pp. 1–2
• KANs provide a structured alternative to MLPs through learned functions represented in a fixed basis, motivating training methods that exploit their spline parameterization.The paper targets a gap in practical algorithms and theoretical insights that preserve or improve KAN performance while exploiting this structure.Source: §1 · p. 2

1.1 Contributions

• The paper introduces a KAN–multichannel-MLP change of basis, analyzes its effect on gradient descent, and designs properly nested multilevel optimization with appropriate interpolation operators.These contributions connect architecture equivalence, optimization geometry, and multigrid-inspired training.Source: §1.1 · p. 2

1.2 The rest of the paper

• The resulting framework preserves coarse-model progress during interpolation and supplies level-complementary optimization dynamics unavailable to the equivalent multichannel MLP basis.The hierarchy is built from the spline structure, while the gradient analysis motivates complementary coarse and fine optimization.Source: §1.2 · p. 3
• Numerical experiments on functional regression and PINNs show significant accuracy and efficiency improvements, whereas the equivalent multichannel-MLP multilevel method provides effectively zero improvement over the coarse model.The MLP failure is attributed to fine-grid training prioritizing modes already captured by the coarse model.Source: §1.2 · p. 3

2 KANs and multichannel MLPs

• Spline-based KAN activations use trainable B-spline coefficients, whose compact support and approximation properties support the paper’s later multilevel construction.KANs use multilayer function composition, while each learned activation is represented in a finite spline basis over specified knots.Source: §2 · p. 3

2.2 Eigenstructure of A[r]

• For uniform knots, A[r] is a banded upper-triangular Toeplitz matrix corresponding, up to scaling and boundary treatment, to a forward finite-difference approximation of the rth derivative.The product (A[r])^T A[r] approximates the 2rth derivative operator.Source: §2.1 · p. 4, §2.2 · p. 5

2.1 Change of basis

• A structured matrix A[r] provides a linear change of basis between spline functions and a ReLU^(r−1) basis, yielding an equivalent multichannel MLP layer.The full-network weight map is expressed as A^T u = w, with block structure over layers, neurons, and spline knots.Source: §2.1 · pp. 4–5

2.2 Eigenstructure of A[r]

• The eigenvalues of (A[r])^T A[r] increase approximately like ℓ^(2r), ordering eigenmodes from smoothest to most oscillatory and strongly weighting high-frequency modes.For n = 10 knots, the most oscillatory modes are scaled more than 100× stronger for r = 1 and 10^6× stronger for r = 3.Source: §2.2 · p. 6

2.3 Computational cost

• The ReLU-based evaluation costs O(PQ(n + r)) operations versus O(PQ(nr + r^2)) for recursive B-splines, while applying the banded change of basis costs O(nr) per layer.The reformulation therefore gives a FLOP speedup by a factor equal to the spline degree, and measured wall-clock improvement grows with r.Source: §2.3 · pp. 6–8

Basis equivalence

• A linear change of basis makes spline-based KANs and ReLU-based multichannel MLPs equivalent as forward operators, while producing distinct training dynamics.The KAN representation uses spline parameters u, whereas the MLP representation uses weights w related through the basis transformation.Source: §3 · p. 8

Optimization geometry

• The change of basis transforms gradients and induces preconditioned descent, with the selected parameterization and gradient representation determining the resulting update.The induced geometry can be expressed through inner products involving AAT or (ATA)^-1, while mixed-basis updates may yield nonsymmetric preconditioning.Source: §3 · pp. 8–9
• The paper compares four update types across parameterizations and gradient representations, then focuses on spline-space iteration using either KAN-native or pulled-back MLP geometry.Spline-space iteration is chosen for interpretability, with the former option corresponding to the gradient descent that arises naturally in KANs.Source: §3 · p. 9, Table 1 · p. 9

Multilevel implications

• The induced preconditioning provides a framework for designing complementary optimization or relaxation across levels of the proposed multilevel hierarchy.Table 1 connects the update rules to the spectral analysis used to ensure that different levels optimize complementary components.Source: Table 1 · p. 9, §3 · p. 9

Optimization geometry

• Despite the basis-induced preconditioning, the neural tangent kernel analysis indicates that ReLU- and spline-basis optimization can use comparable learning rates while retaining significantly different training dynamics.The comparable learning-rate claim follows from a bound on the NTK spectral radius, whereas the distinct dynamics follow from the preconditioning structure.Source: §3 · pp. 8–9

4 Multilevel training of KANs

• Multilevel training for spline-based KANs exploits spline parameterization to provide geometric refinement, transfer operators, and complementary level-specific optimization.The framework uses structure that is absent from ordinary MLPs.Source: §4 · p. 10

4.1 General multilevel formulation

• A properly nested hierarchy makes coarse optimization equivalent to fine-model optimization restricted to an interpolated weight subspace, so interpolation preserves coarse progress.The fine operator exactly preserves the coarse operator's action under interpolated weights, and the corresponding losses match.Source: §4.1 · pp. 10–11

4.2 Transfer operators in KANs

• Geometric spline refinement creates a functionally identical KAN layer with more trainable parameters, enabling efficient hierarchy construction without evaluating every level at fine-model cost.Nested spline spaces admit analytic interpolation operators; the approach avoids the prohibitively costly least-squares refinement used for arbitrary grid sizes.Source: §4.1 · p. 10, §4.2 · pp. 11–12

4.3 Complementary relaxation

• Compactly supported spline coefficients localize gradient updates, allowing refined KANs to immediately use newly added oscillatory expressivity, unlike multichannel MLPs that favor smooth knot-space functions.The change-of-basis geometry can strongly weight smooth modes over oscillatory ones, making refinement ineffective unless level-wise optimization is complementary.Source: §4.3 · pp. 12–13

5.1 Regression

• Multilevel spline-basis KAN training achieves orders-of-magnitude better regression accuracy than coarse or fine KANs and a larger vanilla MLP under comparable work.Spline-basis training itself improves accuracy by 1–3 orders of magnitude over the ReLU basis, while ReLU multilevel training provides effectively no improvement over the coarse model.Source: §5.1 · pp. 13–14
• Fine KAN and MLP models largely stall under equal work, whereas refinement produces abrupt loss reductions and reaches accuracies that standalone fine models would require far more epochs to attain, if at all.The convergence histories show stagnation before refinement and rapid progress afterward.Source: §5.1 · p. 14, Figure 4 · p. 14

5.2 Physics-informed neural networks

• The PINN experiments evaluate KAN methodologies against traditional architectures while limiting specialized data modifications and training techniques, using residual-based attention as an included enhancement.Residual-based attention is applied to the volumetric loss term and improved convergence without obscuring interpretation of the results.Source: §5.2 · p. 14

5.2.1 2d Poisson

• For the 2d Poisson PINN, multilevel spline KAN and vanilla MLP achieve small loss and error, while the equivalent-capacity ReLU KAN stagnates slightly below O(1) relative ℓ2-error.The spline and ReLU KANs have the same approximation space, so the reported difference is attributed to training dynamics; the spline result is hypothesized to reflect progressively higher-frequency learning.Source: §5.2.1 · p. 15, Figure 6 · p. 15
• The 2d Poisson spline KAN reaches smaller relative error faster and with less noise than a comparable MLP, with sharp improvements immediately after knot refinements.Loss and error plateau before refinement and decrease rapidly afterward, indicating that the hierarchy exploits increased expressivity.Source: §5.2.1 · pp. 14–15

5.2.2 1d Burger’s

• For the 1d Burgers PINN, multilevel spline KAN training obtains 2–3 orders of magnitude better accuracy than standalone KANs or MLPs at comparable work.Even the coarsest KAN architectures outperform comparable MLPs in training loss; only the spline basis is reported for this problem.Source: §5.2.2 · pp. 15–16, Table 3 · p. 16

5.2.3 Allen-Cahn Equation

• For the Allen–Cahn PINN, spline-basis refinement accelerates training and reduces loss at every level, whereas ReLU KAN and similarly sized MLP models fail to capture meaningful solution structure.Spline refinement also lowers residual Fourier-mode energy while widening the active spectral support, consistent with targeted improvement of higher-frequency modes.Source: §5.2.3 · p. 16

6 Conclusion

• Spline-basis KANs are equivalent to multichannel MLPs with power ReLU activations, but the basis change alters optimization dynamics and gives splines localized support for sharp, low-regularity functions.The matrix A[r] acts as a preconditioner, while compact support enables feature localization.Source: §6 · p. 17
• Uniform knot refinement creates properly nested model hierarchies whose interpolation preserves coarse-level functional progress.This property prevents interpolated fine models from undoing earlier optimization.Source: §6 · p. 17
• Spline refinement improves accuracy while shifting residual energy toward lower Fourier modes and widening the support of active residual modes.Figure 8 compares solutions, residuals, spectra, and spectral cross-sections from coarsest to finest refinement levels.Source: Figure 8 · p. 18
• 2-3 orders of magnitude better accuracy is achieved than with standard training methods or comparable MLPs while maintaining computational efficiency.The improvement is reported consistently across the numerical experiments.Source: §6 · p. 18
• The framework supports the broader conclusion that multigrid methods can accelerate neural-network training when architectures provide appropriate structure and complementary level dynamics.Future work includes multilevel cycling and extensions to other architectures and coarsening or refinement strategies.Source: §6 · pp. 18–19

A Notational Shift for MLP Architectures

• An MLP is expressed as a composition of affine maps and nonlinear scalar activations across hidden layers.Source: A Notational Shift for MLP Architectures · p. 19
• The final layer omits the nonlinear activation, yielding the network output after the last hidden transformation.Source: A Notational Shift for MLP Architectures · p. 19
• The analysis combines intermediate rearranged expressions into single steps and post-multiplies layers by learnable weights.Biases are shifted using t^(ℓ) = -b^(ℓ) to match spline-literature conventions.Source: A Notational Shift for MLP Architectures · p. 19

B Proofs regarding change of basis matrix A[r]

• The change-of-basis proof begins from a ReLU identity and uses induction to establish the relationship for spline order r.The induction relies on cancellation of insufficiently differentiable terms at spline knots.Source: B Proofs regarding change of basis… · pp. 19–20
• The proof uses the ReLU polynomial identity (x−a)(x−b)^r = (x−b)^(r+1) + (b−a)(x−b)^r.Source: B Proofs regarding change of basis… · p. 19
• A[r] is defined recursively from A[r−1] for splines of successive orders, including a special treatment for the final row.Source: B Proofs regarding change of basis… · p. 19
• For r = 2, the matrix is A = I−2S+S^2, and the general matrix satisfies A[r] = h^(1−r)/(r−1)! (A[1])^r.Source: B Proofs regarding change of basis… · p. 20
• With uniform knot spacing, A[r] is upper triangular Toeplitz, with coefficients determined by the power representation.Source: B Proofs regarding change of basis… · pp. 20–21

C Neural tangent kernel and change of basis

• The NTK is the Jacobian Gram matrix describing how network outputs change with parameters during training.For a batch of k data points, NTK = JJT is a k×k Gram matrix.Source: C Neural tangent kernel and change of… · p. 25
• Under the weight transformation w = A^T u, the Jacobian changes from J to J A, so the NTK becomes J A A^T J^T.The reordered operator A is block diagonal with layer-specific change-of-basis blocks.Source: C Neural tangent kernel and change of… · p. 25
• The section states a theorem relating the spectral radius of the NTK under ReLU and spline bases for uniform knot spacing.Source: C Neural tangent kernel and change of… · p. 25
• The spectral analysis bounds the transformed NTK through the maximum singular value of the block-diagonal change-of-basis operator and Toeplitz generator functions.The largest eigenvalue is controlled through the corresponding singular-value bounds.Source: C Neural tangent kernel and change of… · p. 25

D Hyper-parameters and Specializations

• The experiments were run on Nvidia A100 GPUs across varied problem configurations and random seeds.Source: D Hyper-parameters and Specializations · p. 26

• 20,000 random uniform points sample the grid [0.0001, 0.9999]^2 for the function regression example.Source: §D.1 · p. 26
• The network output is normalized to [0, 1] by an affine transformation.Source: §D.1 · p. 26
• Training uses L-BFGS with learning rate 1.0, while five-run ensembles use seeds 1234 through 1238.Source: §D.1 · p. 26

• Residual-based attention scales each volumetric-loss point and gives larger errors heavier weighting in the gradient.Source: §D.2 · p. 26
• Multilevel KAN training linearly ramps the learning rate at initialization and level transitions from η0 to η over SLR steps.Source: §D.2 · p. 26

2d Poisson Hyper-Parameters

• The 2d Poisson setup uses 2,401 volume points, 200 boundary points, and 49 interface points, with a [2, 5, 1] KAN, order-3 splines, and coarse grid size 4.Source: §D.2 · p. 26
• The Poisson configuration also reports an MLP architecture of [1] and a separate 4,096-volume-point, 64-initial-condition-point setup.Source: §D.2 · pp. 26–27

1d Burger’s Hyper-Parameters

• The 1d Burger’s example varies random seeds by increasing each new run’s seed by 10.Source: §D.2 · p. 27

Allen-Cahn Hyper-Parameters

• The Allen-Cahn example uses seed 1234, Adam, and a ten-epoch linear schedule increasing the learning rate from 10^-4 to 10^-2.Source: §D.2 · p. 27

• Figure 9 presents the solution, log-scale residual, residual Fourier transform, and Fourier cross-sections across refinement levels.Rows run from the coarsest level at the top to the finest at the bottom, with a shared colorbar for the first three columns.Source: Figure 9 · p. 28