Source-linked AI summary
Kolmogorov-Arnold Networks are Radial Basis Function Networks
Ziyao Li
TL;DR
KANs use B-splines to represent complex functions, but spline-basis computation and grid rescaling create efficiency bottlenecks. This paper introduces FastKAN, which replaces 3-order B-splines with Gaussian RBFs and uses layer normalization; it reports faster calculation while retaining comparable accuracy. The paper concludes that KANs are RBF networks with fixed centers.
Problem
KANs use B-splines to approximate smooth univariate functions, but deBoor-Cox basis computation and grid rescaling can create severe efficiency bottlenecks.
Method
FastKAN approximates 3-order B-spline bases with Gaussian-kernel RBFs and uses layer normalization to prevent inputs from leaving the RBF domain.
Results
FastKAN accelerates efficient_kan's forward speed by 3.33 times and is equivalent to, if not better than, KAN on MNIST.
Takeaways & Limitations
The paper concludes that KANs can be viewed as RBF networks with fixed centers.
Abstract
from arXiv · showhide
This short paper is a fast proof-of-concept that the 3-order B-splines used in Kolmogorov-Arnold Networks (KANs) can be well approximated by Gaussian radial basis functions. Doing so leads to FastKAN, a much faster implementation of KAN which is also a radial basis function (RBF) network.
1 Kolmogorov-Arnold Networks
KANs represent complex multivariate functions by superposing learnable univariate functions, using B-splines to approximate smooth components. Their B-spline computation and grid rescaling can create severe efficiency bottlenecks.
- KANs decompose complex multivariate functions into a superposition of simpler, learnable univariate functions.
- B-splines provide a theoretically grounded basis for approximating smooth univariate functions within a given domain.
- During training, variable shifts can move inputs outside the spline domain, requiring additional rescaling of spline grids.
- Computing B-spline bases with deBoor-Cox iteration and rescaling grids can create severe efficiency bottlenecks.
2 FastKAN
FastKAN accelerates KANs by replacing 3-order B-spline bases with Gaussian-kernel RBFs and using layer normalization to keep inputs within the RBF domain, without loss of accuracy.
- FastKAN approximates 3-order B-spline bases with Gaussian-kernel radial basis functions to simplify and accelerate KAN computation.
- Layer normalization is used to prevent inputs from shifting outside the domain of the radial basis functions.
- The resulting FastKAN implementation is substantially simpler and accelerates model calculation without loss of accuracy.
3 Gaussian Radial Basis Functions
RBF networks combine radially symmetric functions centered at different points, with Gaussian functions as a common choice. The paper argues that linear transformations align 3-order B-spline bases with Gaussian RBFs, enabling their replacement.
- Gaussian Radial Basis Functions: RBFs are real-valued functions whose outputs depend only on distance from a center point.
- Gaussian Radial Basis Functions: An RBF network models functions by linearly combining radially symmetric functions centered at different input-space locations.
- Gaussian Radial Basis Functions: The network weights the radial basis functions with adjustable coefficients, while each basis function depends on distance between the input and a center.
- Gaussian Radial Basis Functions: In the Gaussian RBF, r denotes radial distance and h controls the function's width or spread.
- Gaussian Radial Basis Functions: Linear transformations align series of 3-order B-spline bases with Gaussian radial bases, supporting replacement of B-spline calculations.
4 Result
FastKAN substantially improves KAN computation speed while maintaining comparable MNIST accuracy. In the reported benchmark, it is 3.33 times faster in forward computation and equivalent to or better than KAN in accuracy.
- 3.33 times: FastKAN accelerates efficient_kan forward computation in the reported benchmark.The benchmark uses 100-input and 100-output layers on NVIDIA V100 GPUs, with timing over repeated forward calculations.
- FastKAN is equivalent to, or better than, KAN on MNIST validation accuracy.Both models use the [28 × 28, 64, 10] architecture and are trained for 20 epochs.
5 Discussion
The paper concludes that replacing B-splines with Gaussian RBFs accelerates KANs and supports viewing KANs as RBF networks with fixed centers.
- FastKAN accelerates KAN by replacing B-splines with Gaussian RBFs.
- KANs can be understood as RBF networks with fixed centers.