Source-linked AI summary
KAN: Kolmogorov-Arnold Networks
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, Max Tegmark
TL;DR
MLPs are widely used nonlinear approximators, but the paper identifies limitations in their interpretability and proposes KANs as an alternative. KANs place learnable spline functions on edges instead of fixed activations on nodes, and experiments report accuracy and interpretability improvements over MLPs on small-scale AI + Science tasks.
Problem
MLPs are foundational nonlinear-function approximators, but their interpretability is limited and their role in modern models motivates alternative architectures.
Method
KANs use learnable one-dimensional spline-parametrized functions on edges, replacing linear weights, while nodes sum incoming signals without nonlinearities.
Results
KANs improve accuracy and interpretability over MLPs on small-scale AI + Science tasks, including data fitting, PDE solving, and scientific-law discovery.
Takeaways & Limitations
KANs are presented as promising collaborators for scientists and as alternatives to MLPs when accuracy or interpretability matters.
Takeaways & Limitations
KANs are usually 10x slower than MLPs given the same number of parameters, and the paper frames its evidence as applying at least to small-scale AI + Science tasks.
Abstract
from arXiv · showhide
Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. We show that this seemingly simple change makes KANs outperform MLPs in terms of accuracy and interpretability. For accuracy, much smaller KANs can achieve comparable or better accuracy than much larger MLPs in data fitting and PDE solving. Theoretically and empirically, KANs possess faster neural scaling laws than MLPs. For interpretability, KANs can be intuitively visualized and can easily interact with human users. Through two examples in mathematics and physics, KANs are shown to be useful collaborators helping scientists (re)discover mathematical and physical laws. In summary, KANs are promising alternatives for MLPs, opening opportunities for further improving today's deep learning models which rely heavily on MLPs.
1 Introduction
KANs are proposed as an alternative to foundational MLPs by moving learnable activation functions from nodes to edges. The paper argues that this design can improve accuracy and interpretability, particularly for small-scale AI + Science tasks.
- MLPs are foundational nonlinear-function approximators but can consume most non-embedding parameters in transformers and are less interpretable than attention layers without post-analysis.
- KANs replace MLP node activations and linear weight matrices with learnable spline-parametrized functions on edges, while nodes only sum incoming signals.Their fully connected structure is retained, but each weight parameter becomes a learnable one-dimensional function.
- KANs combine spline accuracy and local adjustability with MLP-style compositional feature learning to address their respective weaknesses.Splines are effective for low-dimensional functions but suffer from the curse of dimensionality, whereas MLPs are less accurate in low dimensions.
- KANs can learn compositional structure and univariate functions well, outperforming MLPs by a large margin on the paper’s exponential- and sine-function example.
- Extensive numerical experiments evaluate KAN accuracy and interpretability over MLPs on small-scale AI + Science tasks, including data fitting, PDE solving, and scientific-law discovery.The paper describes applications in Poisson-equation solving, knot theory, and Anderson localization.
2 Kolmogorov–Arnold Networks (KAN)
KANs replace MLP linear weights with learnable univariate spline functions on edges, extending the Kolmogorov-Arnold representation into deeper and wider networks. The paper presents theoretical and empirical evidence for accuracy, scaling, grid-extension, generalization, and symbolic interpretability advantages.
- Approximation theory: Smooth Kolmogorov-Arnold representations yield approximation rates independent of input dimension, potentially overcoming the curse of dimensionality.The guarantee depends on the theorem’s assumptions and on identifying a smooth representation, possibly with extra layers or width.
- Scaling laws: Cubic-spline KANs have theoretical scaling exponent α = 4, and experiments show this bound can be achieved empirically, whereas MLPs may plateau at slower scaling.The paper notes that excessively high spline order can create oscillations and optimization difficulties.
- For accuracy: Grid Extension: Grid extension improves KAN accuracy by fine-graining spline grids without retraining the larger model from scratch.A smaller KAN can be trained first and then expanded through finer spline grids, unlike conventional MLP size scaling.
- For accuracy: Grid Extension: A [2, 1, 1] KAN achieves lower test loss than a [2, 5, 1] KAN on the synthetic task because the task is exactly representable by the smaller architecture.The smaller model also delays the interpolation threshold, and regularization plus pruning is proposed to discover minimal architectures.
- Simplification techniques: KANs support interpretability through node-level pruning and symbolic regression, recovering the formula 1.0e1.0y2+1.0sin(3.14x) in the example.Automatic pruning removes all hidden neurons except the last one, leaving a [2, 1, 1] KAN.
3 KANs are accurate
KANs achieve favorable accuracy–complexity trade-offs across function fitting and PDE solving, with stronger scaling behavior than MLPs. Their spline-based structure also supports compact representations and scientific interpretation, although results rely on assumptions and include practical limits.
- Scaling laws: KANs can almost saturate the fastest theoretical scaling law, α = 4, whereas MLPs scale slowly and plateau quickly.The theory predicts test RMSE scaling as ℓ ∝ N^-4 when a Kolmogorov-Arnold representation exists.
- Scaling laws: KANs display better scaling curves than MLPs across five toy examples, especially for the high-dimensional example.Both model families were trained with LBFGS while varying grid points and model sizes.
- Special functions: KANs can discover compact representations of special functions, including shapes smaller than human-constructed alternatives.The paper frames these representations as potentially revealing mathematical properties of the functions.
- Special functions: Across 15 special functions, KANs consistently achieve better Pareto frontiers than MLPs in the number-of-parameters versus RMSE plane.The reported comparison indicates lower training or test losses for KANs at the same parameter count.
- PDE solving: For PDE solving, KANs converge faster, reach lower L2 and H1 losses, and have steeper scaling laws than MLPs.A 2-layer width-10 KAN reaches 10^-7 MSE versus 10^-5 for a 4-layer width-100 MLP, using 10^2 versus 10^4 parameters.
- Continual learning: KANs’ local spline plasticity can avoid catastrophic forgetting by changing nearby coefficients while preserving far-away coefficients.The continual-learning demonstration is preliminary and uses an extremely simple example.
4 KANs are interpretable
KANs provide interpretable, interactive models that reveal compositional structure, learned univariate functions, and mathematical relations. Applications show KANs can help users (re)discover relations in mathematics and physics, though the unsupervised results remain preliminary and may depend on random seeds.
- KANs reveal compositional structures in symbolic formulas while learning the corresponding univariate functions.
- KANs expose familiar operations through learned functions, including quadratic activations for multiplication and logarithmic/exponential activations for division.
- Unsupervised learning: KAN visualizations can identify dependent-variable groups in unsupervised learning, including (x1, x2, x3) and (x4, x5) in a toy dataset.
- Limitations: The unsupervised results depend on random seeds, and the authors have not yet established a systematic way to discover a complete set of relations.
- Knot theory: KANs rediscovered several known knot-invariant relations, making mathematical discovery more intuitive through visualizations and automatic symbolic regression.
- Anderson localization: In physics, KANs extract mobility edges from quasiperiodic tight-binding data and support human hypothesis generation followed by rapid symbolic testing.
5 Related works
The related-work discussion positions KANs among prior Kolmogorov-Arnold networks, neural scaling-law theories, learnable activations, symbolic regression, PDE methods, mechanistic interpretability, and AI-for-mathematics applications. It emphasizes KANs’ spline-based learnable activations and the open question of whether their fast scaling extends to more complicated tasks.
- Kolmogorov-Arnold networks: Prior Kolmogorov-Arnold network studies often used depth-2 width-(2n + 1) architectures, with limited expressive power and limited use of back propagation.
- Neural scaling laws: KANs contribute to neural scaling-law research by showing that some high-dimensional functions can scale as 1D functions under smooth Kolmogorov-Arnold representations.
- Open questions: Whether the fast scaling observed on synthetic datasets extends to language modeling or other complicated tasks remains an open research question.
- Mechanistic interpretability: The paper relates KANs to intrinsically interpretable architectures, positioning them within active mechanistic interpretability rather than only post hoc analysis.
- Learnable activations and symbolic regression: KANs use B-splines to continuously learn activation functions before symbolic snapping, distinguishing them from methods with manually fixed activations.
- PDE solving and operator learning: KANs are presented as potential replacements for MLP components in PDE-solving and operator-learning systems, including PINNs, PINOs, Fourier neural operators, and DeepONet.
- AI for mathematics: In AI for mathematics, KANs extend prior machine-learning work on knot recognition, invariant prediction, and relation discovery.
6 Discussion
The discussion identifies limited mathematical understanding, underexplored algorithmic choices, and slow training as important KAN boundaries, while outlining efficiency and application directions.
- Mathematical aspects: KANs’ mathematical understanding remains very limited, because existing theory mainly covers the restricted depth-2 shape [n, 2n + 1, 1].The paper suggests deeper Kolmogorov-Arnold representations and a possible notion of Kolmogorov-Arnold depth as future directions.
- Algorithmic aspects: Accuracy: Multiple architecture and training choices remain underinvestigated, so alternative basis functions and adaptive grids could potentially improve accuracy.Suggested alternatives include radial basis functions, other local kernels, and adaptive grid strategies.
- Hybrid of KANs and MLPs: KANs and MLPs differ through activation placement on edges versus nodes and through learnable versus fixed activation functions.Preliminary LAN experiments are proposed to disentangle these two architectural changes.
- Algorithmic aspects: Spline locality supports adaptive, multilevel, and domain-dependent training designs intended to enhance KAN accuracy and efficiency.The discussion connects these ideas to multigrid and multiscale methods.
- Application aspects: The paper presents KANs as preliminary tools for science-related tasks and proposes applications to PDEs, density functional theory, transformers, and scientist–AI collaboration.KANs’ interpretable functions are framed as a way for humans to communicate with AI using the language of functions.
- Algorithmic aspects: Efficiency: KAN training is usually 10x slower than MLP training at the same parameter count, largely because distinct activation functions cannot leverage batch computation.Grouping activations into multiple shared-function “multi-head” groups is proposed as an efficiency compromise.
A KAN Functionalities
This section points readers to a table of common KAN functionalities.
- KAN Functionalities: Table 6 lists common functionalities that users may find useful.The supplied material identifies the table but does not enumerate its functionalities.
B.1 Architecture
LANs modify MLPs by making node activations learnable while retaining weight matrices, providing a simpler and more scalable alternative that appears less accurate and interpretable than KANs.
- Architecture: LANs retain MLP-like weight matrices but replace fixed node activations with learnable spline-parametrized activations.LANs are introduced to separate activation learnability from activation placement.
- Architecture: A LAN with width N, depth L, and grid size G has N^2L + NLG parameters, with NLG typically much smaller than N^2L.This produces little parameter overhead relative to an MLP and permits initialization from pretrained MLPs.
- Comparison of LAN and KAN: LANs are conceptually simpler than KANs because their only architectural change is making activation functions learnable.Unlike KANs, LANs keep activations on nodes.
- Comparison of LAN and KAN: LAN activation parameters scale as N, whereas KAN activation parameters scale as N^2 with model width N.The stated scaling comparison favors LANs as model width increases.
- Comparison of LAN and KAN: LANs seem less interpretable and less accurate than KANs, although they still seem more accurate than MLPs.The interpretability disadvantage is associated with retaining weight matrices.
B.2 LAN interpretability results
The LAN interpretability results suggest that learnable node activations alone do not provide KAN-like interpretability when weight matrices remain present.
- Interpretability results: LANs seem much less interpretable than KANs on the same synthetic examples, where KANs are described as perfectly interpretable.The comparison is presented as preliminary evidence from Figure B.2.
- Interpretability results: Retained weight matrices introduce degrees of freedom that leave LAN learnable activations too unconstrained for straightforward interpretation.The paper gives both matrix interpretability and excessive degrees of freedom as reasons.
- Interpretability results: The preliminary LAN results imply that removing linear weight matrices through edge-based learnable activations may be necessary for interpretability.This conclusion concerns the comparison between LANs and KANs rather than LANs alone.
B.3 Fitting Images (LAN)
LANs adapt implicit neural representations by making activation functions trainable, and Figure B.3 reports higher image-representation PSNR than the comparable MLP.
- Image representation: Implicit neural representations model images as 2D functions mapping pixel coordinates to values.SIREN fits this function with an MLP using periodic activations.
- Learnable activations: LANs replace fixed sine activations with trainable spline activations while retaining the SIREN setup.The LAN and MLP compared here use the same [2,128,128,128,128,128,1] shape.
- Image-representation result: LANs achieve higher PSNR than the MLP in Figure B.3, attributed to their flexibility to fine-tune activation functions.The experiment trains both models with Adam for 10,000 total steps using learning rates 10^-3 and 10^-4.
C Dependence on hyperparameters
Interpretability depends on regularization, initialization, grid resolution, and spline order. Entropy penalties and penalty strength shape sparsity, while random seeds can change pruning outcomes.
- Regularization: Entropy penalty reduces the number of active activation functions and prevents many duplicate functions.The interpretability goal is to keep the number of active functions as small as possible, ideally three.
- Optimization variability: Random seeds affect results, and an unlucky seed can produce a pruned network with different behavior.The passage identifies seed dependence as a source of variability in pruning outcomes.
- Regularization: Overall penalty strength λ effectively controls sparsity.This makes λ a direct control for the number of active components in the interpreted network.
- Grid resolution: When grid number G is too small, limited activation expressiveness encourages ensembling that makes interpretation harder.The grid therefore has a subtle effect on interpretability rather than a uniformly monotonic one.
- Spline order: Piecewise polynomial order k has only a subtle interpretability effect and shows no visible pattern in this toy example.Its behavior is described as somewhat like that of random seeds.
D Feynman KANs
The Feynman experiments visualize KAN solutions and compare their performance with MLPs across equation-fitting tasks. Uniform-grid training can produce a sharp loss increase when the grid becomes large.
- Feynman evaluation: Figure D.1 compares KAN and MLP Pareto frontiers for each Feynman dataset.Figures D.2 and D.3 separately visualize best and minimal KANs for each equation-fitting task.
- Feynman evaluation: Minimal KANs are selected under the constraint test RMSE < 10^-2, whereas best KANs have the lowest test RMSE loss.These criteria distinguish compact feasible solutions from lowest-loss solutions.
- Limitation: Uniform-grid training can show a sudden training-loss increase when the grid size reaches a large level comparable to points in one spatial direction.The passage attributes this possible behavior to higher-dimensional B-spline implementation and calls for further investigation.
F KANs for special functions
The special-function results provide visualizations of minimal and best KANs for each fitting task. Minimal models satisfy a test-error constraint, while best models minimize test loss.
- Special-function results: Figures F.1 and F.2 visualize KANs learned for special-function fitting tasks.The figures show best and minimal models, respectively.
- Minimal KANs: Minimal special KANs are selected under the constraint test RMSE < 10^-2.This criterion identifies compact models with reasonable test loss.
- Best KANs: Best special KANs are selected by lowest test RMSE loss.Figure F.1 presents these best models, while Figure F.2 presents the minimal models.