Source-linked AI summary
KAN or MLP: A Fairer Comparison
Runpeng Yu, Weihao Yu, Xinchao Wang
TL;DR
KAN and MLP comparisons have not consistently matched parameter counts or FLOPs, leaving their relative performance across tasks unclear. This paper compares them under matched budgets across several domains and finds that KAN leads only in symbolic formula representation, while MLP generally leads elsewhere.
Problem
Prior KAN–MLP comparisons used different parameter counts or FLOPs, motivating a fair comparison across task domains.
Source: §1 · p. 2Method
The paper compares KAN and MLP across task domains while controlling either their parameter counts or FLOPs.
Source: §2 · p. 3Results
KAN outperforms MLP only in symbolic formula representation, while MLP typically excels elsewhere; KAN also forgets more than MLP in fixed-iteration class-incremental learning.
Source: §1 · p. 2Takeaways & Limitations
B-spline activation makes MLP match or surpass KAN on symbolic formula tasks, but does not substantially improve MLP on other tasks.
Source: §1 · p. 2
Abstract
from arXiv · showhide
This paper does not introduce a novel method. Instead, it offers a fairer and more comprehensive comparison of KAN and MLP models across various tasks, including machine learning, computer vision, audio processing, natural language processing, and symbolic formula representation. Specifically, we control the number of parameters and FLOPs to compare the performance of KAN and MLP. Our main observation is that, except for symbolic formula representation tasks, MLP generally outperforms KAN. We also conduct ablation studies on KAN and find that its advantage in symbolic formula representation mainly stems from its B-spline activation function. When B-spline is applied to MLP, performance in symbolic formula representation significantly improves, surpassing or matching that of KAN. However, in other tasks where MLP already excels over KAN, B-spline does not substantially enhance MLP's performance. Furthermore, we find that KAN's forgetting issue is more severe than that of MLP in a standard class-incremental continual learning setting, which differs from the findings reported in the KAN paper. We hope these results provide insights for future research on KAN and other MLP alternatives. Project link: https://github.com/yu-rp/KANbeFair
1 Introduction
Under parameter- or FLOP-matched comparisons, KAN outperforms MLP only on symbolic formula representation, while MLP generally performs better across the other evaluated domains. The paper also attributes KAN’s symbolic-task advantage to B-spline activation and reports more severe forgetting for KAN in standard class-incremental learning.
1 Introduction
- The paper frames its comparison as addressing prior KAN–MLP experiments conducted with different parameter counts or FLOPs.It describes MLP as widely used but limited in interpretability and network-scale flexibility.Source: §1 · p. 2
- KAN outperforms MLP only on symbolic formula representation, while MLP typically excels in machine learning, computer vision, NLP, and audio processing.The comparison controls for either the same number of parameters or FLOPs.Source: §1 · p. 2
- Replacing MLP’s activation with B-spline makes its symbolic-formula performance match or surpass KAN, but does not substantially improve MLP on other tasks.The authors attribute KAN’s symbolic-formula advantage to its B-spline activation.Source: §1 · p. 2
- In a standard class-incremental continual-learning setting with fixed training iterations, KAN has more severe forgetting than MLP.This differs from the earlier comparison using translated one-dimensional functions.Source: §1 · p. 2
2 Formulation of KAN and MLP
The paper characterizes KAN’s spline branch as a fully connected operation with an elementwise nonlinear function before a linear transformation. It contrasts KAN’s learnable, input-specific spline activations and operation order with common MLPs, then compares the models under matched parameter counts or FLOPs.
2 Formulation of KAN and MLP
- KAN has a B-spline branch and a shortcut branch combining a nonlinear activation with a linear transformation.The spline function is elementwise, and its B-spline parameters are learned with the other network parameters.Source: §2 · p. 2
- KAN uses learnable spline activations that differ for each input element, whereas common MLP activations are generally fixed and shared across elements.The paper identifies activation-function variation as the primary factor distinguishing the models.Source: §2 · p. 3
- KAN applies a nonlinear transformation before a linear one, unlike the usual description of MLP as linear followed by nonlinear.The paper notes that an MLP can also be described equivalently with a linear tail after nonlinear-first fully connected layers.Source: §2 · p. 3
- The comparison controls either parameter count or FLOPs while evaluating KAN and MLP across multiple task domains.The authors hypothesize that activation-function differences make the models suitable for different tasks.Source: §2 · p. 3
3 Number of Parameters of KAN and MLP
This section identifies the learnable components counted for a KAN layer, gives the MLP layer’s parameter expression, and defines the layer-dimension notation.
3 Number of Parameters of KAN and MLP
- din and dout denote the neural-network layer’s input and output dimensions, respectively.Source: §3 · p. 3
- KAN’s learnable parameters include B-spline control points, shortcut weights, B-spline weights, and bias.The supplied passage introduces the total but does not provide its expression.Source: §3 · p. 3
- One MLP layer has (din × dout) + dout learnable parameters.The expression is given for the layer’s input and output dimensions.Source: §3 · p. 3
4 FLOPs of KAN and MLP
The section specifies how FLOPs are counted and compares the operation accounting for KAN and MLP layers. It also notes that the initial FLOPs term is zero when an MLP applies its nonlinear operation first.
4 FLOPs of KAN and MLP
- The evaluation counts arithmetic operations as one FLOP and Boolean operations as zero, and calculates FLOPs for one sample.It treats the order-0 De Boor–Cox operations as Boolean, unlike the official implementation’s conversion back to floating-point data.Source: §4 · p. 3
- The section presents FLOPs expressions for the KAN B-spline computation and for the combined KAN layer, including its shortcut and branch-merging operations.The supplied expressions are partial fragments rather than complete equations.Source: §4 · p. 3
- The section gives an MLP-layer FLOPs expression and a difference expression for KAN and MLP layers with the same input and output dimensions.If MLP applies the nonlinear operation first, the first term in the difference expression is zero.Source: §4 · pp. 3–4
5 Experiments
Across parameter- or FLOP-matched tasks, KAN performs better for symbolic formula representation, while MLP generally leads in machine learning, computer vision, NLP, and audio. Ablations link the symbolic-task advantage to B-spline activations, and the class-incremental vision experiment finds more severe forgetting with KAN.
5 Experiments
- Experiments compare KAN and MLP under matched parameter counts or FLOPs across machine learning, computer vision, NLP, audio, and symbolic formula representation.The reported experimental objective is to compare the models under the same number of parameters or FLOPs.Source: §5 · p. 4
5.1 Performance Comparison
- With LBFGS, MLP retains an advantage on MNIST, CIFAR-10, and Bean, while KAN retains an advantage on Income.The relative advantages are largely the same as with Adam.Source: §5.1 · p. 8, Figure 10 · p. 8
- KAN suits symbolic formula representation, while MLP has advantages in machine learning, computer vision, NLP, and audio processing.The paper’s overall conclusion distinguishes these task domains.Source: §5.2 · p. 8
5.2 Architecture Ablation
- Spline activations bring little improvement to MLP on computer vision, but boost its machine-learning performance to be comparable to KAN.The reported machine-learning results were on two datasets; spline placement relative to the linear transformation did not significantly affect performance.Source: §5.2 · pp. 8–9
- With B-spline activations, MLP matches or surpasses KAN on all symbolic formula representation tasks.The MLP hidden-layer width was set to 10 or 20 to balance the additional parameters and FLOPs introduced by B-spline.Source: Figure 14 · p. 9, §5.2 · p. 9
5.3 Continual Learning
- In class-incremental vision learning, KAN forgets faster than MLP, with its accuracy on the first two tasks dropping to zero after all three tasks.The three sequential MNIST tasks grouped digits 0–2, 3–5, and 6–9.Source: Table 1 · p. 10, §5.3 · p. 10
6 Related Works
KAN research explores alternative basis functions and integrations with established architectures, while this paper contributes a fairer, comprehensive KAN–MLP comparison rather than a KAN improvement.
6 Related Works
- KAN variants replace B-splines with Chebyshev polynomials, wavelets, Jacobi polynomials, or other orthogonal polynomials, imparting different properties.Source: §6 · p. 10
- This paper offers a fairer, comprehensive comparison of KAN and MLP rather than a new KAN improvement.Source: §6 · p. 11
- KAN has also been integrated with convolutional and vision-transformer networks, U-Net, GNN, and NeRF for varied applications.Source: §6 · p. 11
7 Conclusion
Under matched parameter or FLOP budgets, the paper finds KAN’s advantage limited to symbolic formula representation, while MLP outperforms it on other tested tasks. B-spline activations narrow that gap, but KAN shows more severe forgetting in standard class-incremental learning.
7 Conclusion
- KAN had an advantage only in symbolic formula representation, while MLP outperformed it on the other tested tasks.Source: §7 · p. 11
- Replacing MLP’s activation with learnable B-splines made it outperform or match KAN across the tested tasks.Source: §7 · p. 11
- Under standard class-incremental learning settings, KAN exhibited more severe forgetting than MLP.Source: §7 · p. 11
- The paper mathematically compared the models’ forward processes and identifies learnable B-spline activations as KAN’s distinguishing feature.Source: §7 · p. 11