Source-linked AI summary
Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks
Tilen Cadez, Sanghoon Lee, Kyoung-Min Kim
TL;DR
KANs offer learnable, interpretable edge functions, but their dataset-size scaling and how those functions adapt as data grows remain open questions. The paper studies three KAN variants across image classification and magnetic regression, finding task- and architecture-dependent scaling regimes and examining changes in trained edge functions.
Problem
How KAN performance scales with dataset size and how its learnable edge functions adapt remain less explored than scaling with model size.
Source: §I · pp. 1–2Method
The study measures test-loss scaling for three KAN variants on two image-classification benchmarks and magnetic-parameter regression, and examines trained edge-function evolution.
Source: §I · p. 2Results
Image-classification losses pass through random-guess, faster-scaling, and slower-scaling regimes, while magnetic regression transitions from random guess to a single linear-scaling regime.
Source: §1 · p. 3, §III · p. 9Takeaways & Limitations
KAN scaling depends on implementation and task, supporting case-by-case analysis and selection of variants suited to domain-specific data characteristics.
Source: §III · pp. 10–11
Abstract
from arXiv · showhide
Kolmogorov-Arnold Networks (KANs) represent a compelling alternative to traditional Multi-Layer Perceptron (MLP)-based neural networks. By employing activation functions as learnable elements, KANs offer superior interpretability, making them suited for scientific domains. In this work, we investigate the neural scaling laws of KANs and the structural evolution of their learnable activation functions under dataset expansion. Specifically, we evaluate the scaling behavior of three KAN variants---BSRBF-KAN, Gottlieb-KAN, and Faster-KAN---across standard image classification benchmarks (MNIST and Fashion-MNIST) and a specialized scientific regression task (magnetic parameter estimation from domain images of moir\'e magnetic textures). Our results demonstrate that the test loss ${\cal L}$ exhibits a broken neural scaling law (BNSL) behavior as a function of the dataset size $N_D$. After passing through a random-guess regime, the loss follows architecture- and task-dependent scaling behavior. The loss crosses from a faster- to a slower-scaling branch, ${\cal L}\propto N_D^{-\alpha}$ and ${\cal L}\propto N_D^{-\beta}$ with $\alpha>\beta$ for image classification tasks. The exponents $\alpha$ and $\beta$ depend strongly on both the specific network architecture and the dataset-size regime, ranging from 0.4 to 1.5 and from 0.06 to 0.6, respectively. For the magnetic parameter-regression task, the loss follows a single scaling law with its exponent ranging from 1.28 to 2.59. Additionally, we provide a structural analysis of how activation functions refine their complexity as data volume increases, finding that dataset expansion drives a transition from simple linear-like approximations toward stable, interpretable symbolic forms. These findings provide a quantitative roadmap for the efficient application of KANs while managing the trade-off between model expressivity and computational overhead.
I. INTRODUCTION
KANs place learnable activation functions on edges, offering interpretability alongside expressivity, but their computational costs make efficient deployment a concern. This study examines how their performance and edge functions change with dataset size across image-classification and magnetic-regression tasks.
I. INTRODUCTION
- KANs replace fixed nodal activations with learnable univariate edge functions that can be converted into interpretable symbolic forms.The paper presents this structure as offering expressivity and comparable accuracy with fewer parameters than MLPs.Source: §I · p. 1
- Evaluating basis functions on every edge complicates GPU parallelization, requires substantial memory, and can make KANs an order of magnitude slower than MLPs.KANs may also require hybrid optimization or evolutionary frameworks to handle their more intricate loss landscapes.Source: §I · p. 1
- Dataset-size scaling remains less explored than model-size scaling in KANs, despite its relevance to balancing expressivity and computational resources.The original KAN study examined scaling primarily through spline grid size.Source: §I · pp. 1–2
- The study evaluates BSRBF-KAN, Gottlieb-KAN, and Faster-KAN on MNIST, Fashion-MNIST, and magnetic-parameter regression, finding task-dependent scaling and examining edge-function evolution.The classification losses show smoothly connected scaling regimes, while regression follows a single power law at larger dataset sizes.Source: §I · p. 2
1. Test-loss scaling laws
On MNIST and Fashion-MNIST, all three KAN variants show random-guess, faster-scaling, and slower-scaling regimes. Fitted slopes and crossover scales vary with architecture and width.
1. Test-loss scaling laws
- On MNIST and Fashion-MNIST, all three KAN variants show RG–FS–SS broken scaling, with architecture- and width-dependent fitted behavior.The figures plot test cross-entropy loss against training dataset size for each architecture.Source: §1 · pp. 2, 5, Figure 2 · p. 3, Figure 4 · p. 4
- For nh=4 BSRBF-KAN, the RG plateau persists below ND∼10^2; the fit gives α=0.50, β=0.11, and N×=9.3×10^3, with the crossover in the curve's FS-to-SS bend.The paper distinguishes the faster-scaling branch from the slower-scaling branch by α>β.Source: §1 · p. 3
- For BSRBF-KAN on MNIST, increasing width steepens both fitted branches and shifts N× to smaller dataset sizes; widths 128 and 256 leave the RG plateau earlier.The resolved RG–FS–SS sequence differs from the single power law reported for the MLP baseline over the measured range.Source: §1 · pp. 3–4
- On MNIST, Faster-KAN has larger fitted exponents and crossover scales than BSRBF-KAN over much of the width range, while Gottlieb-KAN has the largest α, β, and N×.Gottlieb-KAN also retains a more persistent random-guess plateau.Source: Figure 3 · p. 4, §1 · pp. 4–5
- On Fashion-MNIST, BSRBF-KAN's FS branch becomes less steep as width increases while its SS branch stays nearly stable; Faster-KAN has a steeper SS branch, and Gottlieb-KAN the steepest branches and largest N×.Across all three variants, N× decreases overall with width.Source: Figure 5 · p. 5, §1 · p. 5
2. Evolution of trained edge functions
As dataset size increases, Gottlieb-KAN edge functions converge toward shared layer-specific shapes rather than becoming more diverse. This limits the practicality of assigning a distinct symbolic expression to every edge at scale.
2. Evolution of trained edge functions
- After normalization, Gottlieb-KAN edge functions within each layer collapse onto a common shape, with the collapse becoming more pronounced as dataset size increases.Normalization removes each function’s amplitude and sign.Source: §2 · p. 5
- Figure 6 compares shared initial functions with trained functions at ND = 1000, 10000, and 60000 across three connection types and two hidden-layer sizes.Its insets show normalized magnitudes; the vertical-axis range differs across trained-function columns.Source: §2 · p. 5, Figure 6 · p. 6
- Increasing Gottlieb-KAN model size from nh = 64 to nh = 256 adds nearly five times as many edge functions, but their shapes resemble those already observed.By contrast, BSRBF-KAN edge functions vary widely and do not form a common shape family.Source: §2 · p. 6
- Because many Gottlieb-KAN edges share a shape family, assigning each edge a distinct symbolic expression is impractical at scale.The authors note that this does not rule out symbolic descriptions of individual edges.Source: §2 · p. 6
1. Test loss scaling laws
KAN test-loss scaling was evaluated for magnetic Hamiltonian regression across dataset sizes and model capacities. The results show task- and architecture-dependent scaling, including a short random-guess regime, capacity saturation, and robust crossover behavior.
1. Test loss scaling laws
- The regression analysis measures MSE test loss across 25 logarithmically spaced dataset sizes from 7 to 57344 and varies edge functions per node over nh = {4, 16, 64, 256}.Crossover behavior is preserved under a fixed learning rate of 10^-4, and results are highly robust to the tested edge-function hyperparameters.Source: §1 · pp. 7–9
- For BSRBF-KAN with nh = 4, θ loss enters a linear-scaling regime near ND ∼102 with α = 1.73; J and D show similar crossovers but smaller exponents and higher final losses.The random-guess regime is barely resolved for this θ model.Source: §1 · p. 8
- Larger BSRBF-KAN models generally lower losses in the linear-scaling regime, but gains largely saturate by nh = 16; longer training can lower their losses below nh = 4.In the short-data regime, wider models do not always achieve lower losses, particularly for θ and J.Source: §1 · p. 8
- θ regression exponents are 1.77–2.59, versus 1.38–1.93 for J and 1.28–2.00 for D across model sizes; these KAN exponents change little as model size increases.The reported MLP exponents span 0.82–2.29 and increase with model size.Source: Figure 8 · p. 8, §1 · p. 9
- Faster-KAN and Gottlieb-KAN show the same qualitative random-guess-to-linear-scaling crossover as BSRBF-KAN, but remain near random-guess loss to ND ∼5 × 10^2.Gottlieb-KAN also shows larger deviations from fits and larger error bars.Source: §1 · pp. 8–9
2. Representation learning of symbolic expressions
The analysis tracks learned Gottlieb-KAN edge-function shapes in magnetic regression as dataset size grows. The functions largely retain shared shape families, while their scale distributions change and effective rank stays near one.
2. Representation learning of symbolic expressions
- Across input-to-hidden, hidden-to-hidden, and hidden-to-output layers, trained edge functions largely retain the shape family present at initialization, with changes mainly in their scale-factor distribution.The comparison samples dataset sizes ND = 224, 7168, and 57344; the θ task is shown, with qualitatively similar behavior reported for J and D.Source: §2 · p. 9
- The leading explained-variance ratio and effective rank remain close to one across layers and dataset sizes, even as test loss falls by several orders of magnitude.The effective rank does not increase appreciably with ND.Source: §2 · p. 9
- PCA supports the interpretation that prediction improvements mainly involve changing scalar multipliers rather than introducing more diverse edge-function shapes.The analysis samples learned edge functions on a fixed grid and applies PCA separately by layer and dataset size.Source: §2 · p. 9
III. DISCUSSION
KAN test losses show different scaling regimes in image classification and magnetic parameter regression, with exponents depending on task and implementation. The discussion also links activation-function behavior to basis choices and data distributions, and calls for architecture-specific evaluation.
III. DISCUSSION
- Image classification has random-guess, faster-scaling, and slower-scaling regimes, while magnetic regression crosses from random guess to a single linear-scaling regime with exponents typically near 2.0.The authors report broken neural scaling laws in both task types, but distinct regime structures.Source: §III · p. 9
- For Gottlieb-KAN, edge functions within layers remain in a common shape family as dataset size increases, with their scale distribution changing mainly.The discussion figure caption reports this pattern for θ regression at ND = 224, 7168, and 57344.Source: Figure 9 · p. 10
- The effective rank for twisted-magnet Hamiltonian parameter regression remains close to one as dataset size changes.Figure 10 reports the mean over 10 realizations for θ, J, and D, with shaded standard-deviation regions.Source: Figure 10 · p. 10
- KAN scaling exponents are sensitive to both task and implementation, and activation-function behavior differs across the studied basis types.BSRBF-KAN and Faster-KAN behavior is attributed primarily to ReLU-based foundations and bimodal data distributions; Gottlieb-KAN is reported to leverage its polynomial basis functions to fit those distributions.Source: §III · p. 10
- The authors argue that KAN variants require systematic, case-by-case scaling analysis and should be selected with domain-specific data characteristics in mind.They note that some data distributions can render particular basis functions ineffective and hinder meaningful pattern learning.Source: §III · pp. 10–11
IV. METHODS
The methods combine MNIST and Fashion-MNIST classification with regression of twisted-bilayer CrI3 magnetic parameters from domain images. The study describes dataset preparation, sampling, and training procedures for these evaluations.
IV. METHODS
- The magnetic task regresses twist angle θ and parameters J and D from images of local out-of-plane magnetization in twisted bilayer CrI3.The Hamiltonian includes intralayer ferromagnetic exchange, single-ion anisotropy, and interlayer exchange; J⊥ is determined by θ.Source: §IV · p. 11
- Magnetic images were cropped from 100 × 100 to 28 × 28 patches, and 70,536 image pairs formed the training reservoir for dataset sizes ND = 8 to 65,536.An independent test set contained 10,000 paired images; training subsets used a 7/8 training and 1/8 validation split.Source: §IV · p. 11
- For MNIST and Fashion-MNIST, subsets of the official training splits were evaluated at 35 approximately logarithmically spaced sizes from ND = 10 to 60,000.Classification loss was final-epoch cross-entropy on the official test split.Source: §IV · p. 11
- Classification and regression models were trained for 15 and 200 epochs, respectively, using AdamW and an adjustable learning rate.The experiments averaged approximately 15 realizations for each classification dataset and 10 for magnetic images.Source: §IV · p. 11