PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Kolmogorov-Arnold Networks

Alexander Dylan Bodner, Antonio Santiago Tepsich, Jack Natan Spolski, Santiago Pourteau

arXiv:2406.13155v3cs.CVcs.AI

TL;DR

Convolutional KANs address whether learnable spline functions can make convolutional models more parameter-efficient while retaining competitive accuracy. The paper integrates B-spline-based functions into convolutional kernels and evaluates them on Fashion-MNIST, where KAN convolutions can achieve better accuracy than classic convolutions with half the parameters. The main scope limits are slow current implementations and evaluation confined to relatively simple datasets.

  • Problem

    The paper examines whether replacing fixed convolutional kernels with learnable spline-based functions can improve parameter efficiency while maintaining competitive accuracy.

  • Method

    The authors integrate learnable B-spline functions into convolutional layers and evaluate the resulting Convolutional KAN architectures against traditional models on Fashion-MNIST.

  • Results

    KAN convolutions achieve better accuracy than classic convolutions on Fashion-MNIST, including comparisons using half the parameters.

  • Takeaways & Limitations

    Convolutional KANs offer a parameter-efficient alternative for image models within the evaluated Fashion-MNIST setting.

  • Takeaways & Limitations

    Current B-spline KAN implementations are considerably slow because they cannot be GPU-parallelized, and broader datasets are needed to test whether the findings hold at scale.

Abstract

from arXiv · show

In this paper, we present Convolutional Kolmogorov-Arnold Networks, a novel architecture that integrates the learnable spline-based activation functions of Kolmogorov-Arnold Networks (KANs) into convolutional layers. By replacing traditional fixed-weight kernels with learnable non-linear functions, Convolutional KANs offer a significant improvement in parameter efficiency and expressive power over standard Convolutional Neural Networks (CNNs). We empirically evaluate Convolutional KANs on the Fashion-MNIST dataset, demonstrating competitive accuracy with up to 50% fewer parameters compared to baseline classic convolutions. This suggests that the KAN Convolution can effectively capture complex spatial relationships with fewer resources, offering a promising alternative for parameter-efficient deep learning models.

1 Introduction

The paper adapts spline-based Kolmogorov-Arnold Networks to convolutional layers, targeting greater flexibility and parameter efficiency in image models. It evaluates this approach against traditional models on Fashion-MNIST.

  • Motivation: CNNs are widely used for high-dimensional image arrays because convolutional layers capture spatial relationships with relatively few parameters.Traditional convolutional layers normally combine linear transformations with optional activation functions.
  • Motivation: KANs integrate spline functions into neural architectures as an alternative to conventional MLPs.Their architecture is based on the Kolmogorov-Arnold theorem.
  • Approach: The paper adapts KANs to convolutional layers and uses spline-based convolutions to model non-linear relationships in structured image data.This extends the KAN approach to a common component of CNN architectures.
  • Evaluation: The study benchmarks several Convolutional KAN architectures against traditional models on Fashion-MNIST, emphasizing parameter efficiency.The stated hypothesis is that Convolutional KANs can use fewer parameters while maintaining competitive accuracy.
  • Evaluation: The paper provides code for the convolutional layer and experiments in a public GitHub repository.The repository is intended to support further exploration and practical application.

2 Related work

The related work connects KANs and spline-based convolutions as mathematical approaches for modeling complex data. The paper distinguishes its method from SplineCNN by applying spline functions directly to structured images and matrices rather than first converting images into graphs.

  • Kolmogorov-Arnold Networks: KANs use the Kolmogorov-Arnold representation theorem to motivate spline-based neural architectures.The theorem represents multivariate continuous functions through compositions of univariate functions and additions.
  • Spline-based convolutions: SplineCNN introduces spline-based convolutional layers for capturing non-linear relationships, particularly in geometric deep learning.Its spline adaptability is described as useful for non-Euclidean data.
  • Distinction from SplineCNN: Unlike SplineCNN, Convolutional KANs apply spline functions directly to structured images and matrices without converting them into graphs.SplineCNN is described as interpreting image data as graphs before classification.

3 Kolmogorov-Arnold Networks (KANs)

KANs replace conventional linear-weight transformations and fixed node activations with learnable one-dimensional spline functions organized across network layers. This design provides adaptive functional mappings that can model complex interactions while targeting parameter efficiency.

  • Foundations: KANs are based on the Kolmogorov-Arnold representation theorem, which expresses multivariate continuous functions through univariate functions and additions.This theorem provides the conceptual foundation for the architecture.
  • Architecture: KANs place learnable activation functions on network edges instead of using fixed activation functions and conventional linear weight matrices.The spline parameters are optimized during training.
  • Layer structure: KAN layers organize learnable one-dimensional functions in a matrix connecting input and output features.The matrix indexes functions by input and output dimensions.
  • B-spline parameterization: The functions can be represented as B-splines, which are linear combinations of basis splines for learning complex data representations.The layer dimensions determine the number of input and output features.
  • B-spline parameterization: The spline formulation uses trainable coefficients, allowing each function to adapt its shape to the data.This adaptive shape is presented as the source of flexibility in modeling input interactions.
  • Architecture: Each KAN layer applies a functional mapping that transforms its input into the next layer’s input.The layers are analogous to stacked MLP layers but use learned function mappings rather than only linear transformations and nonlinear activations.
  • Layer structure: Each activation function in the layer matrix is a spline that provides an adaptable response surface for modeling inputs.The architecture describes these functions as learnable spline mappings.
  • Motivation: Replacing linear weight matrices with spline functions is intended to improve expressive power and efficiency while reducing the parameters needed for high accuracy.The paper also associates this design with faster convergence and better generalization.

4 Convolutional Kolmogorov-Arnold Networks

Convolutional KANs replace fixed convolutional weights with learnable spline-based functions applied independently to pixels, enabling nonlinear spatial processing within a convolution. The architecture also uses grid adaptation and spline flexibility to reduce reliance on parameter-heavy fully connected layers.

  • KAN Convolutions: Convolutional KANs replace CNN convolutional layers with KAN convolutional layers whose kernel elements are learnable nonlinear functions parameterized by B-splines.Each activation function is applied to its corresponding pixel, and the resulting values are summed to produce the output pixel.
  • KAN filters theoretical advantages: Per-pixel nonlinearities let a single KAN filter combine linear weighting with thresholding, adaptive gains, smoothing, or edge-detection behavior.Classic filters use fixed weights and require separate processing steps for comparable conditional or piecewise operations.
  • Threshold + Convolution: A thresholding example uses a displaced ReLU-like spline at the center pixel while applying linear weights elsewhere, producing both operations in one pass.The learned spline is smooth and can only approximate the idealized displaced ReLU behavior.
  • Piecewise Brightness-Based Boost: Piecewise functions can assign different gains to bright and dark pixels, allowing brightness-dependent scaling that a single linear filter cannot replicate.The bright and dark gains are selected according to whether a pixel intensity is above or below a threshold.
  • Grid extension and update: Grid extension updates the spline range when inputs leave the default [−1, 1] interval, while batch normalization standardizes layer inputs toward μ = 0 and σ = 1.Without range updates, out-of-range inputs are dominated by the SiLU component rather than the learned spline.
  • Parameter efficiency: Convolutional KANs use more parameters per convolutional layer than CNN convolutions but can reduce total parameters by requiring fewer fully connected layers.The paper gives K²(gridsize + 2) parameters per Convolutional KAN layer versus K² for a CNN convolutional layer.

5 Experiments

The experiments compare KAN-based and standard convolutional architectures on Fashion-MNIST, using grid search and train/validation tuning. KAN models include spline-specific regularization options, but the reported experiments set those regularization terms to zero.

  • Datasets: Fashion-MNIST was selected for hyperparameter tuning and reporting because its greater complexity made running all experiments on both datasets costly.MNIST and Fashion-MNIST were initially considered, but only Fashion-MNIST received full experimentation.
  • Architectures: The evaluated architectures mix fully connected, KAN, KAN convolutional, and standard convolutional layers.KAN and standard architectures are presented separately in Figures 2 and 3; max pooling follows each convolutional layer and every model ends with Log Softmax.
  • Hyperparameter tuning: Grid search tested 8 hyperparameter combinations per model and compared KAN grid sizes of 10 and 20, while fixing the B-Spline degree at 3.Grid size was varied because it substantially affects parameter count; degree 3 followed the KAN authors’ recommendation and common cubic-spline usage.
  • Evaluation procedure: Hyperparameters were tuned on train and validation splits, after which the optimal models were retrained on the combined data and evaluated on the test set.K-fold cross-validation was omitted because of the models’ long training times.
  • Objective: All models used categorical cross-entropy, while KAN models additionally supported two KAN regularization terms that were set to λ = 0 in the experiments.Early experiments indicated better performance and faster training without those regularizers.

6 Results

The results compare accuracy, classification metrics, parameter counts, and training time for KAN-based and standard models on Fashion-MNIST. KAN convolutions outperform standard convolutions in smaller models, while deeper MLP settings and grid-size choices reveal important trade-offs.

  • Small and Medium models: In smaller models, KAN convolutions outperform classic convolutions when paired with either MLPs or KANs after flattening.The reported fair comparisons include KANC MLP versus CNN and Normal Convs & KKAN versus KKAN for Small and Medium models.
  • Parameter trade-offs: 88.15% versus 88.34%: KANC MLP Small has more trainable parameters than CNN Medium but slightly lower accuracy.The comparison indicates that added KAN-convolution expressibility did not compensate for having fewer distinct convolution filters in this case.
  • Deeper MLPs: 89.44% versus 89.15%: CNN (Big) slightly outperforms KANC MLP (Big) as MLP depth increases.The authors suggest that learning may shift toward the MLP, while KAN convolutions may compress information needed by the MLP.
  • Training trade-offs: 87.67% versus 88.01%: KKAN Small has lower accuracy than Conv & KAN Small, although Conv & KAN trains 6–7 times faster depending on grid size.The passage reports the accuracy comparison and training-speed trade-off for models using a KAN network after convolutional layers.
  • Model scaling: 88.56%: KKAN Medium improves over the 87.92% comparison while increasing parameter count from 38000 to 74875.The higher accuracy comes from increasing the number of convolutions and KAN neurons, with grid size 20 for KKAN Medium.
  • Grid-size sensitivity: Grid size can change accuracy abruptly: increasing it from 10 to 20 sometimes reduces accuracy, but improves KANC MLP Small and KKAN Medium by over 0.5%.Tuning grid size for every KAN layer may achieve the best performance but requires substantially more computational resources.

7 Conclusions

Convolutional KANs adapt learnable B-spline functions to convolutional layers and can improve accuracy or reduce parameters in several Fashion-MNIST comparisons. However, dense KAN layers and current implementations introduce important efficiency and interpretability limitations.

  • Contribution: Convolutional KANs replace conventional convolutional kernels with learnable non-linear B-spline functions.The proposed layer adapts KAN splines to convolutional operations used in computer vision.
  • Results: KAN Convolutions achieved better accuracy than classic convolutions after flattening, even with half the parameters.This comparison is reported for KKAN versus Normal Conv & KAN architectures.
  • Results: In two-layer MLP comparisons, classic CNNs won by 0.41% with approximately 26.62k parameters.KAN convolutions performed better in smaller models, but this advantage did not persist in the cited two-layer MLP comparison.
  • Parameter efficiency: KANC MLP (Medium) reached 88.99% with approximately 9k parameters, versus 89.44% for CNN (Big) with 26k parameters.The authors present this as evidence that KANs can maintain accuracy with fewer parameters.
  • Limitations: MLPs outperformed KANs in the Small and Medium dense-layer settings, while KKANs used many more parameters for only 0.91% and 0.22% accuracy gains.The paper therefore questions the suitability of KANs as dense layers for these image-classification architectures.
  • Limitations: The paper did not find a clear way to visualize the B-splines learned at each pixel, limiting direct interpretability of KAN convolutions.Filter visualization is presented as the most human-accessible current approach.
  • Limitations: KAN linear and convolutional layers require further optimization before they can be scaled properly, as reflected in time-per-epoch metrics.The authors characterize the work as a starting point for integrating KANs into computer vision.

8 Limitations and future work

The paper identifies computational cost, limited scaling evidence, and unresolved interpretability as key directions for future work. Proposed remedies include more GPU-parallelizable function approximators and evaluation on more complex datasets.

  • Computational efficiency: Current B-spline implementations are considerably slow because they cannot be GPU-parallelized, hindering real-world application.The paper identifies this implementation constraint as a major practical limitation.
  • Computational efficiency: Replacing B-splines with GPU-parallelizable approximators such as Radial Basis Functions is proposed as one way to improve KAN efficiency.The authors note that such replacements could expand the practical possibilities of KANs.
  • Evaluation scope: Experiments on more complex datasets such as CIFAR-10 or ImageNet are needed to test whether the reported results hold at scale.The current evidence is based on the paper’s evaluated image-classification setting.
  • Interpretability: Future work should evaluate ways to interpret and prune KAN convolutions.The paper identifies interpretability as a KAN strength that remains insufficiently developed for these convolutional layers.
Loading 2406.13155v3…