PaperlayerView exact revision on arXiv

Source-linked AI summary

KanAdapter: A Kolmogorov-Arnold Network-based Plug-and-Play Module for Efficient Fine-tuning of Foundation Speech Models

Phuong Tuan Dat, Phuong Khai Minh, Tran Huy Dat

arXiv:2609.05281v1cs.SD

TL;DR

Fully fine-tuning speech SSL models is computationally prohibitive, while MLP-based PEFT adapters have limited expressiveness under tight parameter budgets. KanAdapter replaces their bottlenecks with GR-KAN modules in frozen backbones, achieving competitive cross-task performance with up to 97.5% fewer trainable parameters and up to 83.6% error reduction in continual learning.

  • Problem

    Fully fine-tuning speech SSL models is computationally prohibitive, and fixed activations in predominant MLP-based adapters limit representational expressiveness under tight parameter budgets.

  • Method

    KanAdapter inserts trainable GR-KAN modules as lightweight plug-in adapters within frozen speech SSL backbones for expressive, parameter-efficient adaptation.

  • Results

    Across speaker verification, speech emotion recognition, and deepfake detection, KanAdapter remains highly competitive while reducing trainable parameters by up to 97.5% relative to full fine-tuning and outperforms AdaptFormer under comparable budgets.

  • Takeaways & Limitations

    Continual learning evaluations show up to 83.6% error reduction on previously learned tasks, attributed to the localized nature of GR-KAN’s rational activations.

  • Takeaways & Limitations

    The evaluation focuses on speech classification tasks and a two-stage continual-learning setting, leaving sequence generation, other modalities, and longer continual-learning scenarios unverified.

Abstract

from arXiv · show

Fully fine-tuning self-supervised learning (SSL) speech models for downstream tasks is computationally prohibitive, and existing parameter-efficient fine-tuning approaches predominantly rely on MLP-based adapters whose fixed activation functions limit their representational expressiveness under tight parameter budgets. We propose \textbf{KanAdapter}, a lightweight adapter framework that replaces conventional MLP bottlenecks with Group-Rational Kolmogorov-Arnold Network (GR-KAN) modules for more expressive and parameter-efficient adaptation. Following a parallel bottleneck design, KanAdapter inserts trainable GR-KAN branches alongside frozen Transformer encoder blocks and leverages weight transfer from pre-trained MLP layers for stable initialization. Across speaker verification, speech emotion recognition, and deepfake detection, KanAdapter achieves up to 97.5\% reduction in trainable parameters relative to full fine-tuning while remaining highly competitive, and consistently outperforms AdaptFormer under comparable parameter budgets. In continual learning, it yields up to 83.6\% error reduction over full fine-tuning and MLP-based adapters, which we attribute to the localized nature of GR-KAN's rational activations that mitigates catastrophic forgetting. To our knowledge, this is the first work to explore KAN-based modules for parameter-efficient fine-tuning of speech foundation models.

1 Introduction

Speech SSL models provide transferable representations for downstream tasks, but fully fine-tuning them is computationally prohibitive. Existing PEFT adapters reduce this burden yet commonly rely on MLPs whose fixed activations limit expressiveness, motivating KanAdapter’s KAN-based design.

  • Speech SSL models achieve strong performance across speaker verification, speech emotion recognition, and deepfake detection after pre-training on massive unlabeled data.
  • Fully fine-tuning large-scale speech SSL models is computationally prohibitive for practical deployment.
  • PEFT methods inject small trainable modules into frozen backbones instead of updating all parameters.
  • MLP-based adapters use fixed activation functions that limit function-approximation capacity relative to more flexible alternatives.
  • KANs replace fixed MLP activations with learnable univariate mappings on network edges, offering more flexible approximation and improved resistance to catastrophic forgetting.
  • KanAdapter integrates GR-KAN modules as plug-in adapters within frozen SSL backbones for expressive, parameter-efficient adaptation.

1. We propose KanAdapter, a simple yet effec-

The paper introduces KanAdapter as a GR-KAN-based approach for efficient speech SSL fine-tuning. Experiments across three speech tasks and continual learning report competitive performance, lower parameter costs, and stronger resistance to catastrophic forgetting.

  • KanAdapter is presented as the first work exploring KAN-based networks for efficient fine-tuning of speech foundation models.
  • Experiments on speaker verification, speech emotion recognition, and deepfake detection show KanAdapter outperforming existing fine-tuning approaches with significantly fewer trainable parameters.
  • Continual learning evaluations show substantially stronger resistance to catastrophic forgetting than full fine-tuning and existing adapter-based methods.

2 Related Work

Related work covers KAN foundations and applications in vision and speech, alongside established PEFT approaches. These studies motivate GR-KAN-based adaptation while distinguishing KanAdapter’s speech foundation-model setting.

  • KAN foundations: KANs use learnable edge-wise activations and have been reported to offer superior continual learning relative to MLPs through localized basis functions.
  • KANs in Computer Vision: Vision studies apply KANs to classification heads, adapters, and Transformer feed-forward replacements across image and video benchmarks.
  • KANs in Speech Processing: Speech studies use GR-KAN projectors, KAN-integrated Conformers, and systematic KAN analyses for speech deepfake detection and spoken language understanding.
  • PEFT: PEFT includes bottleneck adapters, low-rank weight updates through LoRA, and prompt tuning, while its application to speech SSL models remains underdeveloped.

3 Preliminary

KANs replace fixed node activations with learnable univariate edge functions, while GR-KAN improves their efficiency through rational functions, grouped sharing, and variance-preserving initialization.

  • Kolmogorov-Arnold Networks: KANs construct multivariate mappings from compositions of learnable univariate functions, providing the theoretical basis for edge-wise nonlinear transformations.The representation theorem states that continuous multivariate functions can be represented by finite compositions of univariate continuous functions.
  • Kolmogorov-Arnold Networks: KAN layers place learnable univariate functions on network edges instead of fixed activations at conventional MLP nodes.A full KAN stacks such layers, with each layer represented by a matrix of learnable univariate functions.
  • Kolmogorov-Arnold Networks: Each KAN edge can use a SiLU pathway combined with a learnable B-spline, enabling flexible, data-driven nonlinearities unlike fixed shared MLP activations.The B-spline coefficients are trainable, while the SiLU component provides a smooth residual pathway.
  • Group-Rational KAN: Vanilla KANs face GPU inefficiency, parameter overhead, and unstable default initialization when applied to large-scale models.These challenges arise from recursive B-splines, unique edge activations, and variance-violating initialization.
  • Group-Rational KAN: GR-KAN replaces B-splines with rational functions, requiring 21 FLOPs per evaluation versus 204 FLOPs for an equivalent B-spline at m=5 and n=4.Horner evaluation maps rational functions to parallel hardware operations.
  • Group-Rational KAN: Grouped parameter sharing reduces unique activation functions from din × dout to g, keeping GR-KAN near standard-MLP parameter complexity with constant overhead.Rational coefficients are shared within input-channel groups while each edge retains its own scalar weight.
  • Group-Rational KAN: GR-KAN initializes rational coefficients to approximate known activations and scales weights using estimated variance, supporting stable gradient flow.The scheme fits identity or Swish-like functions before initializing w ∼N(0, α/din).
  • Group-Rational KAN: Together, rational functions, grouped sharing, and variance-preserving initialization make GR-KAN an efficient, stable, expressive alternative to vanilla KAN and MLP.These properties motivate using GR-KAN as KanAdapter’s core component.

4 Proposed Method

KanAdapter adds trainable GR-KAN bottleneck branches in parallel with frozen Transformer MLP branches, adapting speech SSL representations while keeping the backbone unchanged.

  • Architecture: KanAdapter is a lightweight plug-and-play module for parameter-efficient fine-tuning of speech SSL models.It is designed as an additional trainable branch within Transformer encoder blocks.
  • Architecture: The parallel bottleneck design preserves each Transformer block’s frozen MLP branch while adding a trainable KanAdapter branch.This follows the structural idea of AdaptFormer while retaining the original backbone pathway.
  • Architecture: KanAdapter uses down- and up-projections around a GR-KAN transformation instead of a standard MLP intermediate nonlinearity.The GR-KAN layer supplies learnable rational activations within the low-rank bottleneck.
  • Fine-tuning: During fine-tuning, only the bottleneck projections, GR-KAN coefficients, and scaling factor are updated while the SSL backbone remains entirely frozen.Task-specific adaptation therefore occurs exclusively through lightweight KanAdapter branches.

5 Experiments

Experiments across speaker verification, speech emotion recognition, and deepfake detection compare KanAdapter with parameter-efficient and full fine-tuning strategies. KanAdapter remains competitive with substantially fewer trainable parameters and shows strong continual-learning retention.

  • Speaker Verification: KanAdapter achieves competitive speaker verification with 9M trainable parameters, reducing parameters by 97.5% while closely approaching full fine-tuning.It attains EERs of 0.52%, 0.50%, and 1.93% on Vox1-O, Vox1-E, and Vox1-H, versus full fine-tuning at 0.49%, 0.44%, and 1.58%.
  • Speech Emotion Recognition: KanAdapter delivers the strongest parameter-efficient speech emotion recognition results, using 16M parameters and reducing trainable parameters by 95.2% versus full fine-tuning.On the development set it achieves F1-Macro 0.5127 and F1-Micro 0.3436; on the test set, F1-Macro 0.3290 and F1-Micro 0.3180.
  • Deepfake Detection: KanAdapter reduces deepfake-detection trainable parameters from 319M to 17M, a 94.7% reduction, while remaining closest to full fine-tuning across every evaluation set.It achieves 0.34% EER on LA19 and outperforms LoRA and AdaptFormer on cross-corpus LA21 and DF21.
  • Deepfake Detection: On the ITW out-of-domain benchmark, KanAdapter achieves 7.96% EER, outperforming LoRA, AdaptFormer, and full fine-tuning despite using a fraction of the parameters.The reported EERs are 10.94% for LoRA, 10.21% for AdaptFormer, and 8.34% for full fine-tuning.
  • Continual Learning: KanAdapter achieves the best continual-learning performance across all four evaluation sets and reduces LA19 EER by 83.6% over full fine-tuning and 82.6% over AdaptFormer.The paper attributes this retention to localized rational basis functions whose updates affect specific input regions, reducing interference with previously acquired knowledge.

6 Ablation Study

The ablations examine how bottleneck size affects KanAdapter and how well it transfers across SSL backbones. A bottleneck of 128 offers the best reported efficiency-performance trade-off, while performance is strongest on larger, deeper models.

  • 6.1 Effect of Bottleneck Dimension ˆd: KanAdapter’s bottleneck dimension controls trainable parameter count and trades adapter capacity against regularization.The study varies ˆd across 512, 256, 128, and 64 on deepfake detection.
  • 6.1 Effect of Bottleneck Dimension ˆd: 22.18% to 18.40%: LA5 EER drops as ˆd decreases from 512 to 128.The improvement is consistent across all evaluation sets and supports ˆd = 128 as the default.
  • 6.1 Effect of Bottleneck Dimension ˆd: At ˆd = 64, performance slightly degrades because the excessively small bottleneck restricts representational capacity.The results indicate a capacity limit beyond the best-performing bottleneck size.
  • 6.2 Generalizability Across SSL Backbones: Across WavLM Large, UniSpeech-SAT, and mHuBERT-147, KanAdapter reduces trainable parameters by 94.9%, 91.0%, and 91.1%, respectively, while approaching full fine-tuning performance.The evaluation reports EER on LA21 and DF21 across three additional SSL backbones.
  • 6.2 Generalizability Across SSL Backbones: On WavLM Large, KanAdapter reaches 4.19% and 11.76% EER on LA21 and DF21, within 0.12% and 0.51% of full fine-tuning.WavLM Large has 24 Transformer encoder layers and 333M parameters.
  • 6.2 Generalizability Across SSL Backbones: Performance gaps widen on the 12-layer UniSpeech-SAT and mHuBERT-147 models, although KanAdapter remains viable at a fraction of the parameter budget.The ablation therefore identifies large, deep SSL backbones as the most favorable setting.

7 Conclusion and Future Work

The paper presents KanAdapter as a lightweight GR-KAN-based adapter for speech SSL foundation models. Across three speech tasks it remains competitive while sharply reducing trainable parameters, with stronger continual-learning resistance, while broader modality and task applications remain future work.

  • Conclusion: KanAdapter is a lightweight, plug-and-play parameter-efficient fine-tuning framework built by replacing MLP adapter bottlenecks with GR-KAN modules.The design targets expressive task-specific adaptation while preserving deployment efficiency.
  • Conclusion: Across speaker verification, speech emotion recognition, and deepfake detection, KanAdapter reduces trainable parameters by up to 97.5% while remaining highly competitive with full fine-tuning.It also substantially outperforms AdaptFormer under comparable parameter budgets.
  • Conclusion: Continual-learning evaluations report EER reductions of up to 83.6% on previously learned tasks, attributed to localized grouped rational activations.The reported result concerns resistance to catastrophic forgetting.
  • Future Work: The study leaves applications to image, video, and large language models for future work.These extensions are motivated by KanAdapter’s plug-and-play nature and the continual-learning properties of KAN-based architectures.

Limitations

The study’s evidence is bounded by unprobed mechanisms, weaker results on shallow backbones, limited task and modality coverage, and a two-stage continual-learning evaluation.

  • Limitations: The improved expressiveness of GR-KAN over MLP adapters is demonstrated through downstream performance rather than targeted representation-level analyses.The paper does not isolate the contribution of learnable rational activations through spectral or frequency-band studies.
  • Limitations: Benefits are most pronounced on large, deep SSL backbones, while gaps to full fine-tuning widen for shallower models.UniSpeech-SAT and mHuBERT-147 are given as examples.
  • Limitations: Evaluation focuses on speech classification tasks, leaving sequence generation, other modalities, and large language models unverified.The paper specifically identifies ASR and broader modalities as unsupported evaluation areas.
  • Limitations: Continual learning is evaluated only in a two-stage ASVspoof2019-to-ASVspoof5 setting, so longer task sequences and broader benchmarks remain needed.The limitation concerns fully characterizing resistance to catastrophic forgetting.
Loading 2609.05281v1…