Source-linked AI summary
TKAN: Temporal Kolmogorov-Arnold Networks
Remi Genet, Hugo Inzirillo
TL;DR
The paper addresses the challenge of modeling long-term dependencies and complex sequential patterns in multi-step time-series forecasting. It proposes TKAN, which combines KAN nonlinearities with recurrent and LSTM-inspired memory mechanisms; reported results show stronger long-horizon performance and stability than the compared recurrent models.
Problem
Long-term dependencies and complex sequential patterns remain challenging for forecasting models, especially when predicting multiple steps ahead.
Method
TKAN combines RKAN layers with a modified LSTM cell, using learnable KAN transformations and memory management for temporal sequences.
Results
At longer horizons, TKAN achieves an R-squared value at least 25% higher than GRU, while its learning and validation losses remain more closely aligned.
Takeaways & Limitations
TKAN is presented as a promising architecture for multi-step forecasting that improves performance and stability compared with traditional methods.
Abstract
from arXiv · showhide
Recurrent Neural Networks (RNNs) have revolutionized many areas of machine learning, particularly in natural language and data sequence processing. Long Short-Term Memory (LSTM) has demonstrated its ability to capture long-term dependencies in sequential data. Inspired by the Kolmogorov-Arnold Networks (KANs) a promising alternatives to Multi-Layer Perceptrons (MLPs), we proposed a new neural networks architecture inspired by KAN and the LSTM, the Temporal Kolomogorov-Arnold Networks (TKANs). TKANs combined the strenght of both networks, it is composed of Recurring Kolmogorov-Arnold Networks (RKANs) Layers embedding memory management. This innovation enables us to perform multi-step time series forecasting with enhanced accuracy and efficiency. By addressing the limitations of traditional models in handling complex sequential patterns, the TKAN architecture offers significant potential for advancements in fields requiring more than one step ahead forecasting.
I. INTRODUCTION
The introduction motivates time-series forecasting as an important application and identifies long-term dependency handling as a limitation of existing neural networks. It presents KAN-based memory mechanisms as the basis for a new temporal architecture.
- Time-series forecasting supports applications in finance, meteorology, healthcare, and other industries with growing data and computational resources.
- RNNs address dependencies between successive observations, whereas feedforward MLPs are poorly suited to modeling temporal sequences.
- The proposed direction adds external memory to KAN layers so networks can retain and use information relevant to temporal context.
- KANs represent functions through compositions of learnable univariate functions, commonly parameterized with local B-spline basis functions.
- A KAN is organized as a composition of layers whose activation functions connect nodes between successive computational levels.
III. TEMPORAL KOLMOGOROV-ARNOLD NETWORKS (TKANS)
TKAN extends KANs for sequential data by combining recurrent KAN layers with gating and an LSTM-inspired memory cell. This design targets long-term information retention and multi-step forecasting.
- The TKAN layer combines RKAN architecture with a modified LSTM cell to manage sequential data and temporality during learning.
- The architecture combines learnable RKAN nonlinearities with LSTM-based memory of past events over long periods.
- RKAN layers retain short-term memory from previous states while processing each layer.
- Gating mechanisms regulate information flow by selecting which information to retain or forget over time.
A. Recurring Kolmogorov-Arnold Networks (RKAN)
RKAN introduces temporal memory into KAN transformations by incorporating previous hidden states into current computations. Its recurrent kernels and layer substates allow KAN layers to process ordered sequences.
- A recurrent kernel transforms the previous hidden state, while an input kernel transforms the current input at each timestep.
- RKAN modifies each KAN transformation to incorporate previous hidden states, enabling dynamic temporal behavior for sequences where context and order matter.
- The RKAN input combines the current input with the sublayer’s previous state through layer-specific weights.
- Each RKAN sublayer has a memory state initialized with zeros and shaped according to the KAN output dimension.
- The memory step combines past hidden states using weight vectors that determine the importance of past values relative to recent inputs.
B. TKAN Architecture
The TKAN architecture manages temporal information by combining KAN-derived representations with LSTM-inspired memory gates. Its recurrent and gated components support sequence processing and multi-step forecasting.
- Memory management: TKAN uses an LSTM-inspired unit to manage memory during sequential processing.The unit includes internal vectors and gates that regulate information flow.
- Memory management: The forget, input, and output gates regulate what information is discarded, incorporated, and emitted.The forget gate handles prior-state information, the input gate controls new information, and the output gate controls emitted information.
- TKAN state updates: The TKAN output gate determines what information from the current state is output using the RKAN-derived representation r_t.The hidden state captures the unit’s output, while the cell state is updated separately.
- Forecasting objective: TKAN architecture is evaluated for both one-step-ahead and multi-step-ahead forecasting.The learning task tests the model’s prediction power in both settings.
IV. LEARNING TASK
The study evaluates whether extending the model improves out-of-sample prediction accuracy across multiple future steps. It compares the approach with standard GRU and LSTM layers using real market data.
- Evaluation objective: The learning task assesses out-of-sample prediction accuracy over several future steps.The comparison focuses on whether the extended model improves multi-step forecasting.
- Evaluation design: The study compares the model with standard GRU and LSTM layers.The evaluation uses real market data rather than synthetic data because synthetic market data can be biased toward an experiment.
A. Task Definition and Dataset
The task predicts future market notional trades, using hourly asset-level data from Binance to forecast BTC. The dataset covers multiple cryptocurrencies from 2020 through 2022.
- Task definition: The forecasting task predicts market notional trades over future periods.Market volume has internal patterns such as seasonality and autocorrelation, although market behavior is difficult to predict.
- Data source: The dataset uses Binance as its sole data source because it is described as the largest cryptocurrency-market player.The paper also notes concerns about falsified data and wash trading by smaller exchanges.
- Dataset: Hourly notional amounts for 19 assets are used to predict BTC.The assets include BTC, ETH, ADA, XMR, EOS, MATIC, TRX, FTM, BNB, XLM, ENJ, CHZ, BUSD, ATOM, LINK, ETC, XRP, BCH, and LTC.
- Dataset: The dataset spans January 1, 2020 to December 31, 2022.The observations are organized as hourly asset-level notional amounts.
B. Preprocessing
Preprocessing addresses scale differences and nonstationarity in market-volume data, which are especially important for KANs with power-exponent B-Spline activations. The pipeline combines shifted moving-median scaling with per-asset MinMaxScaling.
- Motivation: Preprocessing is necessary because market notional series differ substantially in scale and may be nonstationary.Poor scaling can hinder learning, while KAN B-Spline activations exhibit a power exponent that can produce underflows or overflows.
- Two-stage scaling: The first preprocessing stage divides each series by the moving median of the previous two weeks.The window is shifted by the forecast horizon to avoid foresight, and the step aims to make the series more stationary over time.
- Two-stage scaling: The second stage applies MinMaxScaling separately for each asset.The scaling uses the training-set adjustment and targets the [0, 1] interval to avoid explosive learning effects from the power exponent.
C. Loss Function for Model Training
The paper uses RMSE as the training loss for numerical prediction and reports R-squared to assess performance relative to series variance. It notes that minimizing MSE is equivalent to maximizing R-squared.
- RMSE is used as the loss function for the numerical prediction problem.The paper describes RMSE as a standard machine-learning loss for this task.
- N denotes the number of dataset samples, while the indexed predicted and true values define the RMSE quantities.The accompanying notation identifies N and the predicted and corresponding true values.
- R-squared evaluates error relative to the variance of the estimated series, aiding performance interpretation.The paper identifies R-squared as the metric used to display results.
- Minimizing MSE is exactly equivalent to maximizing R-squared according to the stated formula.
D. Benchmarks
The benchmark compares TKAN layers with GRU and LSTM layers for multi-step prediction using matched simple model configurations and a repeated-value baseline. The comparison targets layers rather than complete architectures.
- TKAN layers are compared with GRU and LSTM for multi-step predictions.The authors select GRU and LSTM as widely used recurrent comparators.
- The comparison evaluates recurrent layers rather than complete model architectures such as temporal fusion transformers.
- All three models use the same simple configuration to support a fair comparison.
- Each model includes two recurrent layers followed by a linear dense layer producing the requested number of future timesteps.
- TKAN uses B-spline activations of orders 0 through 4, while GRU and LSTM use standard activations and are also compared with a repeated-last-value baseline.
2) Note on training details:
Training uses scaled data, repeated experiments, and validation-based controls, while benchmark results track average and variability in R-squared across five runs. TKAN maintains a longer-horizon advantage and greater stability.
- Training details: Metrics are computed on scaled data, with MinMax scaling preserving R-squared and avoiding instability from median-based rescaling.
- Training details: Training uses Adam, a 20% validation split, early stopping after six stagnant periods, and learning-rate halving after three stagnant periods.
- Training details: Mean and standard deviation are reported from five repeated experiments for each model.
- Results: R-squared decreases as the forecast horizon increases because fewer observations inform later predictions.
- Results: At longer horizons, TKAN achieves an R-squared value at least 25% higher than GRU and shows better stability than the other models.
2) Training Dynamics and Model Stability:
Training and validation loss curves provide a dynamic view of learning and generalization. Their closer alignment for TKAN contrasts with divergence for GRU and LSTM, supporting the reported stability difference.
- Training Dynamics and Model Stability: Training and validation loss visualizations compare each model’s learning process and ability to generalize beyond training data.
- Training Dynamics and Model Stability: The TKAN, GRU, and LSTM loss figures display training and validation loss over epochs.
- Training Dynamics and Model Stability: GRU and LSTM show substantial divergence between learning-loss and validation trajectories as epochs increase.
- Training Dynamics and Model Stability: TKAN’s closer alignment between learning and validation loss curves indicates a more stable learning process.
- Training Dynamics and Model Stability: The visualizations corroborate the statistical results reported for model performance and stability.