PaperlayerView exact revision on arXiv

Source-linked AI summary

CKConv: Continuous Kernel Convolution For Sequential Data

David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub M. Tomczak, Mark Hoogendoorn

arXiv:2102.02611v3cs.LG

TL;DR

Sequential models face short effective memory or fixed, costly convolutional horizons. CKConv represents convolutional kernels as continuous neural functions, and CKCNNs achieve strong results across varied sequence settings while supporting irregular sampling and different resolutions. The approach remains subject to linear sequence-length memory use and high-frequency artifacts when changing resolution.

  • Problem

    RNNs suffer from vanishing or exploding gradients and short effective memory horizons, while CNNs require predefined memory horizons and larger horizons increase model size.

  • Method

    CKConv parameterizes convolutional kernels as continuous functions using small neural networks, enabling arbitrarily large kernels without recurrence.

  • Results

    CKCNNs achieve state-of-the-art results on multiple datasets and often surpass architectures designed for particular settings, including irregularly sampled data.

  • Takeaways & Limitations

    CKConv provides parameter-efficient global long-term modeling while handling irregularly sampled data and different resolutions.

  • Takeaways & Limitations

    CKCNN memory complexity grows linearly with sequence length because convolution responses must be stored for every layer and input position.

Abstract

from arXiv · show

Conventional neural architectures for sequential data present important limitations. Recurrent networks suffer from exploding and vanishing gradients, small effective memory horizons, and must be trained sequentially. Convolutional networks are unable to handle sequences of unknown size and their memory horizon must be defined a priori. In this work, we show that all these problems can be solved by formulating convolutional kernels in CNNs as continuous functions. The resulting Continuous Kernel Convolution (CKConv) allows us to model arbitrarily long sequences in a parallel manner, within a single operation, and without relying on any form of recurrence. We show that Continuous Kernel Convolutional Networks (CKCNNs) obtain state-of-the-art results in multiple datasets, e.g., permuted MNIST, and, thanks to their continuous nature, are able to handle non-uniformly sampled datasets and irregularly-sampled data natively. CKCNNs match or perform better than neural ODEs designed for these purposes in a faster and simpler manner.

1 INTRODUCTION

RNNs provide arbitrary sequence unrolling but retain information over surprisingly short horizons, while CNNs avoid recurrent training issues at the cost of fixed, size-dependent horizons. CKConv addresses both limitations by parameterizing kernels continuously, enabling large, parallel, recurrence-free, and irregular-data processing.

  • 1 INTRODUCTION: CKConv parameterizes convolutional kernels as continuous functions generated by a small neural network, allowing kernels to be sampled at arbitrary relative positions.The continuous parameterization supports different resolutions and irregular sampling.
  • 1 INTRODUCTION: CKConvs define arbitrarily large memory horizons within one operation, decoupling horizon size from network depth, dilation, and network size.
  • 1 INTRODUCTION: CKConvs avoid recurrence, allowing parallel training without vanishing or exploding gradients or small effective memory horizons.This directly targets the recurrent mechanism associated with short effective memory.
  • 1 INTRODUCTION: Continuous kernels can be evaluated at arbitrary positions, so CKCNNs natively handle irregularly sampled data and different resolutions.
  • 1 INTRODUCTION: CKCNNs match or outperform state-of-the-art approaches across stress tests involving continuous, discrete, irregular, and resolution-changing data.

2 RELATED WORK

Prior continuous-kernel methods primarily address irregular data locally, while implicit neural representations provide a useful framework for continuous parameterization. CKConv connects these ideas and motivates sine-based kernels for modeling long, complex dependencies.

  • 2 RELATED WORK: Prior continuous-kernel formulations were introduced mainly to handle irregularly sampled 3D data and related point-cloud operations.
  • 2 RELATED WORK: Implicit neural representations encode continuous data in neural-network weights, offering properties such as memory efficiency and analytic differentiability.
  • 2 RELATED WORK: CKConv implicitly represents learned CNN kernels with neural networks, motivating sine nonlinearities that significantly improve over ReLU, LeakyReLU, and Swish parameterizations.

3 THE CONVOLUTION AND COMMON KERNEL PARAMETERIZATIONS

Standard discrete convolutions use finite independent weights, making memory horizons fixed and large global kernels costly; centered, causal, and dilated variants address different sequence dependencies with trade-offs. CKConv is introduced as a dense, arbitrarily long alternative under a fixed parameter budget.

  • 3 THE CONVOLUTION AND COMMON KERNEL PARAMETERIZATIONS: Causal convolution restricts each output to current and past inputs, avoiding the future-input dependence of centered convolution for sequence modeling.
  • 3 THE CONVOLUTION AND COMMON KERNEL PARAMETERIZATIONS: Discrete kernels use independent weights over a finite horizon, so their kernel size must be chosen beforehand and is often much shorter than the input.
  • 3 THE CONVOLUTION AND COMMON KERNEL PARAMETERIZATIONS: Global discrete kernels covering the full input require extreme model-size growth because every relative position has an independent weight.
  • 3 THE CONVOLUTION AND COMMON KERNEL PARAMETERIZATIONS: Dilated convolutions expand the memory horizon with interleaved zeros but cannot model dependencies at positions falling between retained kernel weights.
  • 3 THE CONVOLUTION AND COMMON KERNEL PARAMETERIZATIONS: Varying dilation factors tie memory horizon to network depth and layer-wise dilation choices, constraining architectural flexibility.
  • 3 THE CONVOLUTION AND COMMON KERNEL PARAMETERIZATIONS: CKConv instead handles arbitrarily long sequences with arbitrarily large, dense memory horizons in one layer under a fixed parameter budget.

4 CONTINUOUS KERNEL CONVOLUTION

The section formulates convolutional kernels as continuous neural functions, enabling large horizons, irregular sampling, and flexible kernel representations. SIREN parameterizations support complex long-range dependencies and strong results across sequence tasks.

  • Continuous Kernel Convolution parameterizes each convolutional kernel as a neural network that maps relative positions to kernel values.The resulting kernel can be constructed over an arbitrarily large set of positions rather than stored as independent discrete weights.
  • For an input sequence-length kernel, CKConv can model global long-term dependencies within a single convolution operation.When the kernel size equals the input size, the convolution covers the entire sequence.
  • CKConvs support irregularly sampled, partially observed, and differently resolved data by evaluating the continuous kernel at the relevant sample positions or resolution.Different resolutions require resampling the kernel, with the resulting convolutions approximately equivalent up to a resolution-change factor.
  • CKConvs describe a broader functional family than discrete convolutions and linear recurrent units.The section connects linear recurrent units to convolutional kernels while distinguishing CKConvs as more general.
  • SIREN parameterizations quickly approximate complex, non-smooth functions, giving CKConvs the capacity to model dependencies across large memory horizons.The experiments report near-perfect approximation across target functions and consistent superiority of SIREN kernels in ablations.

5 EXPERIMENTS

Experiments show that shallow CKCNNs handle long sequences, discrete tasks, varying resolutions, and irregular sampling while often outperforming recurrent, convolutional, attention, and continuous-time baselines.

  • Stress experiments: A shallow CKCNN solves both stress tasks across sequence lengths from 100 to 6000 without structural modifications, whereas recurrent models and TCNs fail at longer lengths.Recurrent architectures cannot solve the copy problem and solve the adding problem only up to 200 steps; TCNs solve both only up to 1000 steps.
  • Discrete sequences: Shallow CKCNNs outperform recurrent, self-attention, and convolutional models on sMNIST and pMNIST, achieving state-of-the-art results on both tasks.On sMNIST, a 100K-parameter CKCNN is 80× smaller than the current state-of-the-art; on pMNIST, the wider model improves over the best model of size ≤100K by 0.8%.
  • Time-series modeling: CKCNNs outperform all compared continuous-time models on CT and SC, while SC_raw results demonstrate performance on very-long-term dependencies.The SC_raw model also outperforms several Neural ODE models trained on preprocessed SC data.
  • Testing at different sampling rates: CKCNNs maintain stable performance across large sampling-rate changes, outperform HiPPO, and establish a new state-of-the-art in that setting.The comparison concerns training at one sampling rate and testing at another.
  • Irregularly-sampled data: CKCNNs outperform NCDEs and achieve state-of-the-art performance on PhysioNet, while remaining stable as missing-data rates vary.NCDEs perform slightly better on CT at large data-drop rates, but CKCNNs retain training-speed advantages.

6 DISCUSSION AND LIMITATIONS

The discussion highlights CKConv’s parameter and training-efficiency advantages, while identifying memory, initialization, depth, and frequency-related limitations.

  • Parameter-efficient large convolutional kernels: CKConv constructs global kernels with far fewer parameters than conventional CNNs, requiring 2.14M versus 100K parameters on pMNIST and 46.68M versus 100K on SC_raw.The corresponding parameter-efficiency factors are reported as 21.84× and 445.71×.
  • Is depth important?: Additional experiments suggest that deeper CKCNNs do not improve performance, but the authors state that further analysis is needed.The reported models use only two residual blocks.
  • Faster continuous-time models: CKCNNs execute in parallel and can train faster than recurrent and neural ODE models, although this advantage increases memory costs.NCDEs were reported as 228× slower than an equivalent CKCNN on SC_raw.
  • High-frequency components: Sine nonlinearities enable continuous-function models to represent high frequencies, extending receptive fields or reducing parameters for local functions.The discussion connects this design choice to continuous-kernel convolutions, transformer positional encodings, and graph neural networks.
  • Memory requirements: CKCNNs require memory linear in sequence length because they store convolution responses at every layer and input position.Applying the convolution theorem further increases operation memory consumption.
  • Selection of ω0: Performance is highly sensitive to the SIREN frequency prior ω0, making hyperparameter search costly.On pMNIST, performance varies from 98.54 to 65.22 as ω0 ranges from 1 to 100.

7 CONCLUSION AND FUTURE WORK

CKConv models global long-term dependencies with parameter efficiency while supporting irregular sampling and varying resolutions. The paper reports state-of-the-art results and identifies memory use, hyperparameter sensitivity, and kernel parameterization as remaining concerns.

  • CKCNNs achieve state-of-the-art results on multiple datasets while often surpassing architectures designed for irregularly sampled data.
  • CKConv models global long-term dependencies effectively in a parameter-efficient manner and handles irregularly sampled data and different resolutions.
  • CKCNNs are promising for applications involving global long-term dependencies, including audio, video, reinforcement learning, and autoregressive generative modeling.
  • Reducing CKConv memory consumption is important for broader deployment, especially on embedded devices.
  • Future work should seek kernel parameterizations that are more stable to hyperparameter changes and support additional properties such as smoothness.

REPRODUCIBILITY STATEMENT

The supplied passages describe the continuous-convolution treatment of irregular sampling and its unbiased Monte Carlo correction. They do not provide reproducibility-specific implementation details beyond the separate reproducibility passage.

  • CKConvs handle irregularly sampled and partially observed data because their kernel MLP can be sampled at arbitrary positions.
  • For very non-uniform sampling, kernel sampling can bias the convolution estimate, so the operation must account for the input sample distribution.
  • Inverse sample density weights contributions so densely sampled regions contribute less than sparsely sampled regions in the Monte Carlo estimate.
  • Setting s(τ) = 1 p(τ) gives an unbiased estimate of the continuous convolution.

A.2 DATA SAMPLED AT DIFFERENT SAMPLING RATES

The passages connect continuous convolutions with recurrent units, explain recurrent memory limitations, and examine initialization choices for function approximation. They also show how CKConvs can represent long, complex dependencies beyond fixed discrete horizons.

  • Recurrent kernels based on powers of W produce exponentially growing or decaying contributions, explaining exploding or vanishing gradients and short effective memory horizons.
  • Linear recurrent units are a subclass of CKConvs whose kernels are exponential functions, while general CKConv kernels can represent broader function classes.
  • CKConvs can define arbitrarily large memory horizons and model complex functions over an entire input when the horizon matches its length.
  • For uniformly distributed targets, uniformly distributed knots improve approximation, whereas conventional zero-centered initialization can produce poor reconstructions.
  • Uniform knot initialization is difficult to extend to exponentially many knots, can be unstable, and restricts allowable weights.
  • Replicating first-layer initialization across layers improves ReLU approximation across many target functions, but ReLU, LeakyReLU, and Swish still struggle with highly nonlinear, non-smooth functions.

B.2 KERNEL PARAMETERIZATION VIA Sine NETWORKS

Sine networks parameterize continuous kernels with periodic structure that supports high-fidelity approximation. The passages attribute their performance to periodic bending, Fourier-like expressivity, and suitable bias initialization.

  • Replacing ReLU with Sine nonlinearities lets continuous kernels approximate provided functions near perfectly and consistently improves all evaluated tasks.
  • Sine layers periodically bend space, giving infinitely many bias values that can place bends at arbitrary or multiple positions.
  • Periodic bias flexibility makes Sine parameterizations more robust to parameter selection and may support more reliable, faster approximation.
  • Sine networks can be viewed as using an exponentially large Fourier-like basis, with more terms supporting higher-fidelity approximation.
  • Uniformly initializing Sine biases across each component's period improves convergence speed and results.

C DATASET DESCRIPTION

The section describes benchmark datasets spanning long sequences, irregular sampling, language modeling, and function-approximation experiments, then evaluates Sine nonlinearities in CKCNNs. Sine is reported to outperform alternatives for reconstructing complex functions and solving the tested CKCNN tasks.

  • C DATASET DESCRIPTION: The benchmark suite includes long-sequence classification, speech, character trajectories, language modeling, and irregularly sampled sepsis prediction.Examples include 784-step sMNIST, variable-length CharacterTrajectories, PTB character modeling, and partially observed PhysioNet records.
  • C DATASET DESCRIPTION: Permuting MNIST pixels removes reliance on local features and makes long-term dependencies more important for classification.pMNIST randomly reorders the sMNIST sequences.
  • D.1 USING SINE NON-LINEARITIES OVER POPULAR ALTERNATIVES: Sine is the only tested nonlinearity reported to reconstruct very nonlinear, non-smooth functions, while ReLU, LeakyReLU, and Swish fail poorly.The comparison uses function-approximation experiments with otherwise varying nonlinearity choices.
  • D.1 USING SINE NON-LINEARITIES OVER POPULAR ALTERNATIVES: Sine nonlinearities outperform other tested nonlinearities in CKCNNs trained on sMNIST, pMNIST, SC, and SC_raw.The corresponding comparison uses ReLU, LeakyReLU, Swish, and Sine with the same backbone architecture.

D.2 GOING DEEPER WITH CKCNNS

Deeper CKCNNs do not improve performance over equally sized shallow models, while the analysis leaves open whether depth is poorly parameterized or unnecessary when shallow networks provide global memory horizons. The section also documents implementation choices and sampling-rate handling, including filtering high-frequency kernel components that can hurt higher-resolution evaluation.

  • D.2 GOING DEEPER WITH CKCNNS: Deep CKCNNs underperform shallow counterparts when architecture width and parameter count are held equal.The experiments compare deep and shallow models under matched capacity.
  • D.2 GOING DEEPER WITH CKCNNS: The results may indicate either that current parameterization fails to exploit depth or that depth is unnecessary when shallow networks provide global memory horizons.Both interpretations are presented as possible outcomes rather than established conclusions.
  • D.2 GOING DEEPER WITH CKCNNS: Normalized relative positions and implementation logic handle long sequences and sampling-rate discrepancies, including automatic correction of subsampling alignment.Relative positions are normalized to improve numerical stability as sequence lengths grow.
  • D.2 GOING DEEPER WITH CKCNNS: CKCNNs use Sine-parameterized continuous kernels within the architecture shown in Fig. 8, with Fourier Convolutions replacing spatial convolutions to accelerate computation.The kernel networks are described as vector-valued three-layer MLPs with 32 hidden units and weight normalization.
  • D.2 GOING DEEPER WITH CKCNNS: Continuous kernels can contain frequencies above the training-grid resolution, which hurt performance when evaluated at higher resolutions.The authors mitigate this effect by blurring kernels when the test sampling rate exceeds the training sampling rate.

E.4 HYPERPARAMETERS AND EXPERIMENTAL DETAILS

The experimental details specify dataset-specific model sizes, training configurations, and hyperparameter tables for the reported tasks. They also note sequence-length-dependent convergence and substantial overfitting on sCIFAR10 despite regularization attempts.

  • E.4 HYPERPARAMETERS AND EXPERIMENTAL DETAILS: Convergence time increases with sequence length: Copy Memory requires approximately 250 epochs at length 6000 versus as few as 10 epochs at length 100.Adding Problem converges faster at length 6000, requiring 38 epochs.
  • E.4 HYPERPARAMETERS AND EXPERIMENTAL DETAILS: CKCNNs for sMNIST, pMNIST, and sCIFAR10 use approximately 100K-parameter and 1M-parameter model variants.The corresponding hyperparameters are summarized in Table 8.
  • E.4 HYPERPARAMETERS AND EXPERIMENTAL DETAILS: The models heavily overfit sCIFAR10, and weight decay, dropout, and weight dropout were insufficient to counteract the overfitting.This is reported despite efforts to regularize the models.
  • E.4 HYPERPARAMETERS AND EXPERIMENTAL DETAILS: The section provides dataset-specific settings for CT, SC, SC_raw, irregularly sampled data, and character-level PennTreeBank modeling.Irregular-data configurations are listed separately in Table 9, while PennTreeBank uses 100-dimensional embeddings.
Loading 2102.02611v3…