PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes Convolutional Models Great on Long Sequence Modeling?

Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, Debadeepta Dey

arXiv:2210.09298v1cs.LGcs.AIcs.CVstat.ML

TL;DR

Long-range dependency modeling remains challenging because recurrent, attention, and local convolutional models each face limitations. This paper extracts efficient parameterization and decaying weights as kernel principles, then proposes SGConv; it reports strong results on LRA and Speech Commands and potential use in language and vision models.

  • Problem

    Long-range dependency is difficult to model: RNNs face vanishing gradients, Transformers scale quadratically with sequence length, and CNNs usually have local receptive fields.

    Source: §1 · p. 2

  • Method

    The paper identifies efficient parameterization and decaying kernel weights as principles, then builds SGConv from multiscale sub-kernels combined with decaying weights.

    Source: §1 · pp. 2–3

  • Results

    SGConv achieves state-of-the-art results on LRA and improves S4 by more than 1%, with comparative 10-class and significantly better 35-class Speech Command results.

    Source: §1 · p. 3, §4.2 · p. 8

  • Takeaways & Limitations

    The authors present SGConv as a simpler non-SSM global convolution model and suggest its use as a general-purpose module across domains.

    Source: §1 · p. 3, §5 · p. 12

    S4’s empirical success relies on delicate parameterization and initialization schemes, including numerical linear algebra tricks and a HiPPO-based initialization.

    Source: §1 · p. 2

Abstract

from arXiv · show

Convolutional models have been widely used in multiple domains. However, most existing models only use local convolution, making the model unable to handle long-range dependency efficiently. Attention overcomes this problem by aggregating global information but also makes the computational complexity quadratic to the sequence length. Recently, Gu et al. [2021] proposed a model called S4 inspired by the state space model. S4 can be efficiently implemented as a global convolutional model whose kernel size equals the input sequence length. S4 can model much longer sequences than Transformers and achieve significant gains over SoTA on several long-range tasks. Despite its empirical success, S4 is involved. It requires sophisticated parameterization and initialization schemes. As a result, S4 is less intuitive and hard to use. Here we aim to demystify S4 and extract basic principles that contribute to the success of S4 as a global convolutional model. We focus on the structure of the convolution kernel and identify two critical but intuitive principles enjoyed by S4 that are sufficient to make up an effective global convolutional model: 1) The parameterization of the convolutional kernel needs to be efficient in the sense that the number of parameters should scale sub-linearly with sequence length. 2) The kernel needs to satisfy a decaying structure that the weights for convolving with closer neighbors are larger than the more distant ones. Based on the two principles, we propose a simple yet effective convolutional model called Structured Global Convolution (SGConv). SGConv exhibits strong empirical performance over several tasks: 1) With faster speed, SGConv surpasses S4 on Long Range Arena and Speech Command datasets. 2) When plugging SGConv into standard language and vision models, it shows the potential to improve both efficiency and performance.

1 Introduction

The section motivates global convolution as a route to long-range dependency modeling and asks how to simplify S4’s involved design. It proposes efficient, decaying kernels as the principles behind SGConv, which reports strong results across sequence tasks and broader domains.

1 Introduction

  • RNNs face vanishing gradients, Transformers have quadratic sequence-length complexity, and CNNs usually have only local receptive fields.These limitations make long-range dependency a key challenge in tasks including forecasting, language modeling, and pixel-level image generation.Source: §1 · p. 2
  • S4 demonstrates that global convolution can model long-range dependencies, but its delicate parameterization and initialization make it difficult to use.Its efficient implementation relies on Diagonal Plus Low Rank parameterization, numerical linear algebra tricks, and an initialization scheme based on the HiPPO matrix.Source: §1 · p. 2
  • Effective global kernels need parameter counts that scale slowly with sequence length and decaying weights that favor closer neighbors.The paper argues that excessive parameters can cause overfitting and that S4 exhibits decay through the exponential decay of matrix-power spectra.Source: §1 · pp. 2–3
  • SGConv achieves state-of-the-art results on LRA, improves S4 by more than 1%, and reports comparative 10-class and significantly better 35-class Speech Command results.The paper also reports improved efficiency over S4 and promising language-modeling and sentence-classification performance; on large-scale image classification, SGConv matches ConvNext.Source: §1 · p. 3
  • SGConv combines multiscale sub-kernels, each upsampled from the same number of parameters, with smaller weights at larger scales.Each larger sub-kernel doubles in size, so the parameter count grows logarithmically with input length.Source: §1 · p. 3, Figure 1 · p. 3

2 Related work

Related work addresses the difficulty of efficient long-range modeling through efficient Transformers, re-parameterized convolution kernels, and state space models. The paper situates SGConv as a global-kernel parameterization that targets long-range dependencies beyond prior short-range convolutional approaches.

Efficient Transformers

  • Efficient Transformer variants address quadratic computation and memory costs, but few perform well on long-range benchmarks such as LRA and SCROLLS.The cited benchmarks require long-range modeling ability.Source: §2 · pp. 3–4

(Re-)parameterization

  • Prior structural re-parameterization work mainly targets vision and short-range kernels, while SGConv parameterizes global kernels for long-range dependencies.The related work notes one exception scaling convolution to 31 × 31 with an optimized CUDA kernel.Source: §2 · p. 4

State Space Models

  • S4 implements an SSM as global convolution through specialized matrix parameterization and kernel computations, but remains theoretically involved and requires accelerator-specific optimization.The cited implementation uses Cauchy-kernel diagonalization and low-rank corrections with the Woodbury identity.Source: §2 · p. 4

3 Design of Global Convolutional Models

The design section identifies efficient parameterization and kernel decay as principles for global convolution, then combines them in SGConv’s multiscale construction.

3.1 Design Principles

  • Global convolution kernels need parameter counts that grow slowly with sequence length, avoiding the impractical cost of directly parameterizing every position.A standard depth-wise kernel for a 16K-length sequence would require 4M parameters at 256 channels.Source: §3.1 · p. 5
  • Kernel weights should decay with distance, assigning more weight to close neighbors; S4 exhibits this structure across Pathfinder-X and Speech Command.The paper attributes S4’s decay to the exponential decay of matrix-power spectra.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2 SGConv

  • SGConv combines multiscale sub-kernels of increasing sizes, upsampled from the same number of parameters, with weighted combinations.A decay coefficient controls the decay speed, while a normalization constant preserves the input scale.Source: §3.2 · pp. 5–6
  • On LRA, SGConv achieves significant improvement over previous methods, including a 1% average-accuracy gain over well-tuned S4 variants.The table caption also describes SGConv as faster, referring readers to Table 2 for the speed comparison.Source: Table 1 · p. 6
  • SGConv uses O(log L) parameters and FFT convolution in O(L log L) time, with its decay coefficient usually set to 1/2.The kernel is normalized to unit norm at initialization, and the normalization is fixed during training.Source: §3.2 · p. 6

4 Experiments

Experiments evaluate SGConv on Long Range Arena (LRA) and Speech Commands, test its kernel design, and explore integration into language and vision models. SGConv improves over tuned S4 variants on LRA, outperforms S4 in reported speed comparisons, and performs better than SSM-based methods on most Speech Commands tasks.

4 Experiments

  • Replacing half of Transformer layers with SGConv blocks preserves performance while reducing those layers' complexity from O(L^2) to O(L log L); replacing ConvNeXt's 7 × 7 convolutions yields comparative or better ImageNet performance.The experiments explore SGConv as a general-purpose component for long-range dependencies in standard models.Source: §4 · p. 6

4.1.1 Results

  • SGConv improves average LRA accuracy by 1% over well-tuned S4 variants.The reported comparison is against variants introduced in Gu et al. [2022b].Source: §4.1.1 · p. 7

4.1.2 Ablation Study on IMDB

  • The decay structure is crucial for IMDB performance, while scale dimension d matters less than decay speed t within a reasonable range.Performance drops as d increases from 8 to 64, which the authors attribute to overfitting.Source: Figure 3 · p. 7, §4.1.2 · p. 8

4.1.3 Speed Comparison

  • SGConv is about 50% faster than vanilla S4 and 15% faster than S4's optimized CUDA kernel implementation.The comparison covers inference and backpropagation time on CPU and GPU with 128 channels and batch size 64.Source: Table 2 · p. 7, §4.1.3 · p. 8

4.2 Speech Commands

  • On Speech Commands, SGConv performs better than SSM-based methods on 4 of 5 tasks, including a +2.40% result over existing SoTA on the original 35-class task.For raw-sequence classification on the 35-class task, its accuracy is described as marginally higher.Source: §4.2 · p. 8

4.3 Further Applications of SGConv

The section evaluates SGConv as a drop-in component in language and vision models. It reports competitive or improved task performance alongside efficiency benefits, with image-classification results compared against established vision models.

4.3.1 Language Tasks

  • The language experiments combine SGConv and short-range attention, replacing half the attention blocks and testing layer arrangements with SGConv in deeper positions.The tested WikiText-103 models process 3,092-token inputs directly with SGConv.Source: §4.3 · p. 9, §4.3.1 · p. 9
  • At sequence length 1024, SGConv is ∼2.1X faster than Attention and uses significantly less memory as sequence length increases.The SGConv block has O(L log(L)) time and space complexity.Source: §4.3.1 · p. 9, Table 4 · p. 10
  • The 16L model improves on its baseline by -1.17 perplexity; the 18L model reaches 18.70 perplexity and improves on Transformer-XL by -0.05.Both results are reported on WikiText-103; the 18L comparison uses the same setting and a batch size of 16.Source: §4.3.1 · p. 10, Table 5 · p. 10
  • SGConvBERT achieves comparable performance to BERT on the tested GLUE tasks while using a more efficient SGConv layer.Tasks with fewer than 5K training samples were excluded.Source: §4.3.1 · p. 10, Table 6 · p. 11

4.3.2 Image Classification

  • SGConvNeXt achieves better results than EfficientNets and Swin Transformers on ImageNet-1k, with accuracy compared against FLOPs and throughput.The model replaces ConvNeXt’s 7 × 7 kernels with SGConv and treats 2D features as a 1D sequence.Source: §4.3.2 · pp. 11–12, Figure 5 · p. 12

5 Discussion

The discussion distills two principles for effective global convolution and presents SGConv as a simpler, non-SSM-based model. It connects these principles to related approaches and suggests global convolution as a general-purpose direction for long-range dependency.

5 Discussion

  • The paper presents SGConv as a simple, intuitive global convolution model based on two principles, with the aim of making global convolution easier to understand and try.The discussion contrasts this non-SSM approach with Gu et al. [2022a], whose diagonal state-transition proposal still uses nuanced parameterization and initialization.Source: §5 · p. 12
  • The exponential moving-average layer in Ma et al. [2022] is identified as a global convolution that satisfies both principles and has promising long-sequence results.Source: §5 · p. 12
  • The authors expect similar global-convolution modules to emerge as long-range dependency becomes increasingly critical for sequence modeling.Source: §5 · p. 12

A.1 Long Range Arena

The LRA appendix details kernel settings and task-specific modifications. Most experiments use concatenated sub-kernels and α = 1/2, while image and Pathfinder-X experiments use distinct parameterization or initialization choices.

A.1 Long Range Arena

  • LRA experiments use concatenated sub-kernels, mildly tune each scale’s dimension, and increase dropout on some tasks to prevent overfitting.The appendix points to Table 7 for the detailed hyperparameters.Source: §A.1 · p. 18, Table 7 · p. 18
  • For flattened 2D image experiments, the authors use disentangled parameterization and combination weights, with decay speed 1/pos.They hypothesize that images require a more subtle inductive bias.Source: §A.1 · p. 18
  • For Pathfinder-X, cosine-wave channel initializations and channel-specific α values from 1 to 1/3 improve results by about 1% versus fixed α = 1/2 and random initialization.Source: §A.1 · p. 18

A.2 Speech Command

The Speech Command appendix specifies the training-setting sources for its two classification variants and gives the SGConv scale dimension.

A.2 Speech Command

  • The 10-class Speech Command experiment uses the training setting from Gu et al. [2021a]’s earlier GitHub repository version.Source: §A.2 · p. 18
  • The 35-class Speech Command experiment uses the training setting from the other cited GitHub repository.Source: §A.2 · p. 18
  • The SGConv scale dimension is 32 for these Speech Command experiments.Source: §A.2 · p. 18

A.3 Language Task

The language-task implementation uses SGConv with a sequence length of 3072 in both model configurations, while SAttention uses different attention and memory lengths.

A.3 Language Task

  • Both the 16-L and 18-L models use an SGConv sequence length of 3072.The SGConv scale dimension is 96.Source: §A.3 · p. 18
  • SAttention uses attention and memory lengths of 192 in the 16-L model and 384 in the 18-L model.Source: §A.3 · p. 18
  • The training settings follow the referenced project, except that the batch size is reduced to 64.Source: §A.3 · p. 18

A.4 Image Classification

The image-classification section describes SGConvNeXt’s setup and visualizations, and identifies a table comparing ImageNet-1k Top-1 accuracy with prior work.

A.4 Image Classification

  • SGConvNeXt uses training settings from Liu et al. [2022]4, with scale fixed to 5 and scale dimensions based on each layer’s flattened feature length.Source: §A.4 · p. 19
  • Figure 7 shows the SGConvNeXt structure, while Figures 8 and 9 show its ImageNet-1k intermediate features and kernels across stages.Source: §A.4 · p. 19, Figure 7 · p. 20, Figure 8 · p. 20
  • Table 8 compares ImageNet-1k Top-1 accuracy with state-of-the-art works.Source: Table 8 · p. 19
  • The provided block labels identify a ConvNeXt block and an SGConv block.Source: §A.4 · p. 20

Related papers

Loading 2210.09298v1…