PaperlayerView exact revision on arXiv

Source-linked AI summary

Liquid Structural State-Space Models

Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, Daniela Rus

arXiv:2209.12951v1cs.LGcs.AIcs.CLcs.CVcs.NE

TL;DR

Long-range sequence modeling needs expressive temporal credit assignment, while existing structural SSMs and liquid networks provide complementary strengths. The paper combines them in Liquid-S4, a linearized LTC model using S4 parametrization, and reports state-of-the-art generalization across multiple sequence tasks. Its headline results include an 87.32% average on Long Range Arena and strong performance with smaller parameter counts.

  • Problem

    Existing sequence models provide complementary capabilities for long-range learning, motivating a model that combines S4's structural efficiency with liquid networks' input-adaptive dynamics.

  • Method

    Liquid-S4 uses a linearized LTC state-space model with input-dependent transitions and solves it using S4's diagonal plus low-rank parametrization.

  • Results

    Liquid-S4 consistently outperforms S4, CNN, RNN, and Transformer variants across sequence tasks, including 87.32% average performance on Long Range Arena.

  • Takeaways & Limitations

    Liquid-S4 achieves state-of-the-art results across long-range sequence tasks, including image, text, audio, and medical time-series benchmarks.

Abstract

from arXiv · show

A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on a large series of long-range sequence modeling benchmarks. In this paper, we show that we can improve further when the structural SSM such as S4 is given by a linear liquid time-constant (LTC) state-space model. LTC neural networks are causal continuous-time neural networks with an input-dependent state transition module, which makes them learn to adapt to incoming inputs at inference. We show that by using a diagonal plus low-rank decomposition of the state transition matrix introduced in S4, and a few simplifications, the LTC-based structural state-space model, dubbed Liquid-S4, achieves the new state-of-the-art generalization across sequence modeling tasks with long-term dependencies such as image, text, audio, and medical time-series, with an average performance of 87.32% on the Long-Range Arena benchmark. On the full raw Speech Command recognition, dataset Liquid-S4 achieves 96.78% accuracy with a 30% reduction in parameter counts compared to S4. The additional gain in performance is the direct result of the Liquid-S4's kernel structure that takes into account the similarities of the input sequence samples during training and inference.

1. Introduction

Liquid-S4 combines the input-adaptive dynamics of liquid time-constant networks with the efficient structural parametrization of S4. The resulting model targets long-range sequence tasks and reports state-of-the-art performance across several modalities.

  • Motivation: Liquid networks are nonlinear state-space models with input-dependent transitions that adapt their dynamics to incoming inputs at inference.Their complexity is described as bottlenecked by numerical solution of differential equations.
  • Motivation: Structural SSMs use HiPPO history projection, diagonal plus low-rank transition parametrization, and efficient frequency-domain kernel computation.These mechanisms support sequence-memory and scalable convolutional computation.
  • Method: Liquid-S4 replaces the standard linear dynamics ˙x = A x + B u, y = C x with linearized LTC dynamics ˙x = (A + B u) x + B u, y = C x.The proposed system remains efficiently solvable with S4 parametrization and adds a convolutional kernel accounting for similarities between lagged signals.
  • Results: 87.32% average performance establishes state-of-the-art results on the six-task Long Range Arena benchmark.The paper also reports improvements across pixel-level classification, text, speech recognition, and time-series tasks.
  • Results: 96.78% accuracy on the full raw Speech Command dataset is achieved with a 30% parameter reduction compared with S4.The contribution summary also reports state-of-the-art performance on BIDMC vital signs.

2. Related Works

Related work spans recurrent and continuous-time approaches for learning long-range dependencies, alongside structural state-space models designed to improve memory and computational efficiency. These methods address expressive temporal credit assignment but face distinct optimization or computational challenges.

  • Recurrent sequence models: Discrete and continuous-time RNNs model sequence dynamics through persistent hidden states or ODE solvers with more elaborate temporal steps.Continuous-time RNNs unroll their dynamics using ordinary differential equation solvers.
  • Recurrent sequence models: Continuous-time RNNs have been studied for credit assignment on both regularly and irregularly sampled data, including through vector-field representations.The cited work connects these models with generalization and expressivity in sequential learning.
  • Optimization challenges: Gradient-descent-trained RNNs suffer from vanishing or exploding gradients, making long-term dependency learning difficult.The related literature includes gated, continuous, and ODE-based recurrent alternatives.
  • Optimization challenges: Proposed responses to gradient problems include LSTMs, GRUs, continuous gating, hawk LSTMs, IndRNNs, state regularization, unitary RNNs, and dilated RNNs.These approaches modify gating, recurrence, regularization, or temporal resolution to address long-range learning.
  • Structural state-space models: S4 structural state-space models combine HiPPO-based higher-order projections, diagonal plus low-rank transition matrices, learnable input transitions, and efficient Cauchy-kernel computation.Together, these components are used to memorize input sequences efficiently.

3. Setup and Methodology

The methodology combines continuous-time liquid state-space dynamics with structural SSM parameterization and convolutional computation. Liquid-S4 augments ordinary SSM kernels with input-correlation kernels while retaining efficient computation through S4-based parameterizations.

  • SSM background: Continuous-time SSMs map inputs through latent states using transition, input, and output matrices, then discretize the dynamics for sequence-to-sequence modeling.The discretized model maps u_k to y_k through hidden state x_k, with the direct term D omitted as a skip connection can be added later.
  • SSM background: Unrolling the discretized SSM converts sequential state updates into a convolutional kernel that maps the input sequence to outputs.This representation avoids the non-scalable sequential training procedure and can be evaluated using an efficient kernel computation pipeline.
  • Liquid-SSM formulation: Liquid-S4 linearizes LTC dynamics so the state-transition coefficient depends on the input, enabling more complex dynamics than input-independent linear SSMs.The model uses bilinear dynamics, discretizes them with a trapezoidal rule, and then constructs their convolutional representation.
  • Liquid kernel: Input-dependent transitions produce two weight types: ordinary per-time-step mappings and higher-order input auto-correlation terms.The latter motivate an additional liquid kernel and yield an input-output mapping of the form y = K ∗u + K_liquid ∗u_correlations.
  • Liquid kernel: The liquid kernel captures correlations among incoming samples up to a selected order, with P = 4 including second-, third-, and fourth-order products.For a length-L signal, the correlation vector contains combinations such as u_i u_j and u_i u_j u_k.
  • Efficient computation: S4 parameterization enables efficient liquid-kernel computation, while the simplified PB kernel replaces A with the identity for correlation terms.Both KB and PB modes have complexity O~(N + L + p_max L̃), and typical liquid orders satisfy p_max = 3 with L̃ much smaller than sequence length.

4. Experiments with Liquid-S4

Liquid-S4 is evaluated across long-range sequence, biomedical time-series, image, and speech tasks against broad SSM, Transformer, RNN, and convolutional baselines. It consistently achieves strong results, including 87.32% average accuracy on LRA and best reported performance on several task settings.

  • Long Range Arena: Liquid-S4 instances with liquid order p=2–6 outperform all baselines across six LRA tasks, reaching 87.32% average accuracy.It improves over S4-LegS by more than 3% on ListOps, 2.2% on character-level IMDB, and 0.65% on CIFAR, while reaching 96.54% on the hardest LRA task.
  • Long Range Arena: Liquid-S4 kernels can use state sizes as small as seven units on ListOps and IMDB, reducing parameter counts relative to other variants.The reported performance also improves consistently as liquid order increases on these two tasks.
  • BIDMC Vital Signs: Liquid-S4 outperforms S4 and S4D variants on all three BIDMC vital-sign prediction tasks.The reported improvements over S4-LegS are 36% for respiratory rate, 26.67% for SpO2, and 8.7% for heart rate.
  • 1-D Pixel-level Image Classification: Liquid-S4 with a PB kernel of order p=3 outperforms S4, S4D, Transformer, and RNN baselines on the sCIFAR image-classification task.sCIFAR classifies flattened images represented as 1024-long sequences into 10 classes.
  • Speech Commands: On full raw Speech Commands, Liquid-S4 with liquid order p=2 achieves the best performance among the evaluated benchmarks on the 16KHz full-dataset testbed.It remains competitive but is not best in the half-frequency zero-shot experiment; the authors suggest covariance terms in the liquid kernel may hurt performance slightly there.
  • Speech Commands: On the reduced ten-class SC10 benchmark, Liquid-S4 achieves 98.51% accuracy with the same hyperparameters.The paper distinguishes SC10 from the full 35-label Speech Commands benchmark.

5. Conclusions

Liquid-S4 improves structural state-space models by adding a linear liquid time-constant kernel that captures similarities between input time-lags. With smaller parameter counts, it achieves state-of-the-art results across long-range, biomedical, image, and speech tasks.

  • Conclusions: Liquid-S4 considerably improves structural state-space model performance through a linear liquid time-constant kernel.The kernel computes similarities between time-lags in addition to using S4’s diagonal plus low-rank parametrization.
  • Conclusions: Liquid-S4 achieves state-of-the-art performance on all six Long-Range Arena tasks with smaller parameter counts.The conclusion also reports state-of-the-art results on BIDMC heart rate, respiratory rate, blood oxygen saturation, sequential 1-D pixel classification, and Speech Commands.

S1. Proof of Proposition 1

The proof constructs liquid kernels by transforming S4 convolution kernels to account for correlations between lagged input signals. The same construction extends from second-order correlations to higher liquid orders.

  • S1. Proof of Proposition 1: The liquid kernel is obtained by multiplying the S4 convolution kernel components by B and applying an anti-diagonal transformation.This construction produces the kernel used for liquid state-space dynamics.
  • S1. Proof of Proposition 1: For p = 2, the S4 kernel is multiplied by B to represent second-order correlations before forming the resulting kernel.The proof explicitly constructs the order-2 correlation kernel from the S4 terms.
  • S1. Proof of Proposition 1: The order-2 liquid kernel is formed by flipping the resulting kernel before convolution with two-term correlation components.The flipped sequence contains the corresponding reversed products of C, A, and B terms.
  • S1. Proof of Proposition 1: The same procedure yields liquid kernels for higher liquid orders, completing the proposition.The proof generalizes the order-2 construction to higher-order correlations.

S2. Hyperparameters

The supplementary settings specify learning-rate, time-step, modeling-direction, and hyperparameter considerations for Liquid-S4 experiments. The accompanying tables document hyperparameters and Speech Command evaluation conventions.

  • S2. Hyperparameters: Liquid-S4 generally requires a smaller learning rate than S4 and S4D blocks.This is an observed training-setting difference reported for the experiments.
  • S2. Hyperparameters: All experiments set ∆tmax to 0.2, while ∆tmin is chosen proportional to 1 seq length following prior recommendations.The passage specifies the time-step settings used across experiments.
  • S2. Hyperparameters: Liquid-S4 works better as a causal model without bidirectional configuration.This is reported as an empirical modeling-direction observation.
  • S2. Hyperparameters: Table S1 reports hyperparameters for best-performing models and defines BN, LN, and WD as normalization and regularization abbreviations.BN denotes Batch Normalization, LN Layer normalization, and WD Weight decay.
  • S2. Hyperparameters: Table S2 reports validation accuracy for the reduced ten-class RAW Speech Command dataset, with x marking infeasible computation or non-applicability.Liquid-S4 uses the same hyperparameters as for the full Speech Commands dataset.
Loading 2209.12951v1…