PaperlayerView exact revision on arXiv

Source-linked AI summary

Liquid Gated Attention

Yiheng Jiang, Yuanbo Xu, Yongjian Yang

arXiv:2608.30695v1cs.LG

TL;DR

Irregular sampling and extended horizons require continuous-time modeling without the sequential integration cost of solver-based methods, while existing solver-free approaches lack explicit coupling between observed intervals and input-driven state modulation. LGA addresses this with solver-free liquid gating and fast-weight associative memory, using parallel causal and non-causal computation with linear temporal complexity. Built into LFormer, it achieves competitive performance across six tasks and sixteen datasets, while limitations remain for online normalization, highly sparse noisy conditions, and non-stationary extrapolation.

  • Problem

    Existing methods do not simultaneously incorporate observed time intervals, enable parallel computation, and maintain robustness to sparse and noisy observations.

  • Method

    LGA combines input-driven liquid gating parameterized by observed intervals with fast-weight associative memory and sequence-level normalization in a solver-free parallel operator.

  • Results

    Across six tasks and sixteen datasets, LFormer delivers strong results in long-range dependency modeling, fine-grained state tracking, and trajectory reconstruction from sparse and noisy observations.

  • Takeaways & Limitations

    LGA provides a continuous-time-inspired parallel sequence architecture, positioning LFormer as an efficient backbone for time-series representation learning.

  • Takeaways & Limitations

    Sequence-level normalization assumes offline full-sequence access, learned interpolation can overfit highly sparse noisy data, and extrapolation is constrained for non-stationary signals.

Abstract

from arXiv · show

Real-world time series often exhibit irregular sampling and extended temporal horizons, requiring models to capture continuous-time dynamics across arbitrary intervals without prohibitive scaling costs. Discrete-time methods collapse variable time intervals into static positional steps; solver-dependent continuous-time models preserve temporal structure but rely on sequential integration, precluding parallelization; and solver-free approximations avoid this cost yet none couples observed time intervals with input-driven state modulation. We propose Liquid Gated Attention (LGA), a solver-free parallel temporal operator. By parameterizing an input-driven gating mechanism with observed time intervals, LGA introduces a continuous-time inductive bias and formulates hidden state evolution as a fast-weight associative memory, enabling parallel computation across the temporal dimension. Using matrix associativity in non-causal encoding and a prefix scan in causal encoding, LGA attains linear temporal complexity in sequence length in both modes. A sequence-level normalization bounds cumulative temporal decay for stable long-horizon optimization. Building on LGA, we instantiate LFormer, a modular backbone for continuous-time representation learning. Across six tasks and sixteen datasets spanning up to 17,984 steps, LFormer demonstrates long-range dependency modeling, fine-grained state tracking, and trajectory reconstruction from sparse and noisy observations, while delivering competitive performance against state-of-the-art discrete-time and continuous-time baselines with linear scaling efficiency.

I. INTRODUCTION

Irregular sampling and long horizons expose a gap between discrete-time models that discard temporal distances, solver-based continuous-time models that require sequential integration, and solver-free methods that lack explicit temporal-state coupling. LGA addresses this gap with a solver-free, parallel continuous-time operator, while LFormer provides a modular backbone validated across diverse temporal tasks.

  • Motivation: Irregular sampling requires modeling continuous-time dynamics across arbitrary observed intervals, while long sequences require efficient long-range dependency modeling.Real-world measurements may also be sparse and noisy, requiring trajectory recovery from imperfect observations.
  • Limitations of Existing Paradigms: Discrete-time methods collapse variable intervals into static positional steps and discard temporal distances needed for continuous-time dynamics.RNNs and Transformers are representative discrete-time approaches.
  • Limitations of Existing Paradigms: Solver-dependent continuous-time models preserve temporal structure and can smooth high-frequency noise, but their forward passes rely on sequential integration.This dependence limits parallel computation across long sequences.
  • Research Gap: Solver-free alternatives avoid solver overhead but commit dynamics to fixed forms and lack a parallel architecture coupling observed intervals with input-driven state modulation.The resulting gap can compromise robustness under sparse and noisy conditions.
  • Proposed Approach: LGA unifies time-interval-coupled liquid gating, a trapezoidal-rule-inspired surrogate, fast-weight associative memory, and sequence-level normalization in a solver-free operator.Its causal and non-causal computations retain linear temporal complexity, using a prefix scan or matrix associativity respectively.
  • Empirical Validation: LFormer instantiates LGA in a residual modular backbone and achieves competitive performance across six tasks and sixteen datasets while modeling long-range dependencies, fine-grained states, and sparse noisy trajectories.The architecture directly processes discrete observations while inheriting LGA’s continuous-time inductive bias and linear complexity.

III. BACKGROUND: LIQUID TIME-CONSTANT GATING

LGA derives continuous-time, input-driven gating from liquid time-constant dynamics, then replaces numerical integration with a learnable endpoint interpolation. The resulting gate preserves temporal decay and input adaptability while enabling a solver-free formulation.

  • Continuous-Time Gating: Liquid time-constant dynamics adapt effective inverse time constants to incoming signals, allowing state evolution to reflect varying temporal dynamics.The one-dimensional LTC configuration isolates a scalar state to obtain an analytically tractable formulation.
  • Continuous-Time Gating: The liquid gate combines intrinsic temporal decay over an observed interval with input-driven modulation through an exponential mapping.This yields a bounded gate and provides a continuous-time inductive bias with noise-attenuation properties.
  • A Learnable Integral Approximation: LGA approximates the integral-based liquid gate using a learnable weighted average of two consecutive endpoint representations.The interval is δ_n = t_n − t_{n−1}, and the interpolation coefficient is constrained to µ = σ(θ) ∈ (0, 1).
  • A Learnable Integral Approximation: When µ = 0.5, the endpoint interpolation recovers the classical trapezoidal rule with second-order local accuracy; otherwise it acts as a trainable numerical surrogate.The general learnable configuration is not an algebraically exact fixed quadrature rule.
  • Computational Efficiency: The efficient liquid gate preserves temporal decay and input adaptability while eliminating numerical integration.This completes the computational-efficiency step of LGA.

2) An Expressive Hidden State:

LGA expands the scalar liquid-time-constant state into a matrix-valued associative memory, increasing storage capacity for long-range modeling and enabling parallel sequence computation. Its causal and non-causal forms use prefix scans or global aggregation, while output gating filters retrieved representations.

  • An Expressive Hidden State: LGA elevates the hidden state to S_n ∈ R^{d×d}, expanding memory storage for long-range modeling.An input-dependent associative memory term replaces the static bias to track instantaneous feature variations.
  • An Expressive Hidden State: Fast-weight programming updates the associative memory through vector outer products, creating a d×d interaction matrix for expressive sequence modeling.This matrix-valued state fulfills LGA’s expressive-capability design step.
  • Output Retrieval: The output gate modulates retrieved context before producing the final output, allowing irrelevant or noisy channels to be suppressed.This filtering improves robustness against high-frequency observational perturbations.
  • Parallel Computation: Matrix associativity removes sequential dependencies in the recurrent state and yields linearly scaling causal and non-causal parallel modes.Liquid gates explicitly scale query, key, and value representations after being computed across the temporal dimension.
  • Parallel Computation: Causal LGA uses an associative prefix scan to preserve no-future-information semantics with computational and memory complexity O(nd^2).Non-causal LGA instead aggregates over the full sequence for offline bidirectional encoding.
  • Parallel Pipeline: Figure 2’s pipeline projects X, scales representations with liquid gates, aggregates them through causal or global attention, and applies an output gate.The notation includes element-wise products, row-wise vector-matrix multiplication, row-wise outer products, and temporal cumulative sums.

1) Sequence-level Normalization:

Sequence-level normalization constrains cumulative temporal decay to improve numerical stability over long sequences, while preserving relative temporal-distance allocation. The normalized gates are a regularized surrogate rather than an exact algebraic equivalent of the original LTC transition.

  • Unconstrained cumulative decay can cause exponential underflow and vanishing gradients as sequence length grows.
  • The stabilization normalizes the nonnegative decay vector by its sequence-wise sum plus a small constant before computing local and cumulative gates.
  • The normalized cumulative sum lies within (0, 1], constraining exponential scaling coefficients to (e^-1, 1].
  • The mechanism preserves relative temporal-distance allocation while bounding total decay, reducing gradient decay from exponential to polynomial growth with sequence length.
  • A Decoupled Multi-Head Extension: The decoupled multi-head extension applies independent projections, interpolations, and normalization across H subspaces before concatenating and projecting their outputs.

C. LFormer: An Instantiation of LGA

LFormer instantiates LGA as a modular representation-learning backbone that embeds observations, processes them through stacked Liquid Mixers, and maps the resulting representation to task outputs. Its evaluation spans six temporal tasks and sixteen datasets, examining performance, component contributions, and computational efficiency.

  • LFormer uses a three-stage pipeline of a task-specific Embedder, L stacked Liquid Mixers, and a task-oriented Predictor.
  • 1) Embedder: The Embedder maps raw observations from R^m into a d-dimensional latent sequence representation.
  • 2) Liquid Mixer: Each Liquid Mixer combines multi-head liquid gated attention for temporal processing with SwiGLU channel mixing in a pre-normalized residual structure.
  • The terminal mixer representation is passed to task-specific prediction heads, including linear projections, global pooling, or multilayer perceptrons.
  • V. EXPERIMENTS AND DISCUSSION: The experiments evaluate comparative performance, component contributions, and computational efficiency across six tasks and sixteen datasets.

A. Experimental Setup

LFormer is evaluated across six temporal modeling tasks and sixteen datasets against discrete-time and solver-dependent continuous-time baselines. Results cover long-range dependency modeling, fine-grained state tracking, and trajectory reconstruction under irregular, sparse, and noisy observations.

  • Benchmark: Six temporal modeling tasks spanning sixteen datasets assess long-range dependencies, fine-grained state tracking, and trajectory reconstruction.The benchmark includes LTS-C, LTS-R, PTS-C, PTS-R, TS-I, and TS-E.
  • Baselines: LFormer is benchmarked against discrete-time architectures and solver-dependent continuous-time models.The discrete-time baselines include GRU, Transformer, LRU, Mamba, GRU-∆t, GRU-D, and RKN-∆t; solver-dependent models include neural differential-equation, path-integral, attention, and filtering approaches.
  • Long-range Dependency Modeling (LTS-C and LTS-R): 71.3% average accuracy and 1.33 average rank are reported on LTS-C, with top performance on four of six UEA datasets.LFormer finishes in the top two on every evaluated dataset, while the canonical Transformer encounters OOM exceptions on the 17,984-step EW dataset.
  • Long-range Dependency Modeling (LTS-C and LTS-R): 47.72%, 20.43%, and 71.57% relative MAE reductions are reported for HR, RR, and SpO2, respectively, against the strongest baseline.LFormer consistently outperforms all baselines on the three BIDMC regression datasets in both MAE and RMSE.
  • Fine-grained Dynamics Tracking (PTS-C and PTS-R): 92.9% accuracy on HA and 2.1‰ MSE on PDL quantify point-wise state tracking performance.LFormer improves accuracy over ACSSM by 1.6% on HA and reduces relative MSE over ACSSM by 31.2% on PDL.
  • Trajectory Reconstruction (TS-I and TS-E): 65.3% and 83.3% MSE reductions are reported for TS-I on PHY and USH, while TS-E improves USH by 10.73% but trails mTAND on PHY by 9.41%.On synthetic spirals, LFormer achieves the best MAE and RMSE across all three variants in both interpolation and extrapolation, with larger margins as α decreases.

C. Component Analysis

The component analysis evaluates architectural, liquid-gate, and stabilization variants across benchmark settings. Results identify head specialization, output gating, interpolation choices, input-aware modulation, and normalization as task-dependent contributors.

  • Macroscopic Architecture: Removing MHLGA causes minor classification degradation but substantially larger regression losses, while all macroscopic variants collapse on the highly sparse PDL dataset.Classification changes are LTS-C: -2.62% versus -2.67% and PTS-C: -3.15% versus -2.15%; the cited passage attributes the regression gap to MHLGA.
  • Macroscopic Architecture: Output gating causes the largest macroscopic drop on TS-E (-14.65%) and training collapse on PDL, but has smaller effects on LTS-R (-4.19%) and LTS-C (-2.02%).The output gate is described as a nonlinear filter for retrieved memory, especially relevant when retrieved states are unreliable.
  • Liquid Gate Dynamics: Decoupled interpolation parameters consistently improve performance across tasks, whereas sharing a global µ causes training collapse on PTS-R.The result links head-specific interpolation to multi-scale temporal dynamics.
  • Liquid Gate Dynamics: Fixing µ = 0.5 improves PTS-R MSE by 23.90% but generally degrades other tasks, suggesting a stability-flexibility trade-off under sparse noisy observations.The fixed value recovers the symmetric trapezoidal prior, while learned interpolation can overfit local noise on PTS-R.
  • Empirical Diagnostics: The complexity figures compare peak GPU memory and training-step latency as sequence length grows, with OOM and OOT marking memory and time limits.Figure 7 tracks training-loss convergence and out-of-sample accuracy during stabilization ablation.
  • Liquid Gate Dynamics: Replacing input-aware gating with deterministic time decay degrades regression by LTS-R: -15.83%, TS-I: -12.30%, and PTS-R: -39.02%, while classification changes remain small.Classification effects are LTS-C: -2.00% and PTS-C: -1.15%.

3) Numerical Stabilization:

Sequence-level normalization stabilizes cumulative temporal decay, while LFormer’s parallel implementations achieve linear scaling in causal and non-causal modes. The model occupies a strong accuracy-efficiency position, with limitations for online normalization and offline signature comparisons.

  • Numerical Stabilization: Removing normalization degrades average performance on LTS-R (-8.22%), LTS-C (-3.10%), TS-I (-32.88%), and TS-E (-6.44%), and causes several training collapses.On PTS-C, the ablation exhibits immediate gradient explosion and remains near random-guessing accuracy.
  • Numerical Stabilization: Bounding cumulative decay prevents numerical overflow or vanishing gradients caused by unconstrained prefix sums and exponentiation, especially on long or irregular sequences.The mechanism is less necessary on shorter or regularly sampled tasks, where other components can partially compensate.
  • Empirical Complexity Scaling: Both causal and non-causal LFormer variants scale linearly in peak memory and temporal latency, avoiding the Transformer’s quadratic memory bottleneck.The Transformer exceeds GPU capacity at 2^13 steps, while R-ODE exceeds 10^4 ms at 2^11 steps and is marked OOT.
  • Hardware Efficiency: LFormer matches the Transformer’s parameter footprint and achieves substantially higher throughput, although Mamba remains faster partly because it uses hardware-optimized scan kernels.The throughput comparison uses identical configurations with sequence length 4,096.
  • Accuracy-Efficiency Trade-off: Offline RFormer gains throughput by precomputing signatures and reducing effective sequence lengths to 200 on MI and 10 on EW, assuming access to the full sequence.Its online variant has lower throughput, making the comparison subject to an offline preprocessing advantage.
  • Accuracy-Efficiency Trade-off: LFormer traces the leading Pareto frontier on MI and EW, achieving highest accuracy on MI and second-highest accuracy on EW with negligible throughput degradation.The comparison plots classification accuracy against logarithmic inference throughput.
  • Limitations and Future Work: Sequence-level normalization assumes offline full-sequence access, so online or causal inference would require a prefix-normalized variant.This is identified as a current LGA limitation.
  • Conclusion: The paper concludes that LGA embeds continuous-time principles into parallel sequence architectures, while LFormer supports long-range modeling, state tracking, and sparse-noisy trajectory reconstruction.These claims are reported across six tasks and sixteen benchmarks.

APPENDIX A DERIVATION OF THE CLOSED-FORM INTEGRAL SOLUTION

The appendix derives a closed-form trajectory for a scalar liquid time-constant state by converting its non-autonomous ODE into an integrating-factor form. The derivation reduces the solution to nested and then simplified integral expressions.

  • FOR 1D LTC: The scalar LTC state s(t) is governed by a linear non-autonomous ODE with scalar exogenous input x(t).The derivation begins from the decoupled one-dimensional state formulation.
  • FOR 1D LTC: The system rate coefficient p(t) maps the liquid dynamics onto the canonical first-order linear non-homogeneous ODE form.A balancing inverse time-constant term wτ is introduced into the bias compartment under a structural symmetry assumption.
  • FOR 1D LTC: The standard non-autonomous integrating factor µ(t) is defined to transform the ODE into a product-derivative equation.Multiplying the canonical ODE by µ(t) yields the derivative of µ(t)s(t).
  • FOR 1D LTC: Definite integration from the initial boundary 0 to timestamp t, with µ(0) = 1, introduces the initial condition s(0) and produces a nested integral trajectory.Positivity of µ(t) permits multiplication by µ^-1(t) to isolate the state trajectory.
  • FOR 1D LTC: A substitution using an auxiliary function a(v) simplifies the nested integral by transforming its differential and integration limits.The transformed expression is substituted back into the state trajectory.
  • FOR 1D LTC: The resulting expression is the closed-form integral solution with learnable scalar parameters wτ and bτ.The appendix identifies these parameters explicitly after reducing the nested integral.

APPENDIX B ERROR ANALYSIS OF THE LEARNABLE ENDPOINT

The appendix analyzes the local error of LGA’s learnable endpoint interpolation under smooth trajectories. It shows that midpoint interpolation recovers trapezoidal-rule accuracy, while general learned weights retain controlled error.

  • Error Setup: Assuming a smooth trajectory, the transformed integrand is twice continuously differentiable over each localized interval.The step size is δ_n = t_n − t_{n−1}.
  • Learnable Endpoint Interpolation: LGA replaces sequential integration with a learnable endpoint interpolation using a bounded coefficient µ = σ(θ) ∈ (0, 1).The interpolation is constructed as a numerical surrogate for the integral.
  • Error Analysis: The local truncation error is defined as the algebraic difference between the exact integral and the numerical surrogate.The appendix expands the endpoint value and exact integral before forming this difference.
  • General Case: For µ ≠ 0.5, the leading error magnitude is proportional to |µ − 0.5|, so moderate deviations do not cause unbounded error amplification.This allows data-dependent interpolation while preserving numerical stability.
  • Symmetric Case: When µ = 0.5, the leading δ_n^2 coefficient vanishes and the interpolation recovers the classical trapezoidal rule with second-order local accuracy.For well-behaved integrands, the accumulated global error over a finite interval is O(δ^2).

SEQUENCE-LEVEL NORMALIZATION

Sequence-level normalization constrains cumulative decay while preserving temporal allocation, producing bounded gates and polynomial rather than exponential gradient attenuation. It is suited to offline full-sequence learning, whereas causal use requires prefix normalization.

  • Normalization: Sequence-level normalization forms a nonnegative normalized decay vector from interval- and input-dependent terms to prevent unbounded cumulative gating.The stability constant ϵ prevents division by zero, and normalized decay terms remain nonnegative.
  • Forward stability: The cumulative sum is monotonic and remains within [0, 1), so stabilized gates stay in (e^-1, 1].This bounds the forward-pass decay budget across the sequence.
  • Temporal semantics: The normalized gate is a regularized surrogate rather than an algebraically exact LTC transition, preserving relative temporal allocation while bounding total decay.For online streaming or causal inference, a prefix-normalized variant is required to avoid dependence on future observations.
  • Gradient stability: Normalized gradients attenuate polynomially as O(S^-2), while the unnormalized variant decays exponentially as O(e^-S).Here S denotes the total sum of unnormalized decay terms along the sequence.

B. BIDMC Physiological Database

The evaluation uses physiological, activity, trajectory, climate, and clinical time-series datasets with varied sampling, sequence lengths, prediction targets, and noise conditions. The passages also describe discrete, state-space, differential-equation, attention, and stochastic baselines.

  • BIDMC Physiological Database: BIDMC tasks predict heart rate, respiratory rate, or blood oxygen saturation from dual-channel 32-second trajectories spanning 4,000 time steps.The waveforms are sampled at 125 Hz and organized with sliding windows.
  • Other benchmarks: Human Activity uses 12-dimensional wearable-sensor recordings for dense point-wise classification into seven physical activity categories.The preprocessing yields 6,554 sequence instances, with canonical sequences of 211 steps.
  • Other benchmarks: Pendulum evaluates dense angular-coordinate reconstruction from 24 × 24 pixel frames after random subsampling creates an irregular temporal grid with correlated noise.Each trajectory contains 100 uniformly spaced steps, with 50 frames observed.
  • Other benchmarks: Synthetic 2D spiral trajectories test interpolation and extrapolation under orientation, structural-variance, noise, and partial irregular-observation conditions.Trajectories contain 150 steps, with 30 points sampled from the first half for partial observation.
  • Other benchmarks: USH and PHY provide weather-station and ICU clinical series, with PHY retaining 37 time-varying variables and patient-level train, validation, and test splits.PHY contains 8,000 records covering the first 48 ICU hours.
  • Baselines: Baselines span discrete recurrence and attention, selective or structured state-space layers, continuous ODE and controlled-differential models, stochastic dynamics, and solver-free approximations.Examples include GRU-∆t, Transformer, Mamba, NODE, NCDE, ContiFormer, CfC, and related methods.

C. Solver-Free Continuous-Time Baselines •

The solver-free baseline set includes analytic, algebraic, attention-based, and structured state-space approaches, while LFormer uses modular Liquid Mixer architectures tailored to benchmark input and output dimensions. Training uses Adam with validation-based early stopping.

  • Solver-Free Continuous-Time Baselines: CfC uses a bounded analytic closed-form integral approximation to bypass numerical integration overhead in liquid time-constant networks.This provides an efficient solver-free continuous-time baseline.
  • Solver-Free Continuous-Time Baselines: RFormer, mTAND, S5, and ACSSM provide non-sequential, continuous-time attention, structured state-space, or simulation-free alternatives for irregular trajectories.Their mechanisms include algebraic path signatures, multi-time kernels, associative scans, and parallel variational inference.
  • LFormer specifications: LFormer Liquid Mixer layers are built from multi-head liquid gated attention followed by SwiGLU blocks, with benchmark-specific depth and hidden dimensions.The standard long-time-series classification configuration uses d = 64, H = 4, and L = 2; SCP1 and SCP2 use a larger configuration.
  • Optimization: All tasks use Adam with β1 = 0.9, β2 = 0.999, no weight decay, and early stopping after 20 validation epochs without improvement.Learning rates are listed per benchmark alongside the architecture settings.
  • LFormer specifications: The benchmark architectures vary embedders and predictors for scalar, class, image, weather, clinical, and trajectory outputs.Specified components include convolutional or linear embedders and task-specific predictor dimensions.
Loading 2608.30695v1…