PaperlayerView exact revision on arXiv

Source-linked AI summary

TIDES: Implicit Time-Awareness in Selective State Space Models

Taylan Soydan, Miguel A. Bessa, Dirk Mohr, Rui Barreira

arXiv:2605.09742v2cs.LGcs.AI

TL;DR

Irregular-time sequence models must preserve the meaning of elapsed time, but continuous-time SSMs and selective SSMs trade physical-time handling against input-dependent expressivity. TIDES moves input dependence to the state dynamics while keeping the timestep physical, and reports strong performance across benchmarks, including 6 of 8 natively irregular datasets. Its evaluation focuses mainly on irregular-time tasks.

  • Problem

    Selective SSMs make the discretization step input-dependent, while continuous-time SSMs retain the physical interval but have time-invariant dynamics.

    Source: §1 · p. 2

  • Method

    TIDES makes decay rates and input/output projections input-dependent while keeping the physical timestep implicit in discretization.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    TIDES achieves state-of-the-art results by highest accuracy on UEA classification and average rank on Physiome-ODE, and matches or exceeds the reference baseline on 6 of 8 natively irregular benchmarks.

    Source: §1 · p. 3, §8 · p. 10

  • Takeaways & Limitations

    The paper’s analysis suggests separating discretization from expressivity when designing architectures for irregularly sampled sequences.

    Source: §7 · p. 9

    The results speak mainly to irregular-time tasks; the paper does not evaluate language modeling or other regular-grid long-range benchmarks.

    Source: §7 · p. 10

Abstract

from arXiv · show

Selective state space models (SSMs), such as Mamba, achieve strong per-token expressivity by making the time discretization step $\Tilde{\Delta}$ a learned function of the input. However, in doing so, $\Tilde{\Delta}$ no longer equals the physical time gap $\Delta$ between consecutive observations, limiting the ability of these models to handle irregular time series. Continuous time SSMs, such as S5, keep $\Tilde{\Delta}\equiv\Delta$ and therefore handle irregular timestamps natively, but their dynamics remain linear time invariant (LTI), limiting per token expressivity. We propose \textbf{TIDES}, a selective SSM variant that reconciles selective and continuous architectures by moving input dependence off the step size and onto the diagonal state matrix. As a result, $\Tilde{\Delta}\equiv\Delta$ as in S5, allowing the model to handle irregular timestamps natively without sacrificing the per-token expressivity that makes selective SSMs effective. We show this on a novel \emph{Fading Flash} experimental benchmark, a compact controlled diagnostic for sequence models that jointly tests input dependence and extrapolation to out-of-distribution $\Delta$ values, and isolates the distinct failure modes of current state-of-the-art architectures that TIDES avoids by construction. On large-scale benchmarks, TIDES sets the new best average rank on UEA time series classification and the Physiome ODE regression benchmark, and matches or exceeds the reference baseline model on 6 of 8 natively irregular datasets from astronomy, agriculture, neuromorphic sensing, and climate events. Code available at: \url{https://github.com/TaylanSoydan/TIDES}.

1 Introduction

Irregular sequences encode information in elapsed time, but continuous-time SSMs trade input-dependent expressivity for physical-time handling, while selective SSMs make the timestep a learned gate. TIDES moves selectivity to the state dynamics while retaining the physical timestep, and the paper reports strong results on its controlled diagnostic and large-scale benchmarks.

Motivation

  • Elapsed time between observations carries information about the underlying continuous process, so robust sequence models must handle it correctly.Irregular sampling occurs in clinical, wearable, financial, scientific, and industrial settings.Source: §1 · p. 1

Background and problem

  • Continuous-time S5 uses the physical sampling interval but has time-invariant updates, whereas selective SSMs make the timestep input-dependent and lose its physical-time meaning.Appending timestamps as input features leaves the model to learn their relationship to its internal gate, which the paper says does not extrapolate beyond training.Source: §1 · p. 2

Our approach

  • TIDES makes decay rates and input/output projections input-dependent while keeping the oscillation frequency static and the timestep equal to the physical interval.The paper calls the resulting property selective implicit time-awareness: observation spacing affects the response through discretization without appearing in the input, positional embedding, or learned gate.Source: §1 · p. 2

Mechanistic analysis

  • The controlled toy experiment isolates S5’s LTI rigidity and Mamba’s failure to extrapolate across training ∆ values, and reports that TIDES overcomes both.The experiment is presented as a diagnostic of these failure modes.Source: §1 · p. 2

Empirical results

  • TIDES sets a new best average rank on UEA time-series classification and Physiome-ODE, and matches or exceeds the reference baseline on 6 of 8 natively irregular datasets.The datasets span astronomy, agriculture, neuromorphic sensing, and climate events.Source: §1 · p. 3

2 Background

Continuous-time SSMs integrate a linear system over the physical gap between observations, preserving a physical-time interpretation. S5 keeps diagonal dynamics static, while Mamba makes dynamics input-dependent and turns the timestep into a learned gate.

2.1 Continuous-time SSMs and discretization

  • Continuous-time SSMs integrate the state over each observation interval, with discretized update matrices determined by the continuous parameters and the chosen scheme.The physical interval ∆ enters the recurrence as elapsed time, so doubling it corresponds to integrating the continuous system for twice as long.Source: §2.1 · p. 3

2.2 S5: diagonal linear time-invariant dynamics

  • S5 uses a fixed complex diagonal state matrix whose real part sets decay and imaginary part sets oscillation frequency, yielding LTI updates with native irregular-time handling.The physical interval enters implicitly through discretization and sets the per-step effective timescale.Source: §2.2 · pp. 3–4

2.3 Mamba family: input-dependent selection

  • Mamba-family models make discretization parameters input-dependent, enabling token-specific dynamics, but their learned timestep is not a physical sampling interval.They assume a uniform input grid; elapsed time can enter only by concatenating ∆ to the input as a feature.Source: §2.3 · p. 4

3 Method

TIDES relocates selectivity from the timestep to the continuous-time generator: the physical timestep stays implicit, while selected state dynamics and projections depend on the input. Its architecture adds stable parameterization and projection mechanisms for implementing those choices.

3.1 Moving input-dependence from ∆to Λ

  • TIDES keeps the timestep physical and implicit, makes each mode’s decay rate input-dependent, and leaves oscillation frequency static.Input-dependent B and C projections add expressivity for reading tokens into the state and reading the state out.Source: §3.1 · p. 4

3.2 Architecture

  • The selectivity heads produce input-dependent Re(Λ), B, and C from each token, with biases initialized to S5’s HiPPO values and zero-initialized projection weights.This starts training from a well-conditioned HiPPO recurrence and learns selectivity as a smooth perturbation.Source: §3.2 · p. 4
  • Low-rank B and C projectors reduce per-head cost from O(PH^2) to O(rPH), with rank r serving as a hyperparameter.The figure presents the sequence model, TIDES block, and lower-level SSM; its SSM panel shows ZOH, while bilinear discretization is also possible.Source: §3.2 · p. 5, Figure 3 · p. 5
  • Residual gated-linear-unit blocks provide nonlinear preprocessing before the final selection projections, with zero depth recovering a linear map.The depths of the Λ projector and input encoder are hyperparameters; both default to zero.Source: §3.2 · p. 5
  • Stable reparameterizations constrain decay rates to the negative half-plane, and projected Re(Λ), B, and C are normalized before discretization.The paper treats the decay reparameterization as a hyperparameter and reports normalization matters most with deep Λ projectors.Source: §3.2 · pp. 5–6

4 The Fading Flash experiment: two failure modes

The Fading Flash task tests whether models can represent zone-dependent decays and extrapolate across unseen clock intervals. Its results separate S5’s expressivity limitation from Mamba’s extrapolation limitation, while TIDES retains robust behavior across a wide ∆ range.

Task setup

  • The diagnostic uses 40 detectors in three decay-rate zones; sparse flashes create glows whose dynamics rescale with ∆, and models predict the glows from flashes, zone, and ∆.The setup isolates input-dependent expressivity and extrapolation to clock speeds outside the training range.Source: Figure 4 · p. 6, §4 · p. 6

Findings

  • S5’s effective learned decays stay constant across unseen ∆ values, but its fixed readout cannot reproduce three zone-specific decays simultaneously.It retains robust relative error across unseen ∆ values yet plateaus at the LTI floor.Source: Figure 5 · p. 6, §4 · p. 7
  • MambaS achieves low training error with a learned timestep gate, but its effective decay rates drift and error grows sharply outside the training ∆ range.The learned gate prevents implicit time-awareness in this experiment.Source: §4 · p. 7
  • TIDES remains robust across a wide span of ∆ values, with eigenvalues at the three target decays, while S5 lacks zone-conditional expressivity and MambaS drifts outside training.The figure reports TIDES outperforming MambaS and S5 in- and out-of-distribution.Source: Figure 5 · p. 6, §4 · p. 7

5 Experiments

The experiments evaluate TIDES on classification, irregular-time forecasting, a controlled sampling diagnostic, and natively irregular real-world data. Results are strongest on benchmark averages and show that input-dependent decay rates retain accuracy as sampling becomes sparser.

5 Experiments

  • The experiments compare TIDES with SSM and irregular-time-series baselines, using per-dataset hyperparameter tuning within a fixed budget.The natively irregular benchmarks cover astronomy, agriculture, neuromorphic sensing, and climate; their evaluation follows each dataset's source-paper protocol.Source: §5 · p. 7, §5.4 · p. 8

5.1 UEA time series classification; 5.2 Physiome-ODE

  • TIDES achieves the highest average accuracy across six UEA datasets, about 1.5 points above Mamba-3, and the best average rank on Physiome-ODE, statistically on par with LinODEnet.TIDES ranks first on 2 of 6 UEA datasets and ties LinODEnet on per-dataset wins at 16/50.Source: §5.1 · p. 7, Table 1 · p. 7, §5.2 · p. 7

5.3 Random drop

  • The random-drop diagnostic trains on EigenWorms with a random 50% of timesteps dropped, then evaluates fixed indices at varying rtest while preserving timestamps.Six roughly 30k-parameter SSM variants differ in which of Re(Λ), Im(Λ), B, C, or Δ is input-dependent; three official Mamba variants and RFormer are also evaluated.Source: §5.3 · pp. 7–8
  • On EigenWorms, TIDES achieves mean accuracy 0.739 and stays flat across rtest, while Mamba variants collapse at rtest=0.9.S5 and TIDESΛ plateau near chance; adding input dependence to Im(Λ) slightly hurts rather than helping.Source: §5.3 · p. 8, Figure 6 · p. 8

5.4 Natively irregular benchmarks

  • On natively irregular benchmarks, TIDES sets the best reported results on TimeSen2Crop, EuroCropsML, and USHCN, and exceeds Event-SSM on SHD and SSC.It matches iTCN on MACHO, is slightly below it on ASAS-SN, and is slightly below the updated Event-SSM result on DVS128.Source: §5.4 · p. 8

6 Related work

Prior SSM work spans LTI continuous-time models and selective discrete-time variants, while irregular-time methods handle elapsed time through integration, gated inputs, or positional codes. TIDES combines selective state dynamics with physical-time discretization.

State space models

  • TIDES is described as the first approach to combine selective Λ with physical-time discretization.Unlike purely discrete selective models, it gives irregular timestamps a physical-time role rather than requiring them only as input features.Source: §6 · p. 9

Irregular time series

  • Irregular-time methods use ODE or SDE propagation, elapsed-time gating, or time-based positional codes in attention and transformers.Examples include Latent ODE and CRU, GRU-D, and mTAN, ContiFormer, and Rough Transformer, respectively.Source: §6 · p. 9

Continuous-time deep learning; State space models

  • Neural ODE and CDE methods provide flexible time handling but incur substantial computational cost compared with SSM-style discretization.The SSM lineage includes S4 and S5, followed by selective Mamba/S6 and gated linear recurrence variants.Source: §6 · pp. 8–9

7 Discussion

TIDES separates analytic time handling in the discretization from input-dependent expressivity in the continuous-time generator, a design pattern associated with stronger extrapolation across sampling regimes. Its reported advantages vary with task needs, and its stated scope is primarily irregular-time sequence problems.

7 Discussion

  • On Physiome-ODE, TIDES retains a small average-rank edge over LinODEnet (2.40 vs. 2.62) while tying on per-dataset wins (16/50).On UEA, where Mamba-3 lacks implicit continuous-time discretization, TIDES wins on average rank and accuracy.Source: §7 · p. 9
  • TIDES’ advantage over baselines scales with the task’s need for input-dependent processing, faithful irregular-time handling, and extrapolation to unseen sampling regimes.The discussion attributes variation in performance margins to baselines possessing different subsets of these properties.Source: §7 · p. 9
  • The proposed design keeps physical Δ in the analytic discretization and places expressivity on the continuous-time generator, including (Re(Λ), B, C).The discussion contrasts this separation with Mamba’s learned discretization step, which recovers per-token expressivity but loses extrapolation in Fading Flash.Source: §7 · pp. 9–10
  • TIDES is aimed mainly at irregular clinical, simulation, or sensor time series, not as a general-purpose replacement for Mamba on regular-grid language modeling.The paper does not evaluate language modeling or other regular-grid long-range benchmarks, and language-model-scale performance remains open.Source: §7 · p. 10
  • TIDES retains linear training scaling and constant per-step recurrent inference cost, and the paper identifies irregular wearable-health streams as a resource-constrained use case.The authors say implicit time-awareness avoids engineering an explicit time-encoding pipeline or maintaining separate missing-observation logic.Source: §7 · p. 10
  • Future directions include using Δ for other continuous sequence parameters, coupling faithful step handling to active sampling, and forecasting at arbitrary future steps.How forecasting behavior degrades far from the training-horizon distribution is left as an open question.Source: §7 · p. 10

8 Conclusion

TIDES combines input-dependent expressivity with physical-time discretization, and the paper reports strong benchmark performance in irregular-time settings. The appendices describe its architecture, discretization, Mamba-family comparisons, and the controlled Fading Flash setup and evaluation protocol.

8 Conclusion

  • TIDES combines per-token expressivity with physical-time semantics and reports the highest UEA classification accuracy, best Physiome-ODE average rank, and baseline parity or gains on 6 of 8 irregular benchmarks.The broader design principle is to keep expressivity on the continuous-time generator and time handling analytic in the discretization.Source: §8 · p. 10

A Architecture details

  • The imaginary part of Λ is the only SSM parameter that remains LTI; state grouping trades representational capacity across groups against richer dynamics within each group.The lower learning rate applies to Im(Λ), and ssm_size = ssm_b · ssm_mult.Source: A Architecture details · p. 14

B Discretization derivations

  • The continuous-time model uses diagonal Λ, B, C, and D, then discretizes each observation interval Δ_k into a recurrence for x_k and y_k.The stated sequence may have irregular timesteps.Source: B Discretization derivations · p. 14

B.1 Zero order hold (ZOH)

  • Under zero-order hold, the input is constant across each interval; diagonal Λ makes the matrix exponential elementwise and compatible with parallel scan.The derivation assumes strictly negative real eigenvalue parts for invertibility.Source: §B.1 · p. 14

C Mamba family

  • Mamba-family models compute the step from the current token, so irregular Δ must be appended as an input; Mamba-2 shares a scalar step per head, while Mamba-3 adds trapezoidal updates and input-dependent complex dynamics.The paper notes that the learned mapping from Δ is unconstrained outside the training range; Mamba-3 also uses a token-dependent endpoint weight.Source: C Mamba family · p. 15

D.1 Data generation

  • Fading Flash uses flash indicators and explicit zone identities with piecewise decay rates to test whether models select the right rate and generalize across Δ values.Training samples Δ uniformly from [0.5, 1.5], while the target glow follows an exact zero-order-hold recurrence.Source: §D.1 · pp. 15–16

D.2 Model details

  • The controlled models share a simplified SSM interface; S5 and TIDES use physical Δ, whereas the Mamba surrogate learns its step from encoded input and Δ.The setup omits BatchNorm, GLU, and dropout to isolate the SSM core.Source: §D.2 · p. 16

D.3 Training protocol / D.4 Evaluation protocol

  • Models are trained for 3,000 Adam steps and evaluated at ten Δ values, with relative error normalized by target variance for comparison across step sizes.The training interval [0.5, 1.5] covers only some evaluation values; variance normalization avoids misleadingly low raw MSE when glow values are small.Source: §D.3 · p. 16, §D.4 · p. 16

D.5 Results

At matched parameter count, Mamba variants do not extrapolate to unseen step sizes, while TIDES has much lower relative error across the test-step sweep.

  • At Δ=0.1, relative error is 6.56 for TIDES, versus 141.20 for Mamba-3 and 154.08 for Mamba-2.Source: §D.5 · p. 16
  • The relative-error ordering holds across the whole test-step sweep, with the Mamba family underperforming TIDES.Source: §D.5 · p. 16
  • Mamba-3 improves on Mamba-2 but does not change their failure to extrapolate, which the authors attribute to where input dependence is placed.Source: §D.5 · p. 16

E Dataset details and experimental setup

The setup uses Physiome-ODE for irregular forecasting and six UEA datasets for classification, with protocols and task details specified for each benchmark.

E Dataset details and experimental setup

  • Earlier IMTS benchmarks were judged poor tests of temporal modeling, while Physiome-ODE addresses their deficiencies; natively irregular classification corpora remain scarce.The cited discussion notes that many classification studies instead simulate irregularity by dropping observations from regular grids, losing real-world structural, non-random missingness.Source: E Dataset details and experimental setup · pp. 16–17

E.1 UEA classification (Walker 2024 protocol)

  • The UEA setup uses six equal-length, complete datasets and evaluates closed-set multivariate classification under five-seed 70/15/15 splits.The datasets span EEG/MEG, worm motion, and spectroscopy; sequence lengths range from 405 to 17,984 and channel counts from 3 to 64. Inputs are standardized using training-partition statistics; no augmentation, time warping, or downsampling is applied. Tables 4–5 provide best configurations and the search space.Source: §E.1 · p. 17, Table 4 · p. 17, Table 5 · p. 18

E.2 Physiome-ODE (Klötergens 2025 protocol)

  • Physiome-ODE uses 50 ODE-generated irregular forecasting datasets, with the first 100 trajectory steps conditioning forecasts of the second 100.Each dataset has 2,000 trajectories; 80% of observations are masked per channel and Gaussian noise is added. The published five-fold instance-level split is used, performance is masked MSE, and models are trained for up to 200 epochs with early stopping and protocol-based hyperparameter selection.Source: §E.2 · p. 18

F UEA complete results

The appendix reports complete UEA classification results and identifies the metric, baseline sources, and models included.

  • The complete UEA results include Transformer, NCDE, NRDE, Mamba-2, and Mamba alongside the other evaluated models.Source: F UEA complete results · p. 18
  • Table 7 reports UEA accuracy percentages with standard deviations and marks the best result for each dataset.Source: Table 7 · p. 19
  • Baseline accuracies in Table 7 are reproduced from Walker et al., Moreno et al., and the Mamba-2 source.Source: Table 7 · p. 19

G Physiome-ODE per-dataset results

Across the 50 Physiome-ODE problems, TIDES has the best average rank, but results vary across the JGD spectrum and no architecture dominates it.

H Random drop task details

This section reports the EigenWorms random-drop experiment, including how input-dependent components affect learning and extrapolation and how TIDES scales on long sequences. It also covers reproducibility, limitations, code and diagnostic documentation plans, and the datasets used.

H Random drop task details

  • Input-dependent B,C variants learn the task, whereas S5 and TIDESΛ with LTI B,C plateau at poor accuracy.The experiment attributes the gap to the need to modulate what is read and written at each observation.Source: H Random drop task details · p. 21
  • At rtest=0.9, MambaS accuracy collapses from ∼0.7 to ∼0.4 despite matching TIDES near the training rate r=∼0.5.The passage attributes this failure to the input-dependent step size’s inability to extrapolate to out-of-distribution drop rates.Source: H Random drop task details · p. 21
  • 0.739 mean accuracy is TIDES’s best result across drop rates, with robustness attributed to input-dependent Re(Λ) and B,C while keeping Im(Λ) static.Making Im(Λ) input-dependent in TIDESfull did not hurt extrapolation but slightly reduced accuracy.Source: H Random drop task details · p. 21
  • RFormer averages 0.720 accuracy, below TIDES’s 0.739, and falls to 0.562 at rtest=0.9 as fixed-window signatures shift out of distribution.Its fixed number of windows is embedded in positional embeddings and the attention mask, so changing the window count at test time requires retraining.Source: H Random drop task details · p. 21

NeurIPS Paper Checklist

  • The checklist identifies projector overhead, a per-step projection bottleneck for L > 10^5, and evaluation limited to time-series benchmarks without a clear physical sampling interval.It also says reported UEA and Physiome-ODE entries use mean ± one standard deviation across five independent runs.Source: NeurIPS Paper Checklist · pp. 23, 26
  • The checklist says the paper provides architecture, experiment, split, and hyperparameter details, with supplementary code and data to reproduce the reported results.It specifies the Fading Flash setup, UEA protocol, and Physiome-ODE benchmark procedure.Source: NeurIPS Paper Checklist · p. 24
  • The authors plan to release the TIDES implementation, UEA and Physiome-ODE training pipeline, and Fading Flash data generation, model variants, and training protocol under an open-source license with the camera-ready version.UEA and Physiome-ODE are used as published; no new dataset is introduced.Source: NeurIPS Paper Checklist · p. 28

Related papers

Loading 2605.09742v2…