Source-linked AI summary
Motion Equivariant Convolutional LSTM: When Motion Becomes a Gauge Symmetry
Vahid Hosseinzadeh, T. Anderson Keller, Michael Puthawala
TL;DR
Standard recurrent models do not generally transport memory with moving inputs, while prior flow-equivariant recurrent models address only constant-velocity motion by enumerating velocities. MELSTM estimates velocity and transports its states, treating motion as a gauge symmetry over time. Across three tasks, it outperforms convolutional LSTM and flow-equivariant counterparts at equal parameter counts, with stronger horizon and motion generalization and a smaller state than the flow-equivariant model.
Problem
Standard recurrent models are not generally equivariant to time-varying motion because their memory stays in place while the input moves.
Source: §1 · p. 2Method
MELSTM estimates a time-varying velocity and uses the resulting transporter to move its hidden and cell states, making motion equivariance a gauge symmetry over time.
Source: §1 · p. 2Results
At equal parameter counts, MELSTM is most accurate on all three tasks and alone avoids degradation when motion statistics change; on Moving MNIST it extrapolates to 8.5 times the trained horizon with 37 times lower error than FELSTM.
Source: §1 · p. 3Takeaways & Limitations
With superposition, motion equivariance extends to local motions, and MELSTM represents independently moving scene components in transported slots without solving frame-to-frame correspondence.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Objects move, cameras pan, and weather drifts with the wind: much of what changes in sequential data is motion. Motion changes where things are, not what they are, and is therefore a symmetry of the data. Yet standard recurrent networks are not equivariant to it: their memory stays put while the input moves on. Flow equivariant recurrent networks first brought this symmetry into sequence models, for motions of constant velocity, or flows, by enumerating a bank of velocities. We extend this idea in three directions: from simple recurrent networks to convolutional LSTMs, from flows to motions whose velocity changes in time, and from enumerating velocities to estimating them. Treating a sequence as a field over time, we show that motion becomes a gauge symmetry: a global symmetry made local in time. Built on this view, our motion equivariant convolutional LSTM (MELSTM) uses its velocity model as the gauge field and is provably equivariant to every motion. Across future-frame prediction, classification of digits visible only through their motion, and action recognition under camera shake, MELSTM outperforms a convolutional LSTM and its flow equivariant counterpart with the same number of parameters: it is more accurate, generalizes to longer horizons and to unseen motions, and carries a fraction of the flow equivariant model's state.
1 Introduction
Standard recurrent models keep memory in place while inputs move, and constraining weights alone generally does not provide flow equivariance. MELSTM instead estimates changing velocities to transport memory, giving motion equivariance and supporting local motions; the paper reports stronger results than LSTM and FELSTM across three tasks.
1 Introduction
- Figure 1 contrasts in-place accumulation with transported state: matched transport stacks evidence at the digit’s true location, while mismatched transport leaves a ghost trail.The example includes simultaneous geometric motion and an intrinsic appearance change.Source: §1 · p. 2, Figure 1 · p. 2
- Standard recurrent memory stays in place as inputs move, leaving a decaying trail; flow equivariance via weight constraints is limited to a measure-zero set of LSTM kernels.The paper argues that memory itself must move with the input.Source: §1 · pp. 2–3
- FERNN lifts hidden states into velocity copies, but changing velocity hands content between copies and requires enumerating the possible velocities.Flows form a group only when G is abelian, and velocities outside the enumerated lattice are unrepresented.Source: §1 · pp. 2–3
- MELSTM estimates a time-varying velocity and uses its transporter to move hidden and cell states, achieving equivariance to all motions when the velocity model transforms as a gauge connection.The paper interprets motions as gauge transformations over time and the transporter as parallel transport.Source: §1 · pp. 2–3
- A motion-equivariant model that respects superposition is also equivariant to local motions, with MELSTM representing independently moving scene components in transported slots.Each slot estimates the velocity of its own content, removing the need to solve frame-to-frame correspondence.Source: §1 · p. 3
- At equal parameter counts, MELSTM is most accurate on all three tasks and alone avoids performance degradation when motion statistics change.On Moving MNIST, it trains about six times faster per epoch than FELSTM and extrapolates to 8.5 times the trained horizon with 37 times lower error.Source: §1 · p. 3
2 Equivariance; static vs dynamical symmetries
The paper extends symmetry from time-independent transformations to transformations that vary over time. It distinguishes global transformations, flows, and motions within this dynamical setting.
2 Equivariance; static vs dynamical symmetries
- Static signals are fields on a domain, while time-dependent signals are trajectories whose frames are fields.The paper denotes the trajectory space by F = F^R.Source: §2 · pp. 3–4
- The trajectory symmetry group consists of time-dependent transformations, with a group element acting at each time.The group composition is defined pointwise in time.Source: §2 · p. 4
- Motions are continuous time-varying transformations generated, for Lie groups, by a time-dependent Lie-algebra-valued velocity.Their evolution is governed by d/dt ψ_t = v_t ψ_t.Source: §2 · p. 4
- Flows are the constant-velocity subset of motions, and they form a subgroup only when G is abelian.For a flow, ϕ_t = exp(tν).Source: §2 · p. 4
- Global transformations are time-independent paths, applying the same group element uniformly across the sequence.They form a subgroup isomorphic to G.Source: §2 · p. 4
- For finite sequences, motions can be represented recursively from a sequence of per-step velocities, while flows use repeated powers of a single step.The discrete motion relation is ψ_t = exp(v_t)ψ_{t−1}.Source: §2 · p. 4
3 Spatiotemporal data and Equivariant Sequential Models
The paper models spatiotemporal inputs as sequences of spatial fields and defines equivariant sequential models on their histories. Under superposition, motion equivariance also yields equivariance to independent motions of scene components.
3 Spatiotemporal data and Equivariant Sequential Models
- For the spatial setting considered, signals are fields on R2 with Rc-valued features, and the paper focuses on translations, SE(2), or R2 ⋊ H actions.The group action operates on spatial coordinates without mixing channels.Source: §3 · pp. 4–5
- A linear map between regular fields is equivariant when it is a convolution whose kernel satisfies the relevant steerability constraint.For translations alone, the map is standard convolution without additional kernel constraints; for SE(2), the kernel is rotation invariant.Source: §3 · p. 5
- A sequential model maps an input history to a regular field, and motion equivariance requires its output to transform consistently with the input history’s motion.The paper distinguishes motion equivariance from flow and global equivariance by changing the transformation class.Source: §3 · p. 5
- When a model respects superposition, motion equivariance implies equivariance to local motions that move components independently.The local-motion action applies a distinct motion to each component in the decomposition.Source: §3 · p. 5
4 Motion Equivariant Convolutional LSTM
The section develops MELSTM, which transports recurrent states between time steps to handle general time-dependent motion while keeping representations on the original spatial domain. It contrasts this with flow-equivariant LSTM state lifting and establishes MELSTM’s motion equivariance under stated conditions.
Transporter based motion equivariant LSTM (MELSTM)
- The ordinary convolutional LSTM is globally G-equivariant with steerable kernels, but generic kernels are not flow equivariant because its memory updates in place.The section attributes the resulting residual memory to the failure to transport accumulated state with moving features.Source: §4 · p. 6
- MELSTM transports hidden states between consecutive steps using an estimated group-valued transformation, supporting general motions while retaining fields on the original spatial domain.The velocity model produces Lie-algebra-valued velocities, and the transporter is their parallel transport in the paper’s gauge interpretation.Source: §4 · p. 7
Channel based flow equivariant LSTM (FELSTM)
- FELSTM achieves flow equivariance by lifting fields to M × G, but is restricted to flows from an abelian subgroup and adds a group dimension to the hidden representation.For general non-abelian groups, the construction restricts to an abelian subgroup V and lifts to transport fields on M × V.Source: §4 · pp. 6–7
Transporter based motion equivariant LSTM (MELSTM)
- Viewing sequences as fields over time makes motion a gauge symmetry: the velocity model acts as a connection between time-local coordinates, and the transporter as parallel transport.The velocity model must satisfy the stated equivariance relation for the transporter relation to hold.Source: §4 · p. 7
- With invariant initial states, the transporter-based MELSTM is motion equivariant for any motion and input trajectory.The proof uses induction over time, with cancellation of a motion factor between the transporter and hidden state in the gate equations.Source: §4 · p. 7
5 Experiments
The experiments compare equal-parameter recurrent models that differ in how they transport hidden states, across prediction, motion-only digit classification, and action recognition under camera shake. MELSTM performs well across these settings, including when motion changes over time or differs from training conditions.
5 Experiments
- The three models share a recurrent cell, parameter count, and training protocol, differing only in whether hidden-state transport is absent, enumerated over velocities, or estimated in slots.For Moving MNIST, the velocity schedule is piecewise constant, with velocities resampled every 3–6 frames.Source: §5 · p. 8, §5.1 · p. 8
5.1.1 Future-Frame Prediction
- MELSTM outperforms both FELSTM and LSTM on future-frame prediction with two independently moving digits and frozen velocities during the prediction horizon.The models predict 10 frames after a 15-frame context; MELSTM transports two slots using estimated velocities.Source: §5.1.1 · pp. 8–9
- MELSTM stays flat to within 1% on both motion-generalization axes, while FELSTM’s penalty peaks near velocity unpredictability ρ = 0.5.MELSTM transports slots using estimated velocities rather than transferring evidence between velocity-indexed copies.Source: §5.1.1 · p. 10, Figure 4 · p. 10
5.1.2 Motion-Only MNIST Classification
- On Motion-Only MNIST, MELSTM is slightly more accurate, trains several times faster, and generalizes better than the other models, using 2 state copies instead of FELSTM’s 25.The digit is visible through motion-dependent pixel correlations, not in any single frame.Source: §5.1.2 · pp. 10–11
- In Motion-Only MNIST, LSTM’s 3 × 3 recurrence cannot carry memory with digits moving faster than one pixel per step, so it must detect motion in noise.It finds such motion detectors only on some seeds.Source: §5.1.2 · p. 11
5.2 KTH Action Recognition under Camera Motion
- MELSTM ends with the highest validation accuracy under training camera shake and retains its static-camera accuracy as shake grows, up to 11 pixels per frame.LSTM and FELSTM lose accuracy and approach chance for large, slow shakes.Source: §5.2 · pp. 11–12, Figure 7 · p. 12
6 Discussion
The discussion situates MELSTM among recurrent models for symmetry and motion, then outlines extensions to learned velocity estimation, deformable motion, and transformers.
6 Discussion
- MELSTM estimates motion from data and is equivariant to motions whose velocity changes at every step, unlike flow-equivariant models.The paper states exact equivariance when the transporter is equivariant.Source: §6 · p. 12
- A trainable velocity model can preserve exact equivariance if it satisfies Equation (20); learned-feature phase correlation could separate motion from intrinsic change.A recurrent velocity model could also predict motion beyond the context.Source: §6 · p. 12
- For atmospheric fields that deform while moving, using diffeomorphism transport and diffeomorphism-equivariant maps is proposed as a route to weather forecasting.The proposed maps could be approximated through canonicalization by image registration.Source: §6 · p. 12
- Transporting keys and values from frame s by Tt,s is proposed to extend flow-equivariant transformers from flows to motions.The transported keys and values transform like the query.Source: §6 · p. 12
A.1 The structure of subgroups of G
The appendix establishes subgroup relations for motions and flows within the group of time-dependent transformations, and decomposes that group into time-dependent and global components.
A.1 The structure of subgroups of G
- Motions form a subgroup of G, whereas flows form a subgroup if and only if G is abelian.The proof uses the adjoint action when composing motions; for flows, closure requires commuting velocities.Source: §A.1 · pp. 15–16
- Every time-dependent transformation decomposes uniquely into a component initialized at the identity and a global component, yielding G = G0 ⋊ Gglob.The conjugation action of the global component on G0 determines the semidirect product structure.Source: §A.1 · p. 16
A.2 Global G-equivariance of LSTM model
The LSTM is globally equivariant under the stated invariant-initial-state conditions, but generic local kernels do not make it flow equivariant. The appendix characterizes equivariant transport and lifting convolutions, including a reduction for abelian velocity subgroups.
A.2 Global G-equivariance of LSTM model
- With invariant initial states, the LSTM is globally G-equivariant by induction over its gate, cell-state, and hidden-state updates.The proof uses equivariance of steerable convolutions and the compatibility of the group action with pointwise nonlinearities and the Hadamard product.Source: §A.2 · pp. 16–17
A.3 LSTM is not flow equivariant
- For generic local kernels, an LSTM is not flow equivariant: the flow-equivariant kernels lie in a Lebesgue measure-zero set.At nonzero time lag, Galilean steerability forces a space-time convolution kernel to be spatially constant, so a local kernel cannot couple different times.Source: §A.3 · pp. 17–18
A.4 Transport and flow convolution
- For an abelian velocity subgroup, the transport-convolution constraint reduces to common spatial-translation invariance, yielding a convolution in space and velocity.The transport-lifting kernel is independent of its group argument and reduces to a standard steerable convolution copied into the extra group-indexed channels.Source: §A.4 · pp. 18–19
A.5 Flow equivariance of FELSTM
FELSTM is proved equivariant to flows, with the proof isolating its constant-velocity requirement. The velocity-model argument and the subsequent transporter-based MELSTM proof address equivariance for time-varying motions.
A.5 Flow equivariance of FELSTM
- The channel-based FELSTM is flow equivariant, as shown by induction over its gate, cell-state, and hidden-state updates.The proof uses flow-equivariant flow-convolution and flow-lifting operators, and the flow property at one step of the induction.Source: §A.5 · pp. 19–21
- FELSTM’s equivariance condition holds precisely for flows, because it requires the velocity to remain constant in time.The same argument says global transformations satisfy the condition trivially, so FELSTM is also globally equivariant.Source: §A.5 · p. 21
A.6 Equivariance of the Velocity model
- The velocity-model argument differentiates the transporter’s motion-transformation identity and uses the transporter ODE.The stated derivation invokes uniqueness of the ODE solution before differentiating.Source: §A.6 · p. 21
A.7 Motion equivariance of the Velocity Based MELSTM
- The transporter-based MELSTM is motion equivariant, with its proof proceeding by induction over the gate, cell-state, and hidden-state updates.The induction uses equivariance of the transporter together with equivariance of the convolution and pointwise nonlinearities.Source: §A.7 · pp. 21–22
A.8 Time-dependent data and motion equivariance as a gauge theory
The gauge-theory view treats time-dependent signals as sections over time and motions as gauge transformations. It interprets the velocity model as a connection whose parallel transport gives the motion-equivariant state update.
A.8 Time-dependent data and motion equivariance as a gauge theory
- The velocity model is a gauge connection, and its parallel transport is represented by a time-ordered exponential or the solution of a transporter ODE.The connection’s covariant derivative measures intrinsic time variation after subtracting the contribution induced by motions.Source: §A.8 · pp. 22–23
- Time-dependent signals are sections of an associated bundle, and motions act on them as gauge transformations over time.In the trivial bundle used here, the signal space is identified with the space of sections.Source: §A.8 · p. 23
- In discrete time, the model transports the previous feature to the next step before combining it with newly computed features, making the dynamics motion equivariant by design.The continuous-time formulation instead uses a covariant time derivative to express the dynamics.Source: §A.8 · p. 23
B.1.1 Moving MNIST Dataset
The Moving MNIST setup places digits with time-dependent velocities on a padded discrete torus and generates trajectories with controlled velocity changes. Its evaluation freezes velocity during rollout so it tests how models encode and assign motion from context frames.
B.1.1 Moving MNIST Dataset
- The dataset places MNIST digits on a padded discrete torus, with each digit assigned a velocity from a bounded integer lattice.The padded side length S is at least 28, and the velocity lattice contains vectors v with ∥v∥∞≤N.Source: §B.1.1 · p. 24
- The motion process updates velocity with probability pchange, choosing a neighboring lattice state with probability psmooth and otherwise sampling from the lattice.A piecewise-constant alternative holds velocity for a sampled duration before applying the same update rule.Source: §B.1.1 · p. 24
- The setup measures realized switching rate separately from jump magnitude, which is averaged only over frames where velocity actually changes.The realized switching rate can be lower than pchange because an update proposal may be held at the lattice boundary.Source: §B.1.1 · p. 24
- Both models infer motion from video frames during context, then predict a rollout with frozen constant velocity and no ground-truth velocity input.This setup tests encoding and velocity assignment while avoiding the task of predicting unpredictable future velocity changes.Source: §B.1.1 · pp. 24–25
B.1.2 Training and Evaluation Setup
The experiments use controlled, equal-parameter comparisons and evaluate both longer rollouts and changed motion regimes. The motion-penalty analysis treats handover cost as a heuristic linked to disruption and the value of the history that is abandoned.
B.1.2 Training and Evaluation Setup
- Training uses 25-frame sequences with 15 context and 10 predicted frames, and computes MSE + L1 loss only on predicted frames.Models share the training protocol; optimization uses Adam for 50 epochs with batch size 32.Source: §B.1.2 · p. 25
- All three models have exactly 47,681 trainable parameters, with 1, 25, and 2 hidden-state copies per step for LSTM, FELSTM, and MELSTM, respectively.The recurrent cell has 38,144 parameters and the decoder 9,537; shared weights keep parameter counts equal across architectures.Source: §B.1.2 · p. 25
B.1.3 Length and Motion Generalization
- Length generalization extends the trained 10-frame rollout to 85 free-running predicted frames, with velocity frozen at the context boundary.Figure 8 shows qualitative extrapolation to t = 140; the scored rollout ends at t = 100.Source: §B.1.3 · p. 25, Figure 8 · p. 26
- Motion generalization rescores trained checkpoints without retraining on changed context motions, using each model’s median per-sequence MSE relative to its training-motion reference.The ratio normalizes away overall error differences between models, but not the difficulty of the new motion itself.Source: §B.1.3 · pp. 25–26
- The proposed FELSTM penalty ρ(1 −ρ) is a heuristic combining handover disruption with the predictive value of abandoned history, and its observed relation to penalty is nonlinear.The penalty stays near 1 for regimes with ρ(1 −ρ) ≤0.11, then grows across the other listed regimes toward churn_mid.Source: §B.1.3 · p. 26
B.1.4 Velocity Model
MELSTM estimates motion by tracking each slot’s hidden-state template against incoming frames, binding each estimated velocity to the slot that requested it. Frame-pair velocity schedules provide an alternative during training before the model relies on slot tracking.
B.1.4 Velocity Model
- The channel-handover analysis links FELSTM’s penalty to both motion unpredictability and the predictive value of history stranded when velocity changes.The proposed ρ(1 −ρ) relation is explicitly a heuristic, and the observed penalty is not linear in that quantity.Source: Figure 9 · p. 27
- Each slot estimates its own velocity by phase-correlating its hidden state with the current frame, so the slot identity supplies the tracking correspondence.Once a slot holds one coherently moving object, the estimator faces a single global displacement rather than a region-by-region matching problem.Source: §B.1.4 · pp. 26, 28
- At initialization, the K strongest peaks from the first raw-frame pair seed the slots; subsequent velocities are estimated individually from each slot’s hidden state.This bootstrap breaks the symmetry that would otherwise keep zero-initialized slots identical.Source: §B.1.4 · p. 28
- The parameter-free estimator’s accuracy rises from ≈66% to over 99.9%, indicating that the hidden-state templates improve rather than that the estimator is being fitted.The prediction loss shapes the state, while more accurate transport and cleaner templates reinforce one another; no explicit motion supervision is provided.Source: §B.1.4 · p. 29, Figure 11 · p. 29
B.1.5 Training with Frame-Pair Velocities
- Frame-pair velocities can replace slot tracking during training with either a probability that decays to zero by epoch 25 or a context window that shrinks to none.Validation and testing always use slot tracking; both schedules end at about the test error of training with slot tracking throughout.Source: §B.1.5 · p. 29
B.2 Motion-Only MNIST Classification
Motion-Only MNIST provides no single-frame outline cue, while the KTH evaluation tests action recognition under camera shake. The described methods use recurrent motion information and slot tracking, with evaluation protocols that control how motion and clips are presented.
B.2 Motion-Only MNIST Classification
- White-noise textures have matching statistics and no neighboring-pixel correlation, leaving no digit outline; a single-frame CNN scores 12.2% against 10% chance.Gaussian smoothing leaks outline information: the same CNN reaches 13%, 24%, and 36% at standard deviations of 0.5, 1, and 2 pixels.Source: §B.2 · p. 31
- MELSTM seeds two slots from frame-pair peaks and later assigns each new peak to the slot with the nearest previous velocity; slot tracking instead is about one accuracy point lower.With slot tracking, the digit also appears in the hidden state.Source: §B.2 · p. 31
- On example sequences, the digit accumulates clearly in its MELSTM slot, whereas it appears only faintly in FELSTM copies as velocity changes move it between copies.The frames themselves are pure noise; the digit outline in the figure is drawn only for the reader.Source: §B.2 · p. 31, Figure 14 · p. 32
- LSTM can compare frames across every displacement up to 3 pixels and learn local motion detectors, but finding those detectors in noise succeeds only on some seeds.The stated receptive-field range covers every velocity in the data.Source: §B.2 · p. 31
B.3 KTH Action Recognition under Camera Motion
- KTH evaluation uses 16-frame clips from six actions, with fixed test windows shared across camera-motion conditions.The clips are divided into 1,517 training, 380 validation, and 472 test examples; test sets differ only in camera motion.Source: §B.3 · p. 31
- For KTH, MELSTM uses two slots with sub-pixel velocity estimates and slot tracking at evaluation, while FELSTM carries nine velocity copies.Frame-pair velocities are used only early in training; slot tracking is equivariant to any camera path according to the passage.Source: §B.3 · p. 33