PaperlayerView exact revision on arXiv

Source-linked AI summary

Flow Equivariant World Models: Memory for Partially Observed Dynamic Environments

Hansen Jin Lillemark, Benhao Huang, Fangneng Zhan, Yilun Du, Thomas Anderson Keller

arXiv:2601.01075v2cs.LGcs.AIcs.CV

TL;DR

Partial observability challenges world models because observations omit evolving regions, while existing methods ignore the world’s time-parameterized flow symmetries. FloWM organizes latent memory around these symmetries and substantially improves video world modeling and generalization to sequences longer than training.

  • Problem

    Partial observability requires memory for integrating incomplete observations over time, while existing world models do not exploit the world’s time-parameterized flow symmetries.

  • Method

    FloWM uses flow-equivariant latent memory that transforms with external-object motion and known agent self-motion to maintain structured representations of partially observed worlds.

  • Results

    FloWM substantially improves video world modeling performance and generalization to sequences significantly longer than those seen during training.

  • Takeaways & Limitations

    Flow-equivariant memory supports faster learning, lower error, fewer hallucinations, and more stable rollouts far beyond the training length.

  • Takeaways & Limitations

    The evaluation uses controlled settings with rigid-body geometric actions and known action parameterizations, leaving extension to realistic open-world datasets for future work.

Abstract

from arXiv · show

Embodied systems experience the world as 'a symphony of flows': a combination of many continuous streams of sensory input coupled to self-motion, interwoven with the dynamics of external objects. These sensory streams and the underlying dynamics of the world obey smooth, time-parameterized symmetries which existing world models ignore. Without a memory that respects this structure, partial observability presents a major obstacle to existing methods: each observation reveals only a fraction of the world, while unobserved regions continue to evolve. In this work, we introduce Flow Equivariant World Modeling, a framework that leverages time-parameterized symmetries within a latent memory for stable and accurate dynamics prediction over long horizons. The latent memory shifts and transforms equivariantly with self-motion and inferred external object motion, keeping information about out-of-view regions aligned as time progresses. We demonstrate the advantage of this framework over state-of-the-art diffusion, memory-augmented, and recurrent world model architectures on 2D and 3D partially observed video world modeling benchmarks. More broadly, our results suggest that predictive representations become more powerful when they are organized in line with the temporal and dynamical structure of the world they model. Project page: https://flowequivariantworldmodels.github.io/

1. Introduction

Partially observed embodied agents must model their surroundings, self-motion, and moving bodies despite restricted first-person views that shift and rotate with the agent. Flow Equivariant World Models represent internal and external motion as time-parameterized symmetries, improving long-horizon video world modeling and generalization.

  • Introduction: Embodied agents must estimate surrounding environments, self-motion, and moving-body dynamics from restricted first-person observations rather than an omniscient global view.The field of view shifts and rotates with the agent’s self-motion.
  • Introduction: Flow equivariance treats internal and external motion as mathematical flows represented through time-parameterized symmetries.This formalizes geometrically structured dynamic memory for partially observed dynamical world modeling.
  • Introduction: FloWM handles self-motion structurally while capturing external-object motion, including objects moving outside the observed field of view.Its structured dynamic memory preserves the continuation of unobserved dynamics when the agent later returns to an earlier viewpoint.
  • Introduction: The framework yields substantially improved video world modeling performance and generalizes to sequences significantly longer than those seen during training.The introduction attributes these gains to precise spatial and dynamical structure in world models.

2. Background

The background situates the work at the intersection of world modeling, partial observability, persistent memory, and equivariant representations. It motivates recurrent flow-equivariant memory for structured prediction in dynamic environments while reviewing existing approaches and the definition of equivariance.

  • World Modeling: World models predict future environment states from initial conditions and how those states change under agent actions, with recent work often generating video using latent diffusion transformers.The passage identifies large-scale latent diffusion transformer models as a primary focus of recent world-modeling research.
  • Partial Observability: Partial observability occurs when observations omit the full world state, making memory necessary to represent and integrate information over time.This issue is especially relevant to 3D world modeling because agents have limited fields of view.
  • Memory: Prior latent-memory video diffusion work primarily targets consistency in static 3D scenes rather than unified modeling of partially observed dynamics.The passage contrasts this focus with the proposed direction of recurrent flow-equivariant memory.
  • Memory: A recurrent flow-equivariant memory can represent world dynamics and agent actions, support structured future-state prediction, and preserve important information over unbounded timespans.The memory is described as evolving and shifting to integrate dynamics and actions seamlessly.
  • Equivariance: Equivariance requires a network’s output to transform predictably with its input under every group element, and structured weight sharing can construct such networks.The passage also states that this structure reduces learned parameters while improving performance.

3. Flow Equivariant World Models

Flow Equivariant World Models extend equivariance from static transformations to time-parameterized flows over partially observed world states, motivating a structured latent memory that remains aligned as the agent and external objects move. The framework generalizes flow-equivariant recurrence, incorporates known self-motion, and instantiates the memory with spatially organized Transformer tokens for 3D environments.

  • Flow Equivariance: Flow equivariance models time-parameterized transformations generated by vector fields, requiring outputs to transform consistently when input observations move.The flow ψ_t maps group elements through a vector field, while equivariance informally means that moving inputs induce correspondingly moving outputs.
  • Partial Observability: Under partial observability, flows act on the full evolving world state rather than necessarily appearing as identical transformations in restricted observations.Because observations may omit moving objects, the latent representation must encode a structured memory map of the dynamic environment, not merely the current view.
  • Flow-Equivariant Recurrence: Flow-equivariant recurrence transports recurrent memory by the input flow before incorporating new observations, preserving spatial alignment with moving content.For multiple flows, hidden states use velocity channels that follow separate vector fields and permute according to the difference between channel and input velocities.
  • Generalized Recurrence: The generalized recurrence supports arbitrary encoders and update operations, provided both are equivariant and the encoder trivially lifts inputs across velocity channels.The paper states the corresponding encoder and update equivariance conditions and formally proves retention of flow equivariance for fully observed environments.
  • Self-Motion and Transformer FloWM: Self-motion equivariance transforms hidden state using the action’s latent group representation, while the Transformer-based FloWM uses spatially organized token embeddings as a structured 3D world map.The ViT instantiation adds per-step encoder and decoder expressivity, but its required equivariant lift from a 3D first-person view to a top-down map is treated approximately rather than constructed formally.

4. Experiments

Experiments on newly introduced 2D and 3D partially observed world-modeling benchmarks show that FloWM’s structured dynamic memory supports accurate, stable long-horizon prediction. FloWM also improves downstream planning and learns more equivariant internal representations than the baselines.

  • MNIST World: The MNIST World benchmark tests moving digits under partial observability, with viewpoint actions and continuously evolving out-of-view dynamics.The world is a 2D canvas containing digits moving at random constant velocities, while the agent observes only a smaller view.
  • MNIST World: FloWM remains consistent with ground truth for 150 prediction timesteps beyond a 20-timestep training horizon, whereas ablations fail or diverge.Combining self-motion equivariance and velocity channels also requires orders of magnitude fewer training steps to converge.
  • 3D Dynamic Block World: On 3D Dynamic Block World, FloWM remains consistent for up to 210 future frames, while diffusion and RSSM baselines diverge or produce hallucinated, forgotten, or blurry objects.The comparison uses 70 frames of context and evaluates rollouts of 70 and 210 prediction frames.
  • Downstream Planning: FloWM quickly reaches the red block in an 8-episode MPC task, while baselines often hallucinate the block and choose poor actions.The experiment uses no additional trained parameters and demonstrates the downstream utility of consistent dynamics prediction.
  • Learned Equivariant Representation: Training reduces FloWM’s equivariance error from 6.96 to 0.22, compared with 2.36 for DFoT.The errors are measured in L2 distance; after training, FloWM’s error corresponds to only 0.16 units in each spatial dimension.

5. Related Work

Related work includes memory-augmented diffusion models that mainly address static scenes and lack unified integration of actions and dynamics, alongside structured map memories that motivate FloWM’s equivariant design. Prior map-based approaches shift spatial memories with agent actions, while other retrieval mechanisms face limitations in dynamic environments or long-term consistency.

  • Generative World Modeling with Memory: Memory-augmented diffusion world models mainly target static scenes and use implicit token-based action embeddings, lacking unified action–dynamics integration in memory.The paper identifies jointly integrating agent actions and world dynamics as key aspects of FloWM’s memory representation.
  • Generative World Modeling with Memory: Camera-position retrieval mechanisms cannot handle dynamic environments and rely on self-attention for self-motion, causing long-term consistency errors.Recent voxel-map methods instead maintain 3D environment maps, but rely on depth unprojection and max-pooling updates.
  • Equivariant World Modeling: Neural Map introduced a spatially organized 2D memory that stores observations at estimated agent coordinates and shifts them according to agent actions.The resulting allocentric latent map is effectively equivariant, and one variant can be viewed as a special case of FloWM without velocity channels.

6. Discussion · Appendix Contents

The discussion presents flow equivariance as both a design principle and theoretical foundation for structured memory in partially observed dynamic environments. It also identifies controlled evaluation settings as a limitation and outlines extensions toward realistic datasets, stronger decoding, and indefinitely maintained latent maps.

  • 6. Discussion: Flow equivariance guides the construction of structured memory for world models in partially observed dynamic environments.
  • 6. Discussion: Flow-equivariant world models represent motion in a structured symmetric manner.
  • 6. Discussion: The framework is associated with faster learning, lower error, fewer hallucinations, and more stable rollouts beyond the training length.
  • 6. Discussion: The work combines theoretical groundwork with empirical validation for flow-equivariant world modeling.
  • 6. Discussion: FloWM is evaluated in controlled settings using rigid-body geometric actions and known action parameterizations.These settings isolate the contribution of flow-equivariant memory under partial observability.
  • 6. Discussion: A key future direction is extending the framework to more realistic and open-world datasets.
  • 6. Discussion: A more powerful diffusion decoder could help bring FloWM’s framework to larger settings.
  • 6. Discussion: An adaptively sized latent map maintained for infinite horizons is another proposed future direction.

A. Generalized Flow Equivariance Proof · B. Additional Related Work · C. Block World Latent Representation Probe Experiments

The supplementary sections prove flow equivariance for the generalized recurrence, situate the approach among world-modeling and memory methods, and describe latent representation probes that test block-position prediction under partial observability.

  • A. Generalized Flow Equivariance Proof: The generalized Flow Equivariant Recurrence Relation is proven flow equivariant by induction, using a base case and an inductive step.The proof follows Keller (2026).
  • A. Generalized Flow Equivariance Proof: The proof assumes a trivial-lift encoder, equivariant encoder and update operations, and a flow-independent, invariant hidden-state initialization.These assumptions yield the desired flow-equivariance commutation relation.
  • A. Generalized Flow Equivariance Proof: The resulting relation transforms hidden states through the integrated flow while shifting the velocity argument and spatial group element.The recurrence applies the flow update after combining the input and hidden state, producing a t-indexed flow action.
  • B. Additional Related Work: Generative world models commonly predict future observations in pixel or latent video space, with transformer-diffusion methods extending rollouts through sliding-window inference.Recurrent reconstruction-based methods such as Dreamer V3 maintain consistency but are described as suffering from poor visual quality and limited long-horizon memory.
  • B. Additional Related Work: Existing memory strategies face distinct limitations: attention loses comprehensive history beyond its effective window, viewpoint-conditioned storage becomes brittle, retrieval may miss changed future states, and voxel memory has cubic cost.These limitations motivate memory representations that preserve canonical scene state and dynamics under partial observability.
  • B. Additional Related Work: Novel-view and 3D-reconstruction methods provide geometric and multi-view priors but generally address reconstruction or retargeted rendering rather than action-conditioned future prediction under partial observability.World-model evaluation also requires attention to long-horizon state consistency, forgotten out-of-view objects, and hallucination beyond perceptual metrics.
  • B. Additional Related Work: Equivariant models impose structured transformations on internal states, while related equivariant world-modeling work often targets static environmental symmetries or other domains rather than motion in dynamic scenes.The related-work discussion also connects self-motion signals to predictive processing and position coding in neuroscience.
  • C. Block World Latent Representation Probe Experiments: The Block World probe experiments collect trained-model activations and train simple probes to predict block positions, including cases where blocks are out of the agent’s view.This evaluates whether latent activations retain position information under the dataset’s partial observability.

C.1. Probe Experimental Setup … E.2. Textured Block World Results

The appendices detail probe evaluation, downstream planning, Block World dataset construction, and textured-scene results. FloWM maintains more consistent long-horizon predictions than the baselines in qualitative textured rollouts.

  • C.1. Probe Experimental Setup: Probe activations are collected from 1024 Dynamic Block World videos, using FloWM’s hidden state and intermediate DFoT or DFoT-SSM activations.The subset contains one block of each color and follows the 140-frame validation setup.
  • C.3. Additional Probe Model Results: FloWM’s probe makes only slight coordinate errors and can correct itself, whereas DFoT’s predictions do not closely follow ground positions through time.Additional visualizations show this contrast in top-down maps and temporal prediction plots.
  • C.1. Probe Experimental Setup: Probe training splits videos into train and test sets, then predicts each colored block’s position relative to the agent independently at every timestep.Five percent of videos are retained for testing.
  • C.2. Probe Model Details: Each probe applies two 3 × 3 convolutions followed by flattening and a two-layer MLP, with FloWM using spatial memory while baselines use image-coordinate latent spaces.Probe models are trained separately for each block color and their ensemble accuracy is reported.
  • D. Block World Downstream Planning Experiment Details: The downstream “Find the red block” experiment uses model-predictive control with exhaustive action-sequence search over a horizon of length 3.The four actions are turn left, turn right, move forward, and do nothing; reward is the number of red pixels as a closeness proxy.
  • E.1. Block World Dataset Details: Block World examples contain videos and discrete agent actions, while Dynamic Block World randomizes block colors and uses deterministic nonlinear motion from wall bounces.Actions are left, right, forward, or do nothing.
  • E.1. Block World Dataset Details: The textured dataset preserves Dynamic Block World dynamics while randomizing wall, floor, and block textures.Wall textures include brick, dark wood panel, and wood panel; floor textures include cardboard, grass, and concrete; block textures include metal grill, airduct grate, cinder blocks, and ceiling tiles.
  • E.2. Textured Block World Results: FloWM retains performance improvements over baselines on Textured Dynamic Block World, suggesting applicability to visually realistic partially observed scenes.Qualitative rollouts show hallucinated objects in DFoT and DFoT-SSM over time, while FloWM remains consistent; evaluation uses 70 context frames.

E.3. Static Block World Results · F. MNIST World Additional Dataset Details and Results · F.1. MNIST World Dataset Details

Static Block World results show that FloWM without velocity channels performs best because the environment has no external motion, while DFoT and DFoT-SSM struggle to remember block locations. The MNIST World ablations vary self-motion, dynamics, and partial observability using systematically generated dataset subsets and parameter settings.

  • E.3. Static Block World Results: FloWM (no VC) achieves the best metric scores on the static Block World dataset.The default velocity-channel configuration adds noise because static environments have no external motion to model.
  • E.3. Static Block World Results: Velocity channels are unhelpful in static Block World because there is no external motion to model.The passage attributes the default configuration’s added noise to the absence of external motion.
  • E.3. Static Block World Results: DFoT and DFoT-SSM struggle with consistently remembering where the blocks are in the static environment.The models are evaluated with 70 context frames to match the DFoT-SSM training regime.
  • E.3. Static Block World Results: Static Block World rollout evaluation generates 70 or 210 future frames conditioned on 70 context frames.This evaluation setup is specified for the 3D Static Block World benchmark.
  • F.1. MNIST World Dataset Details: MNIST World ablations examine scaling difficulty in self-motion, dynamics, and partial observability.The corresponding dataset subsets are summarized in Table 7, with generation parameters provided in Table 8.
  • F.1. MNIST World Dataset Details: An MNIST World subset is partially observable when the world size exceeds the window size.The number of digits is also scaled according to the world size.
  • F.1. MNIST World Dataset Details: Each MNIST World example is a video shaped [num frames, channels, height, width].The passage specifies the video tensor organization for each dataset example.

F.2. MNIST World Additional Results … G.4. Update Details

Additional MNIST World results show that self-motion equivariance is central to long-horizon prediction, while velocity channels improve flow-equivariant encoding. The 3D implementation specifies velocity-conditioned hidden-state recurrence, action-induced transformations, transformer processing, and an equivariant gated update.

  • F.2. MNIST World Additional Results: On the simplest fully observable, no-self-motion dataset, all models perform reasonably well, while DFoT has about 0.02 baseline MSE from latent-decoding error.This setting matches conventional world modeling because required information remains in the attention window.
  • F.2. MNIST World Additional Results: Self-motion equivariance is key to solving the partially observed problem, and velocity channels improve input flow equivariance; without them, errors accumulate over long horizons.On static environments with redundant velocity channels, FloWM (no VC) slightly outperforms FloWM, but generally reaches higher errors.
  • G. FloWM Experiment Details: 3D Dynamic Block World: The 3D Dynamic Block World implementation uses 6-layer ViT encoders and decoders, 8 attention heads per layer, and embedding dimension 256.The hidden state uses Chid = 256 channels and spatial dimensions Hworld = Wworld = 32, initialized as h0 = 0.
  • G.2. Velocity Channels: The 3D FloWM hidden state has five velocity channels for ±1 horizontal and vertical map velocities plus zero velocity, with each channel flowed by its corresponding block velocity field.Diagonal velocities are excluded.
  • G.2. Velocity Channels: Agent actions transform the hidden state through the inverse action representation, implemented as a one-element roll for forward motion and a ±90-degree rotation for turns.These transformations keep the world representation aligned with the agent’s movement.
  • G.3. Transformer Details: The ViT encoder jointly processes image tokens and field-of-view-selected map-latent tokens with separate sin-cos absolute position embeddings, while the decoder uses masked image tokens and cross-attention to updated field-of-view latents.Images are patchified with patch size 16; map and image embeddings represent different spatial coordinates.
  • G.4. Update Details: The transformer FloWM update is a gated combination of encoder map-token outputs and the previous hidden-state field of view, with α = σ(W concat[h_t(x,y); o_t(x,y)]).The update operation is equivariant to shifts and rotations of its input spatial coordinates.

G.5. Training Loss Details

FloWM and its ablated variants receive 50 observation frames and predict the next 90 observations using future actions, minimizing MSE over prediction frames 50–139. Training uses Adam with a 1e−4 learning rate, batch size 16, 12.5% teacher forcing, and up to 150k steps.

  • Training objective: FloWM and its ablated variants take 50 observation frames as input and predict the next 90 observations conditioned on the corresponding future action sequence.The training objective averages mean squared error between model outputs and ground truth over prediction frames 50 to 139.
  • Training objective: The models minimize mean squared error averaged over prediction frames 50 to 139.The loss compares each model output with the corresponding ground-truth observation sequence.
  • Optimization: Training uses Adam with a learning rate of 1e −4, batch size 16, and a teacher forcing ratio of 12.5%.Each model is trained for 150k steps or until convergence.

H. FloWM Experiment Details: MNIST World … I. Compute Resource Comparison

The MNIST FloWM uses a compact convolutional recurrent architecture whose latent state explicitly tracks constant velocities and compensates for agent self-motion. Its ablations remove these mechanisms or replace them with action concatenation, while training uses fixed observation/prediction horizons and roughly 75K parameters.

  • H. FloWM Experiment Details: MNIST World: The 2D Simple Recurrent FloWM is a sequence-to-sequence RNN with small CNN encoders and decoders for MNIST digit features.Full code is available on the project page.
  • H.1. Recurrence: The hidden state has 64 channels, matches dataset world dimensions, uses 50×50 spatial dimensions for partial observations and 32×32 for full observations, and starts at h0 = 0.For partial observations, the window size is 32×32.
  • H.1. Recurrence: The recurrence applies a 3 × 3, 64-channel convolution with circular padding and no bias, while omitting inter-velocity terms because the datasets contain only constant object velocities.The kernel can otherwise span velocity channels to model acceleration or more complex dynamics.
  • H.2. Velocity Channels: MNIST World uses 25 velocity channels covering every X- and Y-velocity combination from −2 to 2, and each channel is flowed by its corresponding velocity field.The velocity set is V = {(−2, −2), …, (2, 2)}.
  • H.2. Velocity Channels: Agent actions add inverse self-motion flow, producing the combined channel flow ψ1(ν −at), implemented as a hidden-state roll by (ν −at) pixels.This makes the representation equivariant to both modeled velocities and agent-induced visual flow.
  • H.3. Encoder: The encoder is a single 3 × 3 convolution from one input channel to 64 output channels, and partial observations are padded to the world and hidden-state dimensions.The observation is processed as U ⋆ft before padding.
  • H.4. Decoder: The decoder crops the hidden state at the encoder’s write location, max-pools pixelwise over velocity channels, and applies a two-layer 3 × 3 CNN with 64 hidden channels and an intervening ReLU.This supports independent pixel-velocity prediction.
  • H.5–H.8. Ablations: The ablations remove velocity channels, self-motion equivariance, or both; action-concat instead adds action components as input and hidden-state channels but only marginally improves performance and cannot learn FloWM’s precise equivariance.The no-self-motion variant corresponds to FERNN with partial-observability modifications, while removing both mechanisms yields a convolutional RNN.

I.1. Training and Inference Compute Comparisons … K.4. DFoT-SSM Training Details

The appendix compares FloWM’s compute and memory efficiency with diffusion and recurrent baselines, reports scaling and optimization opportunities, and details RSSM, DFoT, and DFoT-SSM configurations. These comparisons and ablations emphasize FloWM’s stable long-horizon predictions, sublinear memory scaling, and the importance of structured memory under partial observability.

  • I.1. Training and Inference Compute Comparisons: FloWM uses roughly 1.7 to 2.6× more training FLOPs than DFoT and DFoT-SSM, while remaining within the same order of magnitude and producing more stable long-horizon predictions.Forward, backward, and total training compute are reported in Table 13.
  • I.1. Training and Inference Compute Comparisons: FloWM achieves the highest throughput and sublinear memory scaling with generated and context frames, with rollout time significantly faster than the baselines.Inference measurements use one H200 GPU; memory reports resident allocation and peak delta during rollout.
  • I.2. FloWM Memory Map Hyperparameter Effects on Compute Efficiency: Larger FloWM maps and Fields of View increase rollout time and memory usage, but FloWM retains high throughput compared with the baselines.These effects are reported in Table 15.
  • I.3. Limitations and Opportunities for Efficiency: Potential efficiency improvements include decoupling encoding from latent-map updates and designing recurrent updates to be linear and associative for greater parallelism.The current implementation is not heavily optimized, and these changes would reduce computational cost without changing the core modeling assumptions.
  • J. RSSM Baseline Details: RSSM uses action-conditioned recurrent latent dynamics with reconstruction, dynamics-KL, and representation-KL objectives for long-horizon rollout prediction.The architecture maintains deterministic and discrete stochastic latent states and is evaluated using a 400M-parameter variant.
  • J. RSSM Baseline Details: Scaling RSSM from 50M to 400M parameters improves rollout results, but the improvement is not significant, supporting the importance of structured memory.The comparison is reported in Table 16 on the 3D Dynamic Block World dataset.
  • K.1. Video Diffusion Transformers: Video diffusion world models use VAE latent representations, temporal attention, and often autoregressive sampling or sliding-window attention to generate long videos consistently.The appendix motivates History-guided Diffusion Forcing with a CogVideoX-style transformer as the DFoT baseline.
  • K.2. Diffusion Forcing Transformer Baseline: DFoT independently noises each frame during training, conditions generation with clean history frames in the same self-attention window, and uses training lengths of 70 frames for MNIST World and 140 for Block World.DFoT does not distinguish observation from prediction frames during training and applies History Guidance during inference.

K.5. VAE Training Details

The models use an 8× spatially downsampling VAE for latent diffusion, trained on sample frames spanning MNIST World subsets and converting videos into four-channel latent representations. MNIST World reconstruction MSE reaches 0.02, while the authors expect the resulting latent-space error not to affect downstream DFoT convergence.

  • VAE design: The framework performs diffusion in VAE latent space rather than directly on pixels, using an 8× spatial downsampling VAE.The authors note that pixel-space diffusion might produce perceptually different results but is not expected to alter model results.
  • VAE training: The VAE is trained on sample frames from a mixture of all MNIST World subsets, covering combinations of overlapping MNIST digits.This training setup keeps the combinations represented in the dataset within the VAE’s training distribution.
  • Reconstruction quality: 0.02 reconstruction MSE is achieved for MNIST World, so DFoT MSE may be 0.02 higher than pixel-trained MSE without expected downstream convergence effects.The diffusion model only operates on the latent space, which motivates the authors’ expectation.
  • Latent representation: For MNIST World, latent diffusion inputs videos as [num frames, 4, height // 8, width // 8] after encoding frames with latent dimension 4 and spatial downsampling ratio 8.The original video shape is [num frames, channels, height, width].
Loading 2601.01075v2…