Matten: Video Generation with Mamba-Attention

1 Introduction

• Global attention can model interactions across different spatial and temporal positions, but its quadratic computation is costly; Mamba scans struggle with localized patterns.The paper identifies these as complementary challenges in video-content modeling.Source: §1 · p. 1
• Matten assigns global temporal relationships to Mamba and spatial and local-temporal relationships to attention as its most effective design.The paper introduces this Mamba-Attention architecture as a latent diffusion model for video generation.Source: §1 · p. 2
• Across unconditional and conditional video-generation benchmarks, Matten reports comparable FVD scores and efficiency with SOTAs, with lower computational and parameter requirements and strong scalability.The authors also report a direct positive relationship between model complexity and generated-sample quality.Source: §1 · p. 2
• The authors designed four model variants to explore Mamba-attention combinations and identify an arrangement for local spatio-temporal details and global information.Their broader evaluation reports comparable performance with lower computational and parameter requirements.Source: §1 · p. 2

2.1 Video Generation

• Video generation research includes GANs, autoregressive models, and diffusion models, with the latest work primarily using CNN-based U-Nets or Transformers.The passage notes possible GAN mode collapse and substantial computational demands for autoregressive methods.Source: §2.1 · p. 2

2.2 Mamba

• Mamba and related state-space models address long-range dependencies across diverse applications, while Matten targets global temporal dimensions beyond prior video-diffusion work focused mainly on local temporal modeling.The passage identifies [61] as closely related for using Mamba in a video-diffusion temporal layer.Source: §2.2 · p. 3

3.1 Background

• Latent diffusion encodes samples into latent states, adds noise over timesteps, then uses a Mamba-based noise estimator during denoising.The model employs both Lsimple and Lvlb to refine effectiveness and efficiency.Source: §3.1 · p. 3
• SSMs map 1-D inputs to outputs through an N-dimensional latent state; the bidirectional vision block scans flattened visual sequences forward and backward using a work-efficient parallel scan.The parallel scan reduces sequential dependencies associated with recurrent computation.Source: §3.1 · pp. 3–4

3.2 The Model Variants of Matten

• Four Matten variants process latent video tokens using global-sequence Mamba, interleaved spatial and temporal Mamba, hybrid attention and Mamba, or spatial-first Mamba with temporal attention.The variants use patch and reshaped latent-video embeddings.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

3.3 Conditional Way of Timestep or Class

• Conditioning enters either as tokens or through M-AdaN, which derives γc and βc from class or timestep input and applies adaptive normalization before Mamba residual connections.The adaptive normalization is integrated within the Mamba block.Source: §3.3 · pp. 5–6

3.4 Analysis of Mamba and Attention

• Self-attention cost grows quadratically with sequence length J, while SSM operations scale linearly; Mamba suits long global sequences, and attention can suit shorter local relationships.The analysis states that N is typically fixed at 16.Source: §3.4 · p. 6

4 Experiments

• The section covers experimental settings, ablations of design choices and model size, and comparisons on four common datasets.The model-configuration tables report parameter counts and FLOPs for Matten variants and describe model-size setups.Source: §4 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 10

4.1 Experimental Detail

• Experiments use four datasets, 16-frame clips resized to 256x256, and FVD scores computed from 2,048 clips per setting.The comparisons include nine prior methods; ablations use AdamW with a fixed 1 × 10^-4 learning rate and EMA-enhanced models.Source: §4.1 · p. 7

4.2 Ablation study

• On SkyTimelapse, Variant 3 performs better as iterations increase, M-AdaN outperforms conditional tokens, and larger models generally improve FVD.The authors attribute the M-AdaN difference to more cohesive timestep and class embedding across video tokens.Source: §4.2 · pp. 8–9, Figure 8 · p. 9

4.3 Comparison Experiment

• Roughly 25% fewer FLOPs than Latte accompanies results that surpass prior works and match methods with image-pretrained weights.The authors also report realistic 256x256 generations across four datasets.Source: §4.3 · pp. 9–10

• Matten uses a Mamba-Attention backbone for video generation, and experiments on four standard benchmarks show strong performance and scalability.Source: §5 · p. 10