Scaling Diffusion Mamba with Bidirectional SSMs for Efficient Image and Video Generation

• Diffusion and self-attention models face quadratic input-length complexity, a barrier as image resolution increases.Source: §1 · p. 1
• Mamba efficiently models long sequences, but its direct application to image generation has not been thoroughly explored.Source: §1 · p. 1
• DiM replaces attention mechanisms with Mamba-based state-space modeling, maintaining linear sequence-length complexity and targeting high-resolution generation.Source: §1 · pp. 1–2
• DiM achieves better FID-50k results than DiT [4] across all training steps and model sizes.Source: Figure 1 · p. 2
• The paper benchmarks DiM against state-of-the-art models across resolutions and reports experiments on ImageNet images and UCF-101 videos.Source: §1 · p. 2
• The contributions describe DiM as adapting across operational scales while reducing Gflops compared with previous methods.Source: §1 · p. 2

• State-space models such as S4 and Mamba are described as processing long sequences with fewer parameters and lower computational overhead than traditional recurrent neural networks.Source: §2 · p. 2
• Diffusion models are presented as capable of high-quality image synthesis, image restoration, and speech generation.Source: §2 · p. 2
• Diffusion Transformers extend latent-patch denoising to high-fidelity images and complex 3D structures, with extensions covering multiple modalities.Source: §2 · p. 2
• The illustrated DiM pipeline encodes images with a pre-trained Stable Diffusion VAE, forms patch embeddings, applies bidirectional SSM blocks, and predicts latent noise.Source: Figure 2 · p. 3

3 Method

• DiM applies DDPM denoising to latent image embeddings through a diffusion framework and a DiM block.Source: §3 · p. 3

3.1 Preliminaries

• The setup uses class-labeled RGB images for classifier-free guidance and trains a denoising network to reverse the DDPM noising process.The forward process progressively adds noise to latent variables; the reverse process predicts noise and denoises them.Source: §3.1 · pp. 3–4

3.2 Diffusion Mamba

• DiM converts image latents into flattened patches, projects them, adds positional embeddings and a class token, then processes them through encoder layers.Source: §3.2 · p. 4

3.3 DiM Block

• DiM blocks normalize and project tokens, process them bidirectionally with SSM-derived parameters, and combine both directional outputs.Configurations cover patch sizes 2, 4, and 8 and model sizes S through XL, with parameter counts comparable to DiT counterparts.Source: §3.3 · p. 5, Table 1 · p. 5

3.4 Efficiency Analysis

• 8NLD (N=16) is the stated DiM computation cost, linear in sequence length L and feature dimension D.The section compares DiM’s efficiency with DiT and DiffuSSM and describes its handling of large-scale data without sacrificing performance.Source: §3.4 · p. 5

3.5 Scaling to Video Generation

• For video, DiM maps each frame’s latent into flattened spatial patches and processes sequences bidirectionally across both frames and spatial dimensions.This design is described as preserving temporal continuity by informing each frame with its predecessors and successors.Source: §3.5 · p. 6

4.1 Experimental Setup

• Image evaluation uses ImageNet and FID, Inception Score, sFID, and Precision/Recall; video evaluation uses 16-frame UCF-101 clips and FVD.FID-50K uses 250 DDPM sampling steps; FVD is computed over 2,048 clips.Source: §4.1 · p. 6
• The implementation follows DiT specifications for images and Latte specifications for videos, using AdamW and EMA weights for reported results.Video generation uses a pretrained Stable Diffusion 1.4 variational autoencoder for latent representations.Source: §4.1 · p. 6

4.2 Comparison to prior work

• Table 2 tracks DiM-B/4 and DiT-B/4 across 400K ImageNet 256x256 training steps, with FID-50K checkpoints used to compare convergence.The passage describes the checkpoint analysis as showing how quickly each model converges to high-quality synthesis.Source: §4.2 · p. 7, Table 2 · p. 7
• DiM+IMG reports FVD 206.83 versus Latte+IMG at 333.61 on UCF-101; DiM is also reported to outperform DiT across image model scales.The image comparison uses FID-50K across S/4, B/4, L/4, and XL/4 configurations.Source: Table 3 · p. 7, §4.2 · p. 7
• Qualitative video examples show high-fidelity, diverse outputs while identifying complex scenes and motion blur as areas of strength or limitation.Figure 3 presents UCF-101 video-generation visualizations.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3 Experimental analysis

• Ablations examine Mamba-related Gflops, patch-size effects on quality and computational demand, and model-size trade-offs between compute and image quality.The patch-size and model-size passages identify their respective tables as the source of the findings but provide no cell values.Source: §4.3 · pp. 8–9
• The CFG analysis tests whether removing direct classifier dependencies improves conditional-image versatility and quality; CFG=1.0 denotes generation without guidance.The supplied description frames improvement as the question being assessed, not as a reported outcome.Source: §4.3 · p. 9, Table 7 · p. 9

5 Conclusion

• The conclusion reports reduced computational demands and scaling with input size and model dimensions, alongside high-fidelity, diverse image and video outputs.It also describes ablations as showing efficiency improvements without compromising generated quality.Source: §5 · p. 9
• DiM's video evaluation has not covered a wide range of video types and qualities, and its performance on highly dynamic or low-visibility content remains unassessed.The conclusion also says long-term dependencies over extended video sequences may not be captured effectively, a concern for long-form generation.Source: §5 · p. 9
• The paper identifies media production and education or training as potential applications for efficient visual-content generation.These are presented as possible positive impacts, including more accessible media production and enhanced learning materials or simulations.Source: §5 · p. 9