PaperlayerView exact revision on arXiv

Source-linked AI summary

MarDini: Masked Autoregressive Diffusion for Video Generation at Scale

Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yanping Xie, Xiao Han, Juan C. Pérez, Ding Liu, Kumara Kahatapitiya, Menglin Jia, Jui-Chieh Wu, Sen He, Tao Xiang, Jürgen Schmidhuber, Juan-Manuel Pérez-Rúa

arXiv:2410.20280v1cs.CVcs.AI

TL;DR

Video generation must model temporal consistency and complex motion, while diffusion models incur high inference costs. MarDini combines masked autoregressive temporal planning with high-resolution diffusion generation in an asymmetric network design. It achieves state-of-the-art video interpolation performance on FVD and supports video expansion using input motion cues.

  • Problem

    Video generation requires modeling temporal consistency and complex motion, while diffusion models incur high inference costs from their multi-step process.

    Source: §1 · p. 2

  • Method

    MarDini uses a large masked-autoregressive planning network on low-resolution masked frames to guide a lightweight diffusion model that generates high-resolution frames.

    Source: §2.1 · p. 2, §1 · p. 2

  • Results

    MarDini achieves state-of-the-art performance on video interpolation under FVD and can predict video sequences from input motion cues in video expansion.

    Source: §3.2 · p. 10, §3.4 · p. 11

  • Takeaways & Limitations

    One model supports long-term video interpolation, video expansion, and image animation with a lower-resolution planning component and frame-level generation.

    Source: §6 · p. 14

    MarDini is not inherently equipped with a text encoder, and the authors excluded text embeddings and motion scores from this version.

    Source: §5 · p. 14

Abstract

from arXiv · show

We introduce MarDini, a new family of video diffusion models that integrate the advantages of masked auto-regression (MAR) into a unified diffusion model (DM) framework. Here, MAR handles temporal planning, while DM focuses on spatial generation in an asymmetric network design: i) a MAR-based planning model containing most of the parameters generates planning signals for each masked frame using low-resolution input; ii) a lightweight generation model uses these signals to produce high-resolution frames via diffusion de-noising. MarDini's MAR enables video generation conditioned on any number of masked frames at any frame positions: a single model can handle video interpolation (e.g., masking middle frames), image-to-video generation (e.g., masking from the second frame onward), and video expansion (e.g., masking half the frames). The efficient design allocates most of the computational resources to the low-resolution planning model, making computationally expensive but important spatio-temporal attention feasible at scale. MarDini sets a new state-of-the-art for video interpolation; meanwhile, within few inference steps, it efficiently generates videos on par with those of much more expensive advanced image-to-video models.

1 Introduction

Continuous, high-dimensional visual data are harder to model through one-dimensional autoregression, and video adds demands for temporal consistency and complex motion. MarDini combines masked autoregression with diffusion, assigning temporal planning and spatial generation to distinct networks.

1 Introduction

  • MarDini combines masked autoregression and diffusion in an asymmetric design: MAR handles long-range temporal modeling, while DM generates spatial detail.The MAR network has more parameters and operates at lower resolution; the DM has fewer parameters and operates at higher resolution.Source: §1 · p. 2
  • MAR predicts planning signals from randomly masked low-resolution frames, and the diffusion model uses them with unmasked frames to recover masked high-resolution frames from noise.Training uses distinct processes at different resolutions and a shared training batch.Source: §1 · p. 2
  • A single model supports image-to-video generation, video expansion, and interpolation through different masking strategies, and can generate slow motion by hierarchically masking middle frames across inferences.Examples include masking all but the first frame, masking subsequent frames, or masking the middle frames between first and last frames.Source: §1 · p. 2
  • Mask-ratio tuning lets MarDini scale from video interpolation toward full video generation without relying on generative image pre-training.The training recipe progressively adjusts task difficulty.Source: §1 · p. 2
  • Allocating more computation to lower resolutions makes MarDini memory-efficient and fast at inference, allowing computationally intensive spatio-temporal attention at scale.The authors connect this design to improved ability to model complex motion dynamics.Source: §1 · p. 2

2 MarDini: An Efficient and Asymmetric Video Diffusion Model

MarDini splits video generation between low-resolution masked auto-regressive planning and high-resolution diffusion denoising in an asymmetric network. Its progressive training recipe moves from interpolation toward generation with fewer preserved frames.

2.1 Design Overview

  • The heavy MAR planner turns randomly masked low-resolution frames into signals that guide a lightweight diffusion model in reconstructing noisy, masked high-resolution frames.The diffusion model also receives high-resolution reference-frame conditions, supporting temporal coherence and attention to fine-grained details.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2 Data Representation and Notations

  • A pretrained VAE compresses video into continuous latents; MAR processes masked low-resolution latents, while the generation model operates on high-resolution latents.The VAE uses 16 latent channels and 8× spatial compression; planning signals are shaped as Nlow × Clow.Source: §2.2 · p. 3

2.3.1 MarDini Block Design

  • Both networks use transformers, but MAR uses spatio-temporal attention on low-resolution inputs while the DiT-style DM uses temporal attention at high resolution.MAR also uses 2D RoPE for spatial and temporal positions; the DM incorporates diffusion-step and planning conditioning and cross-attends to planning features.Source: §2.3.1 · pp. 4–5, Figure 2 · p. 4

2.3.2 Identity Attention

  • Identity Attention isolates clean [REF] tokens while giving [NOISE] tokens global attention, and the authors report improved training stability in both models.The authors associate instability with the distributional difference between clean and noisy tokens and their varying positions and lengths.Source: §2.3.2 · p. 5, Figure 3 · p. 5

2.3.3 Model Configuration

  • The four configurations pair 3.1B- or 1.3B-parameter planners with generation models 3× or 10× smaller; spatio-temporal variants are limited to nine frames for VIDIM-Bench comparison.Autoregressive generation means output video length is not constrained by that evaluation setting.Source: §2.3.3 · p. 5, Table 1 · p. 5

2.4.1 Training Tasks: From Frame Interpolation to Video Generation

  • Training proceeds from separately warming up the planner and generator to joint end-to-end interpolation training, then joint-task training; the multi-stage recipe is described as essential for stable training.The paper presents this staged recipe as enabling training from scratch using unlabeled video data.Source: §2.1 · p. 3, §2.4 · pp. 5–6, §2.4.1 · pp. 6–7
  • In the final joint-task stage, progressively decreasing the proportion of preserved frames shifts training from interpolation toward single-image-to-video generation and supports inputs at arbitrary temporal positions.The passage describes generation conditioned on a variable number of input frames.Source: §2.4.1 · p. 7

2.4.2 DM Architecture: From Spatio-Temporal to Temporal Attention; 2.4.3 Data: Progressive Configuration of Specifications

  • After using spatio-temporal attention in initial DM training, integration replaces it with cheaper temporal attention, while training progressively increases video resolution and duration.The attention change is made to increase generation-model efficiency; the data progression is presented as supporting efficient resource use and scaling to more complex, high-resolution video.Source: §2.4.2 · p. 7, §2.4.3 · p. 7

3 Experiments

The experiments examine MarDini’s component roles, efficiency, benchmark performance, and additional generation settings.

3.1 Ablation Studies and Analysis

  • MAR-only interpolation performs poorly, and a small DM without global guidance is suboptimal; combining MAR planning with DM yields the best reported results.The authors conclude both components benefit video generation.Source: §3.1 · p. 8, Table 2 · p. 8
  • The asymmetric design generates 12-frame clips at 512 resolution in a few seconds, while a symmetric MAR/DM setup cannot fit available GPU memory as resolution or frame count grows.The reported speed could improve further; experiments used mixed precision without caching, FSDP, or static compilation.Source: §3.1 · p. 8, Table 3 · p. 8
  • Without MAR planning, DM produces motion distortions or incorrect movement; the planning signal addresses these flaws and supports long-term coherence.The figure compares generations conditioned on the middle frame, with examples of pixel distortions and incorrect motion.Source: §3.1 · p. 8, Figure 5 · p. 9
  • During 256-to-512 resolution scaling, interpolation and image animation improve consistently; Identity Attention stabilizes early DM training, with a loss decrease after 6K steps attributed to learning-rate warm-up.A carefully tuned mask ratio supports training both tasks in a unified manner.Source: §3.1 · p. 9, Figure 6 · p. 9

3.2 Results on Video Interpolation

3.3 Results on Image-to-Video Generation

  • On VBench, MarDini performs on par with existing methods across several metrics while showing higher efficiency and requiring no generative image pre-training.The authors caution that video-quality comparisons, especially for dynamic degree, are not entirely fair without language or motion-score conditioning.Source: §3.3 · p. 10, Table 5 · p. 11

3.4 Additional Applications

  • Exploratory applications include two-view novel-view synthesis without 3D training or camera control, and video expansion from frame-set motion cues; these applications lack rigorous ablations or quantitative comparisons.For the expansion visualization, the model predicts 12 frames from 16 consecutive input frames.Source: §3.4 · p. 11, Figure 8 · p. 12
  • Hierarchical autoregressive inference recursively expands clips, turning four images into a 128-frame slow-motion video with a 32-frame window and demonstrating generation beyond the training window size.The process expands each clip segment and uses the expanded segment for recursive interpolation.Source: §3.4 · p. 11

4 Related Work

Related work motivates MarDini through limits in visual autoregression and the cost of video diffusion, then positions its asymmetric planning-and-generation design as a different way to make spatio-temporal modeling feasible.

4 Related Work

  • Visual autoregressive approaches replace causal with bidirectional attention, while discrete tokenization remains a concern because of training instability and limited capacity (Yu et al., 2024; Chang et al., 2022; Li et al., 2023a; Yu et al., 2023a; Chang et al., 2023; Yu et al., 2023a; Yu et al., 2023b; Chang et al., 2023; Team, 2024; Xie et al., 2024; Esser et al., 2021; Yu et al., 2021; Van Den Oord et al., 2017; Li et al., 2024; Ramesh et al., 2021; Razavi et al., 2019; Zhou et al., 2024).Recent work shifts masked auto-regressive models toward continuous latent spaces to address these concerns.Source: §4 · p. 12
  • Dynamic mask ratios and asymmetric input resolutions distinguish MarDini from earlier continuous-latent masked auto-regressive work.The paper says progressive mask ratios improve scalability and stability, while the asymmetric design allows MAR to be trained effectively with full-resolution inputs.Source: §4 · p. 13
  • Diffusion models are a leading approach for image and video generation, refining real samples gradually from Gaussian noise and demonstrating significant scalability and stable training.Source: §4 · p. 13
  • MarDini partitions most parameters into a low-resolution auto-regressive planning model and uses a lightweight diffusion model to turn its signals into high-resolution outputs.Cross-attention connects the asymmetric-resolution inputs for more efficient training and inference; the paper contrasts this with auto-regressive diffusion whose costs rise with resolution.Source: §4 · p. 13

5 Limitations and Future Works

The authors devote most computational resources to pretraining and identify post-training and richer conditioning as areas for future work.

5 Limitations and Future Works

  • Most computational resources went to pretraining, leaving post-training less emphasized despite its recognized importance in generative models (Dai et al., 2023a; Dubey et al., 2024; Touvron et al., 2023).Future post-training targets include long-term planning, motion quality, and higher resolutions.Source: §5 · p. 13
  • Long-term planning, motion quality, higher resolutions, and broader applications are stated future-work goals.These goals depend on planned post-training and conditional-signal additions.Source: §5 · pp. 13–14
  • MarDini has no text encoder and omits text embeddings and motion scores to conserve resources and quickly validate the method.The authors plan to add these conditional signals to broaden the model’s applications.Source: §5 · p. 14

6 Conclusion

MarDini combines a large masked auto-regressive planner with a smaller diffusion generator, and the authors report competitive benchmark results at lower computational needs than comparable-parameter counterparts.

6 Conclusion

  • MarDini’s heaviest component runs once on lower-resolution inputs, while its generative module adds frame-level details under high-level conditioning.The model applies a masked auto-regressive loss directly at the frame level.Source: §6 · p. 14
  • A single MarDini model supports long-term video interpolation, video expansion, and image animation.Source: §6 · p. 14
  • MarDini achieves competitive results on interpolation and animation benchmarks with lower computational needs than counterparts of comparable parameter size.Source: §6 · p. 14

Ethics Statement

The authors frame the work as theoretical neural-architecture research and report that MarDini was trained exclusively on licensed Shutterstock data.

Ethics Statement

  • The paper frames its focus as theoretical neural-architecture design rather than specific commercial applications.Source: Ethics Statement · p. 14
  • The authors say MarDini’s potential negative impacts align with those of other video-generation models and present no unique risks requiring special consideration.Source: Ethics Statement · p. 14
  • MarDini was trained exclusively on a licensed Shutterstock dataset, which the authors say avoids the copyright concerns associated with web-scale training data.Source: Ethics Statement · p. 14

A Reconstruction metrics in Video Interpolation.

The interpolation examples show a mismatch between visual sharpness and reconstruction scores: sharper MarDini frames can score worse, while blurrier images sometimes receive higher reconstruction errors.

A Reconstruction metrics in Video Interpolation.

  • Blurrier images sometimes receive higher reconstruction error scores in video interpolation.The observation concerns SSIM and LPIPS in Figure 9.Source: A Reconstruction metrics in Video… · p. 20
  • MarDini’s generated frames are sharper than competitors’ in the Figure 9 examples, despite worse corresponding reconstruction scores.The figure shows generated frames alongside their ground-truth counterparts.Source: Figure 9 · p. 20
  • Figure 9 compares two generated frames with their corresponding ground-truth frames alongside SSIM and LPIPS reconstruction scores.Source: Figure 9 · p. 20

B MarDini Training Strategies

MarDini is trained on 34 million Shutterstock videos using 256 H100 GPUs, with batch sizes adapted to resolution and frame count. Dynamic-resolution training is omitted from the main stages, then added in a brief fine-tuning phase.

B MarDini Training Strategies

  • Training uses 34 million Shutterstock videos and 256 H100 GPUs with a distributed MAST scheduler.Source: B MarDini Training Strategies · p. 20
  • At 256 × 256 resolution with 9 frames, the batch size is 1024; at 512 × 512 with 9 frames, it is 720.These settings process 9K and 6480 frames per iteration, respectively.Source: B MarDini Training Strategies · p. 20
  • Both stages use AdamW with a 1.4 × 10−4 learning rate and a cosine learning-rate scheduler.Source: B MarDini Training Strategies · p. 20
  • Dynamic-resolution training is excluded from the main stages because it slows training, but 10K–20K fine-tuning steps after convergence enable support for it.Source: B MarDini Training Strategies · p. 20
  • For inference, image-to-video uses CFG scale 2.5 with DDIM, while video interpolation omits classifier-free guidance as redundant.FSDP and activation checkpointing are also enabled to save GPU memory.Source: B MarDini Training Strategies · p. 20

C Visualization of Video Interpolation

The interpolation visual compares MarDini with FILM, LDMVFI, and VIDIM under first-and-last-frame conditioning. The accompanying text reports an advantage for MarDini in large-motion modeling.

C Visualization of Video Interpolation

D Benchmarks

The paper evaluates interpolation on VIDIM-Bench and image animation on VBench, with specified task conditions and metrics. It also illustrates hierarchical generation of 128 frames from an initial four, beyond the 32-frame training window.

D Benchmarks

  • Evaluation covers interpolation on VIDIM-Bench and image animation on VBench.Source: D Benchmarks · p. 21
  • VIDIM-Bench interpolation generates seven intermediate frames conditioned on the first and last frames, using about 400 DAVIS and UCF-101 videos.FVD and FID measure generation; SSIM and LPIPS measure reconstruction on the middle, fifth frame.Source: D Benchmarks · p. 21
  • VBench omits video-text camera motion for lack of text supervision and excludes Dynamic Degree from comparable video-quality averages because MarDini lacks dynamic-degree guidance.Latency is compared on one Nvidia A100 80G GPU, with results normalized by frame count at 512 × 512 or 768 × 768 resolution.Source: D Benchmarks · p. 21
  • MarDini hierarchically generates a complete 128-frame video from an initial four frames, extending beyond its 32-frame training window.Source: Figure 11 · p. 22

Related papers

Loading 2410.20280v1…