PaperlayerView exact revision on arXiv

Source-linked AI summary

MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Driving

Haiguang Wang, Daqi Liu, Hongwei Xie, Haisong Liu, Enhui Ma, Kaicheng Yu, Limin Wang, Bing Wang

arXiv:2503.15875v1cs.CV

TL;DR

Autonomous-driving systems need rare, diverse data, while existing world models struggle with error accumulation and consistency in long videos. MiLA uses coarse-to-(Re)fine generation with JDC and TPD modules, achieving state-of-the-art nuScenes video-generation performance and videos up to one minute. The framework generates low-FPS anchors, interpolates high-FPS frames, and jointly restores anchors to enhance temporal consistency.

  • Problem

    Rare and diverse autonomous-driving data is costly to collect, while existing long-video methods accumulate errors and lose consistency, especially around dynamic objects.

    Source: §1 · p. 1, §2.2 · p. 4

  • Method

    MiLA predicts low-FPS anchors, interpolates high-FPS frames, and jointly corrects anchors using Coarse-to-(Re)fine generation, JDC, and TPD modules.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    MiLA achieves state-of-the-art short- and long-video generation performance on nuScenes, surpassing DriveDreamer-2 by 7.1 FID and 40.8 FVD for multiview generation.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    MiLA generates controlled, high-fidelity driving scenes lasting up to one minute while maintaining long-term video fidelity.

    Source: §1 · p. 2, §5 · p. 8

    The paper’s JDC formulation assumes anchor-frame noise can be decomposed into structured and stochastic components.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

In recent years, data-driven techniques have greatly advanced autonomous driving systems, but the need for rare and diverse training data remains a challenge, requiring significant investment in equipment and labor. World models, which predict and generate future environmental states, offer a promising solution by synthesizing annotated video data for training. However, existing methods struggle to generate long, consistent videos without accumulating errors, especially in dynamic scenes. To address this, we propose MiLA, a novel framework for generating high-fidelity, long-duration videos up to one minute. MiLA utilizes a Coarse-to-Re(fine) approach to both stabilize video generation and correct distortion of dynamic objects. Additionally, we introduce a Temporal Progressive Denoising Scheduler and Joint Denoising and Correcting Flow modules to improve the quality of generated videos. Extensive experiments on the nuScenes dataset show that MiLA achieves state-of-the-art performance in video generation quality. For more information, visit the project website: https://github.com/xiaomi-mlab/mila.github.io.

1. Introduction

MiLA targets the high cost of collecting rare autonomous-driving data and the error accumulation and consistency problems that limit long-duration video generation. It combines coarse-to-(Re)fine generation with temporal denoising and joint correction to produce high-fidelity extended driving videos.

Motivation

  • Long-duration generation is needed because rare and long-tail autonomous-driving scenarios require substantial equipment and human-resource investment to collect.World models are presented as a way to understand and predict future environmental states for addressing this data challenge.Source: §1 · p. 1

Challenges

  • Recursive batch-by-batch generation accumulates errors over time, degrading both static backgrounds and dynamic objects in long-term videos.Generating all frames simultaneously is described as computationally and memory intensive, motivating recursive strategies despite their error accumulation.Source: §1 · p. 2
  • Conditioning only on initial observations weakens over extended sequences, causing scene degradation and loss of coherence in later frames.The passage notes that maintaining consistency without extra signals such as HD-maps and predicted-frame 3D boxes remains largely unexplored.Source: §1 · p. 2

Contributions

  • MiLA produces extended driving scenes lasting up to one minute without degradation of fidelity through its coarse-to-(Re)fine process.The framework is designed as a high-fidelity long-term video generation world model controlled by signals and previous frames.Source: §1 · p. 2

Method

  • MiLA first computes low-FPS anchor frames, then completes high-FPS video while jointly correcting anchor artifacts and improving long-video smoothness.The Coarse-to-(Re)fine module addresses temporal incoherence in predicted anchors, while TPD enriches temporal information through the generation pipeline.Source: §1 · p. 2

2. Related Work

World-model research spans environmental simulation and autonomous-driving video generation, while long-term video methods mainly use recurrent or divide-and-conquer generation. MiLA’s context addresses the accumulated-error and anchor-artifact limitations of these paradigms.

2.1. World Models

  • World models learn to simulate environments and predict their dynamics, supporting agent training in simulated spaces.In autonomous driving, they include future 3D-representation prediction and RGB-video generation paradigms.Source: §2.1 · pp. 2–3
  • MiLA’s coarse-to-(Re)fine pipeline combines low-FPS anchor prediction with high-FPS interpolation and recurrent refinement using prior high-FPS frames and coarse anchors.Its overview also incorporates multi-view video, FPS, waypoints, camera parameters, and scene descriptions as inputs.Source: Figure 3 · p. 3
  • Autonomous-driving world models either forecast future geometric representations such as lidar and occupancy or generate future RGB videos.RGB-video methods use signals including HD-maps, bounding boxes, reference frames, and waypoints, but Vista still struggles with undistorted long-term video.Source: §2.1 · p. 3

2.2. Long-term Video Generation

  • Recurrent generation creates each new batch from the preceding batch, but accumulated error remains insufficiently addressed.An asynchronous diffusion variant divides batch generation into frames while maintaining batch size with a new noise token.Source: §2.2 · p. 3
  • Divide-and-conquer methods generate anchor and interpolation frames through separate processes, yet within-clip consistency and predicted-anchor artifacts remain concerns.The literature has paid limited attention to how artifacts inherent in predicted anchor frames affect overall video quality.Source: §2.2 · pp. 3–4

3. Method

MiLA combines a multi-view diffusion backbone with a Coarse-to-(Re)fine long-video pipeline, Joint Denoising and Correcting Flow, and Temporal Progressive Denoising Scheduler. These components generate anchor and interpolated frames while addressing temporal artifacts, motion inconsistency, and denoising efficiency.

3.1. Model Overview

  • MiLA uses Open-Sora’s DiT-based video-generation backbone to predict noisy frames across multiple views, with camera parameters and waypoints embedded as controls.Multi-view spatial attention expands each DiT block’s receptive field across all views’ patches.Source: §3.1 · p. 4

3.2. Long-term Video Generation Framework

  • The Coarse-to-(Re)fine pipeline first predicts low-FPS anchor frames, then recurrently generates high-FPS frames using both anchor and previous refined frames.This combination captures motion dynamics while preserving broader environmental context.Source: §3.2 · p. 4

3.3. Joint Denoising and Correcting Flow

  • Joint Denoising and Correcting Flow synchronously optimizes anchor and interpolated frames to preserve structural fidelity while restoring temporal continuity.The module targets artifacts from sparse anchor frames, including unstable object motion and reduced video smoothness.Source: §3.3 · pp. 4–5
  • The correction procedure adds Gaussian noise to predicted anchor frames with a timestamp-dependent scale, then uses denoising to rectify anchor-frame artifacts.The noise scale and standard deviation are derived from the predicted and ground-truth anchor-frame formulation.Source: §3.3 · p. 5

3. Method

  • On the nuScenes validation set, MiLA outperforms state-of-the-art world models in generation quality in both Table 1 and Table 2 comparisons of generation fidelity.Source: Table 1 · p. 6, Table 2 · p. 6

3.4. Temporal Progressive Denoising Scheduler

  • Temporal Progressive Denoising Scheduler denoises earlier frames rapidly and trailing frames more slowly before accelerating them near the final steps.Its schedule is computed from noisy temporal frame indices and denoising timestamps, using a normalized decreasing cosine-based function.Source: Figure 4 · p. 5, §3.4 · p. 6

3.5. Loss Function

  • The training objective minimizes error between predicted flow and the flow from Gaussian starting noise to ground-truth latent frames, using a mask for noisy anchor frames.The predicted flow is v(x_t, t), while y = (x_1 − x_0) is the target flow.Source: §3.5 · p. 6

4. Experiments

Experiments on nuScenes evaluate MiLA’s multi-view and long-video generation using FID/FVD, qualitative comparisons, and module ablations. MiLA maintains fidelity across views and long sequences, while TPD and JDC improve generation quality.

4.1. Experimental Setup

  • Experiments use nuScenes validation data with FID and FVD, including front-view, temporal-range, and multi-view evaluation metrics.The dataset contains 700 training and 150 validation scenes from Boston and Singapore.Source: §4.1 · p. 7

4.2. Evaluation Results

  • MiLA generates spatially and temporally consistent multi-view videos and maintains fidelity in long-term generation without the degradation seen in recurrent generation.The method also enhances dynamic-object structure and reduces white-van distortions.Source: §4.2 · p. 7, Figure 6 · p. 7
  • MiLA significantly surpasses prior approaches in front-view and multi-view FID and FVD, including gains of 7.1 FID and 40.8 FVD over DriveDreamer-2 for multi-view generation.In a fair comparison, MiLA also surpasses Vista by 2.0 FID and 46.5 FVD.Source: §4.2 · p. 8

4.3. Ablation Studies

  • Integrating Temporal Progressive Denoising and Joint Denoising and Correcting improves overall generation quality compared with alternative component settings.Anchor-frame ablations distinguish absent, ground-truth, and predicted low-FPS anchor frames.Source: §4.3 · p. 8, Table 4 · p. 8
  • JDC requires calibrating noise amplitude because excessive noise harms scene structure, whereas insufficient noise fails to model dynamic-object deformation.The trade-off balances artifact suppression against motion fidelity.Source: Figure 7 · p. 7, §4.3 · p. 8
  • The Temporal Progressive Denoising Scheduler improves generation quality across videos with varying durations, frame counts, and frame rates.Table 3 evaluates the scheduler using the same FID and FVD metrics as the main experiments.Source: §4.3 · p. 8, Table 3 · p. 8

5. Conclusion

MiLA generates driving-scene videos from prior frames and future waypoints through a coarse-to-(Re)fine process. Its JDC and TPD modules target temporal consistency, while anchor frames are interpolated and jointly restored.

5. Conclusion

  • MiLA generates driving-scene videos from prior frames and future waypoints using a coarse-to-(Re)fine framework.The framework first generates low-FPS anchor frames, then interpolates high-FPS frames and jointly restores the anchors.Source: §5 · p. 8
  • JDC and TPD are designed to enhance temporal consistency in MiLA’s long-term video generation.Source: §5 · p. 8

A.1. Equation Derivation

The derivation specifies how MiLA computes the noise parameters used for predicted low-FPS anchor frames. It preserves the rectified-flow requirement that the noise term follow a standard Gaussian distribution.

A.1. Equation Derivation

  • The supplementary derivation focuses on computing α2 and σ2 for the noise-adding procedure applied to predicted low-FPS anchor frames.Source: §A.1 · p. 12
  • MiLA constrains the rectified-flow noise term to a standard Gaussian distribution, n ∼ N(0, 1), before rewriting the supplementary equations.Source: §A.1 · p. 12
  • The final derivation combines the supplementary equations to obtain the required noise distribution for the anchor-frame correction process.Source: §A.1 · p. 12

B. Implementation Detail

MiLA is trained and evaluated through staged video modeling, progressive denoising, and anchor-frame correction. Implementation uses nuScenes clips, multi-stage optimization, and visualizations of temporal denoising and anchor correction.

B. Implementation Detail

  • MiLA trains on 16×N frames with N ∈ [1, 2, 3], using 1 or 4 initial visual-condition frames and predicting the remaining frames.Source: B. Implementation Detail · p. 12
  • MiLA resizes images to 360 × 640 and contains roughly 700M parameters excluding the VAE and T5 text encoder, trained on 64 NVIDIA A100 GPUs.Source: B. Implementation Detail · p. 12
  • Training proceeds in three stages: next-12-frame prediction, short- and long-term generation, and additional anchor-frame correction training.The third stage corrupts tail anchor frames and trains correction with a 75% rate.Source: B. Implementation Detail · p. 12
  • The Temporal Progressive Denoising module improves clarity and consistency across views at the 12th predicted frame.Figure 8 compares results with and without the module.Source: Figure 8 · p. 12
  • Inference samples nuScenes validation clips and generates video tokens for 50 steps using the temporal progressive denoising scheduler with classifier-free guidance scale 3.0.Source: B. Implementation Detail · pp. 12–13
  • The implementation includes trajectory-guidance and video-smoothness comparisons, with Divide-and-Conquer and Coarse-to-(Re)fine identified as competing frameworks.Source: Table 5 · p. 13, Table 6 · p. 13

C.1. Additional Results for Progressive Denoising

MiLA’s progressive denoising captures more scene detail, while trajectory guidance becomes increasingly important as video length grows.

C.1. Additional Results for Progressive Denoising

  • Temporal progressive denoising successfully captures more detailed scene information in generated videos.The denoising schedule assigns earlier denoising to frames closer to the conditional frames.Source: §C.1 · p. 13

C.2. Additional Results for Trajectories’ Guidance

  • Trajectory guidance has a stronger effect for longer videos, with 24-frame models significantly outperforming unguided models.For 12-frame videos, FID and FVD are compatible with or without trajectory data for both front-view and multiple-view models.Source: §C.2 · p. 13

C.3. Additional Results for Long Video Generation

MiLA’s Coarse-to-(Re)fine framework preserves fidelity in long dynamic-scene generation, improves consistency and smoothness through correction, and handles difficult visual cases.

C.3. Additional Results for Long Video Generation

  • MiLA’s Coarse-to-(Re)fine framework maintains fidelity throughout 12-second generation, unlike recurrent variants that distort dynamic scenes after 10 seconds.The comparison uses Vista and MiLA with recurrent generation against the proposed framework using front-view frames.Source: §C.3 · p. 13
  • The correction process fixes distorted objects and yields better video consistency and smoothness than the Divide-and-Conquer framework.Objects highlighted in red boxes are corrected in the resulting green-box views.Source: §C.3 · p. 13

C.4. Additional Visualization Results

  • MiLA generates high-fidelity videos in most difficult cases, including complicated streets, numerous vehicles, and fast cars in darkness.The demonstrations include a 12-FPS long-term clip and videos across varied scenes.Source: §C.4 · p. 14

C.5. Additional Results of Downstream Model Recognition

The evaluation addresses structural recognition beyond visual quality and reports that MiLA-generated frames remain compatible and recognizable to a downstream VAD model.

C.5. Additional Results of Downstream Model Recognition

  • Video quality alone is insufficient for evaluating generated driving scenes, motivating structural assessment through 3D layout perception or road segmentation.The paper notes that no existing method provides code for evaluating such structural similarities.Source: §C.5 · p. 14
  • MiLA-generated frames exhibit compatibility and recognizability when evaluated by a VAD model trained on the nuScenes training set.The evaluation replaces all image inputs with frames generated by MiLA and does not use explicit 3D structural information during generation.Source: §C.5 · p. 14
  • The supplementary visualizations cover denoising schedules, 12-second and 30-second videos, and short-term 12-FPS generation conditioned on four frames.These figures provide qualitative demonstrations of temporal denoising, long-term generation, and hard-case performance.Source: Figure 10 · p. 15, Figure 11 · p. 16, Figure 12 · p. 17

Related papers

Loading 2503.15875v1…