PaperlayerView exact revision on arXiv

Source-linked AI summary

MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation

Ishaan Preetam Chandratreya, David Charatan, Basile Van Hoorick, Sergey Zakharov, Vitor Guizilini, Phillip Isola, Vincent Sitzmann

arXiv:2606.09056v1cs.CVcs.LG

TL;DR

Long-range video generation must preserve distant content despite impractically short transformer contexts. MILLIVID learns hierarchical frame latents and generates them coarse-to-fine, producing substantially more consistent long Minecraft rollouts than existing baselines. The approach prioritizes long-range consistency of coarse structure while relaxing consistency requirements for fine detail.

  • Problem

    Autoregressive video models struggle with long-range consistency because limited transformer contexts cause earlier content to be forgotten, while longer contexts are computationally prohibitive.

  • Method

    MILLIVID trains a hierarchical autoencoder and a diffusion model that generate video tokens coarse-to-fine, using many compressed frames and fewer detailed frames within a fixed sequence length.

  • Results

    MILLIVID produces significantly better consistency than FramePack and typical autoregressive rollout under a common sequence-length constraint without sacrificing per-frame quality.

  • Takeaways & Limitations

    The study suggests that multi-scale video generation can recall 3D scene structure over hundreds of frames while conventional models afford only a handful of context frames.

  • Takeaways & Limitations

    MILLIVID is not designed for straightforward fine-tuning from standard video diffusion models and requires about 33% more rollout steps than FramePack.

Abstract

from arXiv · show

Video generative models have become increasingly powerful, but long-range consistency remains challenging to achieve because even a few dozen frames require impractically long transformer sequence lengths. We show that this issue can be mitigated by generating video using coarse-to-fine rollout within a multi-scale token space. Our approach is simple: first, we pre-train an autoencoder that compresses each frame into a hierarchy of tokens, with levels ranging from the typical latent resolution to only a handful of tokens per frame. The coarsest levels capture the most consequential information, such as scene layout and semantics, while finer levels add high-frequency appearance and texture. Then, we train a video diffusion model to generate these tokens using coarse-to-fine rollout. By carefully controlling the level of detail at which frames are generated and used as context during each rollout step, we are able to preserve long-range consistency in geometry and object permanence while spending less compute on the long-range consistency of less perceptually relevant details. We validate this approach using a custom dataset of long Minecraft videos, where it produces substantially more consistent rollouts compared to existing baselines.

1 Introduction

Long-range video consistency remains difficult because autoregressive models forget content once it leaves the context window, while expanding transformer context is computationally prohibitive. MILLIVID addresses this by combining hierarchical tokenization with coarse-to-fine rollout under a fixed sequence length.

  • Autoregressive video models generate chunks sequentially, but content becomes forgotten after its chunk exits the context window.
  • Quadratic transformer costs and hundreds of tokens per frame make full-video context infeasible, requiring models to operate with S ≪ F × H × W.
  • Coarse scene structure must persist over long horizons, whereas exact texture patterns can be forgotten with less perceptual impact.
  • MILLIVID trains a hierarchical latent space in which coarse levels capture global structure and finer levels add appearance and texture.
  • Its diffusion model generates many compressed frames at coarse scales before progressively refining toward finer scales with shared transformer weights and a fixed sequence length.
  • On long Minecraft gameplay videos, MILLIVID produces more consistent long rollouts than FramePack and typical autoregressive rollout without sacrificing per-frame quality.

2 Related Work

Prior work extends video context through retrieval, 3D memory, flexible tokenization, and multi-scale generation. MILLIVID instead learns its scale space and explicitly allocates a fixed token budget across coarse-to-fine video generation.

  • Retrieval-augmented methods append selected distant frames or chunks, whereas MILLIVID extends temporal context without retrieval and could be combined with it.
  • 3D-memory approaches preserve past information using voxel grids or point clouds with camera geometry, while MILLIVID assumes neither camera poses nor an explicit 3D world.
  • Flexible-length tokenization varies representation length with compression level and can induce a coarse-to-fine ordering for image and video generation.
  • Multi-scale generation reduces generative cost by producing coarse structure first and refining detail later across hierarchical and cascaded models.
  • Unlike FramePack, MILLIVID learns the scale space itself and explicitly allocates a fixed transformer token budget across scales.

3 Method

MILLIVID combines a hierarchical frame autoencoder with a transformer-based latent diffusion model and coarse-to-fine rollout. Under a fixed sequence-length constraint, it alternates between many compressed frames and fewer detailed frames while selecting context to preserve consistency.

  • Overview: The two-stage method first encodes frames into a hierarchical latent space and then generates those latents with a video diffusion model.Each hierarchy level uses a different number of tokens, supporting flexible detail allocation during generation.
  • Hierarchical Autoencoding: Coarser levels halve spatial resolution repeatedly and compress frames, while finer levels preserve higher-quality reconstructions.Each level can be decoded independently without tokens from other levels.
  • Hierarchical Autoencoding: The transformer encoder simultaneously produces the complete latent hierarchy from a multi-resolution image pyramid with level-aware positional encodings.The decoder reconstructs a frame from one level’s latent tokens by masking the other levels before decoding.
  • Coarse-to-Fine Video Generation: Coarse-to-fine rollout generates many coarse frames, fewer medium-detail frames, and individual fine frames within the fixed sequence length.The rollout alternates detail levels rather than generating every frame at full resolution.
  • Sequence-Length Constraint: A 30-second, 20-fps video with 16 × 16 tokens per frame requires 153,600 tokens, far exceeding feasible transformer sequence lengths.The model therefore operates under S ≪ F × H × W while targeting temporally coherent long videos.
  • Coarse-to-Fine Video Generation: Each rollout step conditions generation on the most recent highest-resolution frame and future frames available only at coarser levels, preventing inconsistent refinements.Training randomly samples rollout steps and denoises target tokens using clean context tokens while hiding fixed and pending tokens.

4 Results

The evaluation compares MILLIVID with FramePack and full-resolution autoregressive rollout on LOOPCRAFT, using separate measures for long-range consistency and visual quality. Across these evaluations, MILLIVID is reported to outperform the baselines in consistency and quality.

  • Evaluation setup: The evaluation compares MILLIVID with FramePack and full-resolution autoregressive rollout under long-context video-generation settings.All models are action-conditioned and use the same decoder; the baselines use the autoencoder’s highest-resolution latent space.
  • Dataset: LOOPCRAFT contains 200,000 Minecraft gameplay videos, each 1024 frames at 256 × 256 resolution, with action and pose metadata.Trajectories include intermittent random 90-degree turns, supporting evaluation of content that exits and later re-enters the frame.
  • Metrics: Consistency measures whether generated rollouts recall previously seen content, while quality measures frame fidelity independently of consistency.Consistency is evaluated by conditioning on up to 256 ground-truth frames and comparing a 768-frame rollout against the ground truth; LPIPS is especially robust to slight misalignments.
  • Model budgets: MILLIVID uses four latent levels with 3, 12, 48, and 192 frames of context, totaling 3840 tokens and denoising up to 192 coarse frames simultaneously.FramePack matches the per-frame context token count and uses the same 3840-token budget, while autoregressive rollout uses seven context and seven target frames for 3584 tokens.
  • Results: MILLIVID clearly outperforms the baselines in consistency and quality, while Figure 4 reports matched quality and clearly higher consistency than the baselines.Table 1 summarizes PSNR, LPIPS, SSIM, DINOv2, LightGlue, FID, and FVD over short, medium, and long horizons.

5 Analysis

The analysis tests whether learned hierarchical latents and coarse-to-fine generation explain MILLIVID’s behavior. Learned hierarchical representations preserve coarse scene structure under compression, while cascaded and several FramePack variants perform worse.

  • Hierarchical latent space: The hierarchical autoencoder preserves coarse scene structure as token budgets decrease, while discarding fine-grained texture and geometry.Its highest-resolution level reconstructs structure and texture, whereas the most compressed level retains coarse structure but forgets exact textures.
  • Hierarchical latent space: Hierarchical reconstructions are significantly better than reconstructions from mean-pooled full-resolution latents, whose outputs become blurry and lose most structure.The mean-pooled alternative corresponds to a resolution cascade of the kind used in cascaded diffusion models.
  • Cascaded comparison: The cascaded variant performs worse than the hierarchical model on both consistency and quality.This variant operates on downscaled mean-pooled versions of the highest-resolution latents rather than the full learned latent hierarchy.
  • FramePack variants: Among FramePack variants, the mirrored variant improves consistency despite its shorter context window, while all variants perform worse on quality.The hierarchical variant’s consistency becomes worse than random frames because of rollout instability.
  • FramePack variants: The mirrored FramePack model predicts coarse long-range targets during training but discards them during sampling, behaving like a model with about half the usual context length.The authors hypothesize that predicting farther into the future forces greater use of distant context.

6 Conclusion

With a fixed sequence length, MILLIVID trades long-range fine-detail consistency for a longer temporal horizon for coarse structure. The study reports substantially improved long-range consistency, while noting limitations in fine-tuning standard video diffusion models and rollout efficiency.

  • 6 Conclusion: MILLIVID fills a fixed token budget with many past coarse frames and only a few recent fine frames.This coarse-to-fine hierarchy is learned with an adaptive-length frame tokenizer.
  • 6 Conclusion: The method suggests dramatic gains in long-range consistency, recalling 3D scene structure over hundreds of frames.Conventional video generative models can afford only a handful of context frames in the comparison described.
  • 6 Conclusion: MILLIVID is not designed for straightforward fine-tuning from standard video diffusion models.The authors suggest future work on hierarchical autoencoders and associated latent diffusion models for this purpose.
  • 6 Conclusion: Compared with FramePack, MILLIVID requires about 33% more rollout steps.The authors characterize this as a tradeoff for increased consistency.

A.1 Test Set Generation

The test set contains long Minecraft videos selected for high overlap between the initial context and later frames. The authors use expected top-down trajectory overlap rather than camera-frustum overlap, which behaves counterintuitively for looping trajectories.

  • A.1 Test Set Generation: The test set contains 1,000 videos with high overlap between the first 256 context frames and the remaining 768 frames.Each test trajectory is selected from sampled complete action sequences using expected top-down overlap.
  • A.1 Test Set Generation: The test-set generation procedure produces interesting videos with good context overlap.The authors describe this as a reasonable outcome of their selection procedure.
  • A.1 Test Set Generation: Camera-frustum overlap was rejected because looping trajectories make the context frustums cover almost every angle.Under those conditions, frustum overlap becomes a meaningless metric.
  • A.1 Test Set Generation: Perfect context overlap is not required because low-overlap continuations mainly measure generation quality rather than consistency.The authors note that even a near-perfect model can only produce plausible hallucinations when visual overlap is minimal.

A.2 Implementation and Training Details

The appendix specifies implementation, training, inference, sequence-budget, and evaluation settings for MILLIVID and its baselines. Models are compared with closely matched sequence lengths, while MILLIVID and FramePack access substantially more past frames than full-resolution autoregressive rollout.

  • Hierarchical Autoencoder Training Setup: The hierarchical autoencoder uses a ViT-B encoder and decoder with four hierarchy levels, 256×256 maximum resolution, 16×16 patches, and 32-channel output tokens.Level embeddings and AdaLN condition the decoder across the coarse-to-fine hierarchy.
  • Generative Model Training Setup: The generative model is a DiT-B diffusion model trained jointly on latents from all four levels with 3D position embeddings and learned level embeddings.Categorical actions are conditioned through a learned embedding layer.
  • Inference Details: Evaluation generates frames 257–1024 after starting at frame 256, while comparison models use closely matched but not identical sequence lengths.Inference uses deterministic DDIM sampling with 50 denoising steps, and evaluation samples omit CFG.
  • Model-Specific Details: MILLIVID uses S = 3840, allocating 3072 tokens to context and 768 tokens to prediction, with up to 255 past frames at inference.The context budget covers [3, 12, 48, 192] frames at hierarchy levels 0–3.
  • Model-Specific Details: FramePack uses the same S = 3840 and 255-frame maximum past context, predicting three future frames at the finest level.Its 3072-token context budget covers the same level-specific frame allocation as MILLIVID.
  • Model-Specific Details: Full-resolution autoregressive rollout uses S = 3584 and sees at most seven past frames at inference.Half of its budget is allocated to context and half to predicted frames.
  • Training Details: Main models are trained for 192,000 steps on 8 H200 GPUs, while smaller ablation and comparison models use one-third of the sequence-length budget.The smaller models are trained for 256,000 steps and require approximately one day per model.

A.3 Sampling Speed

All methods require O(F) rollout steps for F generated frames, but their sampling times differ. MILLIVID matches FramePack at approximately 30 minutes for eight 768-frame videos while producing better consistency and per-frame quality in the reported metrics.

  • A.3 Sampling Speed: 11 minutes are required to sample eight 768-frame videos with full-resolution autoregressive rollout.This method generates only at the highest resolution and requires O(F) rollout steps.
  • A.3 Sampling Speed: 30 minutes are required to sample eight 768-frame videos with FramePack.FramePack also requires O(F) rollout steps because it predicts a small number of frames at the finest resolution.
  • A.3 Sampling Speed: 30 minutes are required to sample eight 768-frame videos with MILLIVID.Its coarse-level rollout steps decrease exponentially by hierarchy level, yielding O(F) total steps; it uses about 33% more sampling steps than equivalent FramePack.
  • A.3 Sampling Speed: MILLIVID reports better consistency across PSNR, LPIPS, SSIM, DINOv2 cosine similarity, and LightGlue keypoints, alongside better FID and FVD.Figure 8 also reports less exposure bias.
  • A.3 Sampling Speed: At 3840 tokens, MILLIVID and FramePack each hold 255 frames of context, whereas the high-resolution autoregressive model holds seven.At 1280 tokens, the corresponding context lengths are 85, 85, and 2 frames.
  • A.3 Sampling Speed: MILLIVID’s rollout procedure interleaves context and target tokens across hierarchy levels while excluding fixed and pending tokens.The figure illustrates this procedure for 21 rollout steps between an initial 22-frame state and a final 38-frame state.
  • A.3 Sampling Speed: Other methods differ in how they combine rollout, upscaling, patchification, and hierarchical latents.FramePack always predicts frames at full resolution, while its mirrored hierarchical variant uses hierarchical latents.
Loading 2606.09056v1…