Source-linked AI summary
VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
Zhihao Xie, Junfeng Wu, Xinting Hu, Junchao Huang, Li Jiang
TL;DR
Video generative models lack compact latents that preserve rich semantic and spatio-temporal structure, and it remains unclear whether frozen video foundation-model features can fill this role. VideoRAE compresses hierarchical frozen VFM features into continuous or discrete latents, achieving state-of-the-art generation and faster convergence across regimes.
Problem
It remains unclear whether frozen video foundation-model features can become compact, reconstruction-capable, and generation-compatible latents for video generation.
Method
VideoRAE compresses hierarchical features from a frozen video foundation encoder with a lightweight projector and aligns decoder representations to support continuous and discrete latents.
Results
gFVDs of 40 and 93 were achieved by VideoRAE’s AR and DiT variants on UCF-101, while both converged approximately 5× faster than autoencoder baselines.
Takeaways & Limitations
Frozen VFM representations can serve as versatile, reconstruction-capable, and generation-friendly video latents across continuous and discrete generative paradigms.
Abstract
from arXiv · showhide
Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.
1 INTRODUCTION
VideoRAE addresses the limitations of pixel-trained video autoencoders by converting frozen Video Foundation Model representations into compact, semantically rich latents for video generation. It supports continuous and discrete regimes and achieves strong reconstruction and faster generative convergence across evaluation settings.
- Motivation: 3D-VAEs and discrete tokenizers often fail to capture high-level semantics or physical structures, leaving generative models to learn complex spatio-temporal dynamics and global semantics.This increases the difficulty and data requirements of downstream training.
- Method: VideoRAE uses a frozen VFM encoder, aggregates hierarchical features, and compresses them with a lightweight 1D self-attention projector.The design captures coarse semantics and fine-grained spatio-temporal dynamics while meeting video-generation compression requirements.
- Method: A local-and-global representation alignment objective improves decoding from frozen VFM features and supports the unified autoencoding framework.The objective is tailored to preserving information in the teacher representations.
- Results: VideoRAE achieves state-of-the-art discrete reconstruction and highly competitive continuous reconstruction on UCF-101 and TokenBench.The AR variant also outperforms prior discrete tokenizers while using fewer tokens and a smaller generator.
- Results: 5× faster convergence is achieved by both AR and DiT generators than their respective autoencoder baselines.A controlled 2B-scale text-to-video replacement study also finds faster convergence than LTX-VAE under comparable settings.
2 RELATED WORKS
Related work spans video autoencoders and tokenizers for compact latent spaces, visual foundation models for transferable representations, and the historical separation between understanding- and generation-oriented representations.
- Video VAEs and Tokenizers: Video VAEs and tokenizers compress high-dimensional video into compact latent spaces for efficient generative modeling.Continuous-space methods include VFRTok, VidTwin, and LeanVAE, which target robustness, reconstruction quality, and training efficiency, respectively.
- Visual Foundation Models: Visual Foundation Models learn general, transferable representations from large-scale, diverse data, with video models extending this paradigm beyond images.Image VFMs acquire semantically grounded representations through self-supervised and language-supervised pre-training on image-text pairs.
- Understanding and Generation Representations: Visual-understanding representations and generation-oriented representations have traditionally followed distinct trajectories.Understanding-focused representations emphasize high-level semantics while discarding low-level noise.
3 METHOD: VIDEORAE
VideoRAE replaces pixel-driven video tokenization with semantic compression from frozen VFM features, using hierarchical fusion and a lightweight 1D self-attention projector. It supports continuous and discrete latents, then decodes them with semantic alignment that preserves VFM structure while removing the need for KL regularization.
- Hierarchical VFM Encoding: VideoRAE extracts shallow-to-deep hierarchical features from a frozen VFM and fuses them by element-wise summation to capture temporal dynamics and abstract semantics.The fused representation contains flattened 3D spatio-temporal tokens with embedding dimension Dtoken.
- Hierarchical VFM Encoding: A lightweight 1D self-attention projector models global dependencies and compresses redundant spatio-temporal features into a shorter latent sequence.The compressed sequence length satisfies Nlatent ≪ Nvfm.
- Latent Formatting: VideoRAE formats base tokens as continuous latents for Diffusion Transformers or high-dimensional discrete tokens using Multi-Codebook SimVQ for autoregressive models.Multi-Codebook SimVQ partitions Zbase into K channel-wise sub-vectors, quantizes each with a sub-codebook, and concatenates the results into Zdisc.
- Decoding and Alignment: The Transformer decoder reconstructs pixels from compressed 1D tokens by combining them with 1024 learnable 3D tokens and mapping processed tokens into 3D feature maps.The decoder separates the learnable 3D tokens after stacked self-attention and applies depatchification for pixel reconstruction.
- Decoding and Alignment: Representation Alignment aligns decoder intermediate features with final VFM features at local and global scales, preserving semantic geometry and eliminating KL-divergence regularization.The alignment objective constrains latent representations to the foundation model’s semantic manifold.
- Training Objective: End-to-end training combines L1 and LPIPS reconstruction, GAN-based high-frequency texture recovery, REPA semantic alignment, and discrete-only codebook commitment losses.Lvq is included exclusively when training the discrete VideoRAE.
4 EXPERIMENTS
Experiments show that VideoRAE reconstructs videos strongly in discrete and continuous latent spaces and enables high-quality generation with faster convergence. Ablations indicate that semantic representation quality, representation alignment, and multi-scale features are central to these gains.
- Reconstruction Quality Comparison: VideoRAE achieves state-of-the-art discrete reconstruction, with rFVD 13 on UCF-101 and 28 on TokenBench.Multi-Codebook SimVQ reconstructs high-fidelity pixels within a highly compressed discrete semantic space.
- Discrete-space class-conditional generation: 40 is VideoRAE(V-JEPA 2)’s state-of-the-art gFVD on UCF-101 class-conditional generation, versus SweetTok’s gFVD 65.VideoRAE uses fewer tokens and a smaller generator than SweetTok’s 1.9B-parameter, 1280-token configuration.
- Discussion: Reconstruction Fidelity vs. Generative Quality: VideoRAE’s stronger semantic understanding and visual perception benefit generation more than rigid low-level pixel fidelity.VideoMAEv2 provides superior pixel-level reconstruction fidelity, but the experiments emphasize semantic representation quality for generation.
- Training convergence: ∼5.0× acceleration is demonstrated by VideoRAE reaching gFVD 139 at 400 epochs, while also accelerating convergence against LTX-VAE.The representation reduces the generator’s burden of learning complex spatio-temporal dynamics from scratch.
- Representation Alignment: REPA reduces continuous-setting gFVD from 105 to 93 and discrete-setting gFVD from 67 to 40.The Representation Alignment objective improves performance across most evaluation metrics in both latent spaces.
5 CONCLUSION
VideoRAE is introduced as a semantic-driven video autoencoder bridging visual understanding and generation. It demonstrates that frozen VFM representations can provide robust generative-modeling latents while supporting continuous and discrete paradigms.
- VideoRAE is a semantic-driven video autoencoder designed to bridge visual understanding and generation.
- Frozen VFM representations can serve as robust latent spaces for generative modeling despite their presumed unsuitability for pixel-level reconstruction.
- VideoRAE provides a unified framework supporting both continuous and discrete generative paradigms.