PaperlayerView exact revision on arXiv

Source-linked AI summary

V-RAE: Rethinking Video Latent Spaces for Generation

Minghui Guo, Shengqiong Wu, Hao Fei

arXiv:2608.13556v1cs.CV

TL;DR

Video autoencoder latents are usually optimized for pixel reconstruction, which may not make them suitable for generation. V-RAE builds latents from frozen semantic vision representations with temporal compression and decoding, improving reconstruction, semantic preservation, and generation quality while converging up to 6× faster.

  • Problem

    Video autoencoders mainly optimize pixel reconstruction, while reconstruction quality alone may not indicate whether their latents are suitable for downstream generation.

  • Method

    V-RAE combines a frozen visual representation encoder with learned temporal pooling and a spatiotemporal Transformer decoder to form compact semantic video latents.

  • Results

    V-RAE improves reconstruction, semantic preservation, and generation across evaluated settings, achieving 19.16 gFVD on K600 while converging up to 6× faster than evaluated VAE-based latent spaces.

  • Takeaways & Limitations

    The study shows that frozen semantic representations can support video reconstruction and generation, while tFVD better characterizes generative utility than reconstruction quality alone.

  • Takeaways & Limitations

    The experiments do not establish whether V-RAE’s advantages extend to large-scale open-domain text-to-video generation with larger datasets, resolutions, video lengths, and DiT backbones.

Abstract

from arXiv · show

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.

1. Introduction

V-RAE addresses the limitations of reconstruction-oriented video latent spaces by building generative latents from frozen visual representations with learnable temporal compression and decoding. Across four encoders, it preserves semantic information, improves reconstruction and generation, and introduces tFVD as a diagnostic beyond reconstruction fidelity.

  • Motivation: Video representation encoders create temporally dense features, making compression and temporal-coherence preservation central challenges for affordable, stable generation.Insufficient temporal compression raises DiT training and inference costs, while poor coherence causes flicker and jitter.
  • Method: V-RAE uses a frozen visual representation encoder, lightweight temporal pooling, and a spatiotemporal Transformer decoder to compress features while preserving semantic structure.The decoder is equipped with 3D RoPE.
  • Results: 2.13 rFVD on K600 is achieved by V-RAE with V-JEPA 2.1, surpassing all evaluated large-scale pretrained video VAEs.V-RAE is evaluated with DINOv3, SigLIP2, EUPE, and V-JEPA 2.1 across reconstruction fidelity, semantic preservation, and generative quality.
  • Results: 89.13% top-1 accuracy on UCF101 is reached by V-RAE with DINOv3-L, compared with 30.83% for the strongest VAE baseline.The result demonstrates stronger semantic information in the learned latent space.
  • Results: 117.86 gFVD on UCF101 and 19.16 gFVD on K600 are achieved by the best-performing variant, while convergence is up to 6× faster under matched training settings.The work also introduces tFVD and shows that reconstruction fidelity alone does not fully characterize generative utility.

2. Methodology

V-RAE combines a frozen visual representation encoder with learnable temporal pooling and a Transformer decoder to construct compact video latents and reconstruct videos. After reconstruction training, its parameters are frozen while a DiT learns generation in the resulting latent space.

  • Architecture: V-RAE freezes the visual encoder and optimizes only temporal pooling and the Transformer video decoder.The decoder reconstructs pixels from the pooled latent sequence.
  • Visual Representation Encoder: The framework supports image-pretrained encoders through frame-wise encoding and video-native encoders through joint spatiotemporal encoding.Examples include DINOv3, SigLIP2, EUPE, and V-JEPA 2.1-L.
  • Temporal Pooling: Learnable temporal pooling compresses adjacent, temporally redundant features while preserving semantic structure and operates independently at each spatial position.Initialization makes attention weights uniform, so pooling begins as temporal mean pooling before learning content-adaptive weighting.
  • V-RAE Decoder: The decoder reconstructs videos with lightweight ViT blocks and extended RoPE that supplies explicit temporal and spatial positional information.It maps the compressed representation Z back to the reconstructed video X̂.
  • Latent Generation: After reconstruction training, a DiT is trained in the frozen V-RAE latent space using rectified flow with dimension-dependent noise shifting and clean-latent prediction.Inference uses a 100-step Euler sampler and the frozen decoder to convert generated latents into video frames.

3. Experimental Settings

The experiments benchmark V-RAE on UCF101 and K600 for reconstruction and class-conditional generation under standardized clip, resolution, latent-sequence, and DiT settings. Comparisons include large-scale pretrained video autoencoders and tokenizers, with reconstruction, generation quality, and convergence speed evaluated.

  • Datasets: UCF101 and K600 serve as the main reconstruction and class-conditional generation benchmarks, with reconstruction training using their combined training sets.Reconstruction is evaluated on the UCF101 test set and K600 validation set.
  • Implementation: Reconstruction uses LPIPS, PSNR, SSIM, and rFVD on 256 × 256 clips, with V-RAE and AToken using 16 frames and causal VAEs using 17.Evaluation clips come from the beginnings of UCF101 test and K600 validation videos, unlike LARP’s training-split evaluation.
  • Implementation: Class-conditional generation standardizes the DiT backbone, spatial resolution, and sequence length at 1,280 tokens while adjusting patchification for each autoencoder’s compression ratio.Causal VAEs generate 17 frames directly; V-RAE and AToken generate 20 and retain the first 17 for evaluation, using gFVD at 256 × 256.
  • Baselines: V-RAE is compared with Wan2.1 VAE, Wan2.2 VAE, HunyuanVideo VAE, CogVideoX VAE, Cosmos VAE, AToken, MAGVIT-v2, OmniTokenizer, and LARP.Under the matched DiT setup, comparisons cover generation quality and convergence speed.

4. Results and Discussion

V-RAE’s frozen semantic latents achieve competitive reconstruction while retaining substantially richer semantics and enabling better, faster video generation. The results further show that reconstruction fidelity alone is insufficient: temporally smooth, decodable latent spaces better predict and support downstream generation and future prediction.

  • Semantic Information: 90.92% on UCF101, 72.91% on SSv2, and 83.12% on K400 show that V-RAE latents retain more semantics than the strongest VAE baselines.Temporal compression keeps image-encoder variants within 3.85 percentage points of frozen encoders, while V-JEPA 2.1 remains within 3.67 points on SSv2 and K400.
  • Generation Quality and Efficiency: 117.86 gFVD on UCF101 and 19.16 on K600 make V-JEPA 2.1 the best V-RAE variant, improving 25.14 and 22.50 points over non-V-RAE baselines.Under matched settings, V-JEPA 2.1 and EUPE-B maintain lower gFVD than Wan2.2 VAE at every evaluated UCF101 checkpoint; V-JEPA 2.1 reaches comparable gFVD in roughly 30K versus 150K updates.
  • Reconstruction versus Generation: 211.53 versus 125.98 gFVD on UCF101 and 53.75 versus 34.48 on K600 show that lower rFVD does not guarantee easier generative modeling.HunyuanVideo VAE has lower UCF101 rFVD than V-RAE EUPE-B, while Wan2.1 VAE and V-RAE SigLIP2-L have similar K600 rFVD despite substantially different gFVD.
  • Temporal Fréchet Video Distance: tFVD correlates with gFVD at r=0.621 on UCF101 and r=0.919 on K600, exceeding rFVD correlations of r=0.200 and r=0.473.tFVD probes temporal smoothness and decoder robustness under deviations between encoded states, where V-RAE shows less ghosting and structural instability than the VAE.
  • Future Video Prediction: V-RAE reduces future-prediction gFID from 15.02 to 11.52 and gFVD from 144.47 to 111.36, despite worse rFVD of 29.2931 versus 7.0256.Its tFVD is lower at 224.6040 versus 319.0233, supporting temporal smoothness and decoder robustness as explanations for superior future prediction.

5. Related Work

Prior work incorporates pretrained visual representations into generative models through latent alignment, auxiliary objectives, or frozen encoder features. Video autoencoders provide generator latent spaces through continuous, discrete, and query-based designs.

  • Representation Learning for Generation: Generative models use pretrained representations by aligning tokenizer latents, aligning generator features through auxiliary objectives, or directly adopting frozen encoder features.In video, DERA aligns tokenizer representations, while VideoREPA and Wu et al. align diffusion features; Divot models continuous video representations.
  • Video Autoencoders: Video autoencoders supply latent spaces for modern video generators through continuous video VAEs, discrete tokenizers, and query-based tokenizers.The cited representative designs include Wan et al., Kong et al., Yang et al., Wang et al., Yu et al., Kondratyuk et al., Agarwal et al., Tan et al., and Xiong et al.

6. Conclusion · A. Implementation Details

V-RAE repurposes frozen visual representation models as semantically structured latent spaces for video reconstruction and generation. The paper also documents architecture, reconstruction, and generation configurations for reproducibility.

  • 6. Conclusion: V-RAE uses frozen visual representation models to provide semantically structured latent spaces for video reconstruction and generation.
  • 6. Conclusion: V-RAE achieves near-best reconstruction on UCF101 and outperforms all evaluated video autoencoders on K600.
  • 6. Conclusion: V-RAE retains substantially richer semantic information than conventional VAE-based video tokenizers.
  • 6. Conclusion: Using V-RAE as a DiT latent space further improves generation quality.
  • A. Implementation Details: Section A.1 details the V-RAE architecture and temporal pooling designs.
  • A. Implementation Details: Section A.2 specifies the reconstruction recipe and shared VideoMAE discriminator.
  • A. Implementation Details: Section A.3 reports generation architecture, training, sampling, and evaluation settings.

A.1. V-RAE Architecture

V-RAE freezes a vision representation encoder, compresses its temporally redundant features with lightweight pooling, and reconstructs video using a shared ViT-XL decoder. Its architecture compares pooling designs that trade semantic preservation against adaptive aggregation and feature transformation.

  • Representation encoders: V-RAE instantiates four variants with frozen DINOv3-L, SigLIP2-L, EUPE-B, or V-JEPA 2.1-L representation encoders.The first three use 1024-dimensional features, EUPE-B uses 768-dimensional features, and V-JEPA 2.1-L halves temporal length with tubelets of two.
  • Latent normalization: Fixed, non-affine LayerNorm prevents latent rescaling from bypassing the intended noise level during reconstruction training.Setting γ=1 and β=0 keeps latent scale calibrated and forces the decoder to reconstruct under the injected noise level.
  • V-RAE decoder: The shared decoder is a ViT-XL with width 1152, 28 transformer blocks, 16 attention heads, and a 4096-dimensional GELU feed-forward network.It combines learned 2D spatial embeddings with 3D RoPE (θ=10,000); most variants use chunk-causal attention, while V-JEPA 2.1-L uses non-causal full self-attention.
  • Temporal pooling designs: The pooler compresses non-overlapping temporal groups while preserving the frozen encoder’s semantic geometry.The designs vary in content adaptivity and the strength of feature-space transformation, under fixed encoder, decoder, data, compression ratio, and reconstruction objective.
  • Temporal pooling designs: Temporal attention provides content-adaptive frame weighting while retaining shared spatial aggregation and a lightweight transformation of encoder features.Identity-initialized projections and zero-initialized queries and temporal biases make its initial behavior close to mean pooling.

A.2. Reconstruction Training Recipe

Reconstruction training jointly optimizes a temporal pooler and video decoder while keeping the vision encoder frozen. It uses augmented 16-frame clips, noisy latent reconstruction with multiple losses, and a shared VideoMAE-based adversarial objective introduced late in training.

  • Reconstruction training: The temporal pooler and video decoder are jointly trained while the representation encoder remains frozen in evaluation mode.
  • Reconstruction objective: 0.8 is the upper bound of uniform latent-noise perturbation, with z_train = z_clean + σϵ, σ∼U(0, 0.8), and ϵ∼N(0, I).The decoder uses L1, LPIPS, GAN, and Gram losses; LPIPS operates at 256 × 256, while Gram uses 224 × 224, and both cover all 16 frames.
  • Adversarial objective: Epoch 30 marks the start of a shared VideoMAE-based adversarial objective across all four variants.The VideoMAE-B backbone is frozen; temporal Conv1d heads are optimized, generator and discriminator losses are weighted by 0.3, and the discriminator updates once per five generator steps.

A.3. Generation Training Recipe

The generation comparison uses matched, high-capacity DiT backbones and latent-token budgets so quality and convergence primarily reflect latent-modeling differences. Training freezes each V-RAE, uses dataset-specific latent statistics and schedules, and applies shared flow-matching and sampling procedures.

  • DiT architecture: Matched DiT backbones and latent-token budgets isolate how readily each autoencoder’s latent distribution can be modeled.The high-capacity DiT reduces the risk that generator expressiveness becomes the dominant bottleneck.
  • DiT architecture: UCF101 and K600 use a shared 28-block, 1536-dimensional class-conditional DiT, while Cityscapes retains 28 blocks with a 1152-dimensional encoder.EUPE-B changes input channels and time shift for UCF101 and K600; Cityscapes uses 16 attention heads of dimension 72 and 768-channel EUPE-B latents.
  • Conditioning: UCF101 and K600 use class embeddings with classifier-free dropout 0.1, whereas Cityscapes conditions on clean-context latents with random context dropping at probability 0.1.Cityscapes maps mean-pooled context through a two-layer MLP and adds an auxiliary prediction branch after encoder block 8.
  • Training details: The V-RAE is frozen, with latent statistics computed separately per dataset and encoder; UCF101 trains 1,700 epochs and K600 100 epochs.Both use linear learning-rate decay from 10−4 to 5 × 10−5, with batch sizes 64 and 128 and warmups of 10 and 2 epochs, respectively.
  • Sampling and evaluation: Sampling uses EMA weights, a 100-step Euler solver, CFG 1.0, and internal guidance active over t∈[0.10, 1.0].The flow-matching objective combines main and auxiliary losses with unit weight for each term.

B. Evaluation Protocols

The evaluation uses standardized Fréchet-metric implementations and dataset-specific protocols for reconstruction, temporal interpolation, class-conditional generation, and future prediction. These protocols align clip formats, sampling procedures, feature extractors, and evaluation populations across tokenizer comparisons.

  • General protocol: Evaluation uses EMA checkpoints, I3D Kinetics-400 clip features for rFVD, tFVD, and gFVD, Inception-v3 pool3 features for gFID, and float64 Fréchet statistics.These conventions apply unless stated otherwise.
  • Reconstruction: Reconstruction uses UCF101 test and K600 validation clips resized and center-cropped to 256 × 256, with 16 frames for V-RAE and AToken and 17 for causal VAEs.Clips begin at the video start with temporal interval 3; K600 contributes 27,874 clips without temporal padding.
  • Temporal interpolation: tFVD replaces four interior codes in six temporally ordered latent codes with local midpoints, decodes them from temporal position zero, and compares the resulting 16 frames against aligned ground truth.The comparison uses ground-truth frames 4–19 and the same I3D implementation as rFVD.
  • Class-conditional generation: Class-conditional generation evaluates 17-frame, 256 × 256 videos using shared populations and 100-step Euler sampling, with 2,048 UCF101 clips and 50,000 K600 clips.V-RAE and AToken generate 20 frames and retain the first 17, while causal VAEs generate 17 directly; V-RAE uses EMA DiT weights and CFG 1.
  • Future prediction: Future prediction conditions on Cityscapes frames 4–15 and predicts frames 16–27 at 432 × 768 and 16 fps across all 500 validation sequences.The 12-frame predictions support gFID over 6,000 frames and gFVD over 500 clips using EMA models, 100 Euler steps, CFG 1.0, and internal guidance 1.2.

C. Additional Qualitative Results

Additional qualitative results present class-conditional generation samples from V-RAE’s DINOv3-L and SigLIP2-L variants, completing coverage of all four evaluated representation encoders.

  • C. Additional Qualitative Results: Together with Figures 6 and 7, the reported samples cover all four V-RAE representation encoders evaluated in the generation experiments.Figure 13 supplies the DINOv3-L and SigLIP2-L variants, while Figures 6 and 7 provide the remaining coverage.
  • C. Additional Qualitative Results: Figure 13 shows class-conditional generation results for V-RAE using the DINOv3-L and SigLIP2-L representation encoders.These samples provide additional qualitative evidence for the two variants.

D. Limitations

The study’s experiments use relatively controlled video benchmarks and moderate-scale generative models, leaving the broader applicability of V-RAE unresolved. Future work must test whether its advantages extend to open-domain text-to-video generation across larger datasets, higher resolutions, longer videos, and larger DiT backbones.

  • D. Limitations: The experiments primarily target relatively controlled video benchmarks and moderate-scale generative models.These settings isolate the effect of the latent representation.
  • D. Limitations: It remains unclear whether V-RAE’s advantages extend to large-scale open-domain text-to-video generation.
  • D. Limitations: Future evaluations should cover larger datasets, higher resolutions, longer videos, and substantially larger DiT backbones.
Loading 2608.13556v1…