PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive 1D Video Diffusion Autoencoder

Yao Teng, Minxuan Lin, Xian Liu, Shuai Wang, Xiao Yang, Xihui Liu

arXiv:2602.04220v1cs.CV

TL;DR

Existing video autoencoders use fixed-rate compression, inflexible CNN architectures, and deterministic decoders that limit adaptive latent modeling and detail recovery. One-DVA addresses these issues with query-based transformer encoding, variable-length latent dropout, and a pixel-space diffusion decoder. It achieves reconstruction performance comparable to 3D-CNN VAEs at standard compression and supports downstream latent diffusion, with latent alignment and decoder fine-tuning addressing generation artifacts.

  • Problem

    Existing video autoencoders use fixed-rate compression, inflexible CNN architectures, and deterministic decoders, motivating more adaptive and generative designs.

  • Method

    One-DVA combines query-based ViT encoding, variable-length dropout, and a pixel-space Diffusion Transformer decoder for adaptive 1D video compression and reconstruction.

  • Results

    One-DVA achieves reconstruction performance comparable to advanced 3D-CNN VAEs and supports downstream latent diffusion models for video generation.

  • Takeaways & Limitations

    One-DVA’s adaptive 1D tokenization and diffusion decoding provide a unified autoencoder design for dynamic compression and generative reconstruction.

  • Takeaways & Limitations

    One-DVA’s experiments do not fully realize streaming generation, and the theoretically optimal token length for each video complexity remains open.

Abstract

from arXiv · show

Recent video generation models largely rely on video autoencoders that compress pixel-space videos into latent representations. However, existing video autoencoders suffer from three major limitations: (1) fixed-rate compression that wastes tokens on simple videos, (2) inflexible CNN architectures that prevent variable-length latent modeling, and (3) deterministic decoders that struggle to recover appropriate details from compressed latents. To address these issues, we propose One-Dimensional Diffusion Video Autoencoder (One-DVA), a transformer-based framework for adaptive 1D encoding and diffusion-based decoding. The encoder employs query-based vision transformers to extract spatiotemporal features and produce latent representations, while a variable-length dropout mechanism dynamically adjusts the latent length. The decoder is a pixel-space diffusion transformer that reconstructs videos with the latents as input conditions. With a two-stage training strategy, One-DVA achieves performance comparable to 3D-CNN VAEs on reconstruction metrics at identical compression ratios. More importantly, it supports adaptive compression and thus can achieve higher compression ratios. To better support downstream latent generation, we further regularize the One-DVA latent distribution for generative modeling and fine-tune its decoder to mitigate artifacts caused by the generation process.

1. Introduction

One-DVA addresses fixed-rate compression, inflexible CNN architectures, and deterministic decoding with a transformer framework for adaptive 1D video compression and diffusion-based reconstruction.

  • Motivation: Existing video autoencoders use fixed token counts, although simple videos can require fewer tokens than videos with complex textures and motion.This motivates adaptive latent sizes tailored to video content.
  • Approach: One-DVA uses a ViT with 1D queries to extract spatiotemporal features and produce structural and 1D latent representations.The query mechanism forms the basis for variable-length encoding.
  • Approach: A variable-length dropout mechanism dynamically adjusts the 1D latent sequence length to match video complexity.The mechanism is integrated into the unified adaptive compression design.
  • Approach: The decoder is a pixel-space Diffusion Transformer that reconstructs videos using the latents as conditional inputs.This replaces deterministic decoding with diffusion-based generative reconstruction.
  • Training and downstream generation: A two-stage training strategy combines encoder optimization with variable-length compression and diffusion decoding.The first stage prioritizes encoder optimization, while the second integrates stochastic compression and diffusion-based decoding.
  • Training and downstream generation: One-DVA matches 3D-CNN VAEs in reconstruction performance at standard compression and is further adapted for downstream latent diffusion modeling.Latent alignment regularizes the latent space, and decoder fine-tuning uses generated latents to address generation artifacts.

2. Background and Related Work

The related work organizes autoencoders by latent representation, latent shape, architecture, and decoder paradigm, then positions One-DVA among video autoencoder alternatives.

  • Image Autoencoders: Image autoencoders are classified by continuous or discrete latents, 2D or 1D shape, CNN or transformer architecture, and deterministic or generative decoding.These attributes structure the paper’s comparison of prior autoencoder designs.
  • Image Autoencoders: Query-based transformer image autoencoders use learnable 1D queries and attention to extract continuous latents or discrete tokens from image features.Their decoders use learnable vectors and transformers to retrieve image information from latent features.
  • Video Autoencoders: Video autoencoders include 3D CNN-based continuous and discrete frameworks supporting diffusion-based or autoregressive video generation.The literature also includes 1D autoencoders and 3D transformer-based autoencoders.
  • Positioning One-DVA: One-DVA integrates 1D variable-length encoding, query-based transformer extraction, and a diffusion decoder within one video autoencoder.These features target dynamic compression, flexible video information extraction, and improved reconstruction quality.
  • Positioning One-DVA: The design achieves dynamic compression ratios through 1D variable-length encoding while using a diffusion decoder to improve reconstruction quality.The query-based transformer provides the flexible extraction mechanism underlying the variable-length representation.

3. Method

One-DVA encodes videos into hybrid structural and variable-length 1D latents, then reconstructs them through conditional pixel-space diffusion with multistage training and latent adaptation for generation.

  • Query-based Vision Transformer Encoder: The encoder combines ViT-derived structural latents with a query-extracted 1D latent sequence to represent each input video.Video patches are projected into spatiotemporal embeddings, flattened, and processed with learnable 1D queries in transformer blocks.
  • Variable-Length Compression: The variable-length dropout module shortens 1D latents from tail to head using motion-dependent random dropout during training.Dropped tokens are replaced with padding tokens in the decoder, while some conditions retain full or partial latents.
  • Diffusion Decoding: Lossy compression motivates treating decoding as conditional generation, with variable-length latents as conditions and noisy videos as diffusion inputs.The decoder progressively refines noise into a clean video during inference.
  • Diffusion Decoding: During diffusion training, noisy inputs interpolate ground-truth video x_0 with Gaussian noise x_1 at timestep t.The interpolation is defined as x_t = (1 − t) · x_0 + t · x_1.
  • Diffusion Decoding: The decoder predicts velocity for diffusion updates while conditioning on the latent representation.The update uses consecutive timesteps and the decoder output D_θ.
  • Decoder Architecture: The pixel diffusion transformer concatenates transformed latent and noisy-input features before reconstructing pixels through an unpatchifier.The unpatchifier uses a long skip connection, linear projection, pixelshuffle, and convolution.
  • Autoencoder Training: Training minimizes a weighted sum of diffusion, perceptual, KL, and REPA losses.The KL term regularizes latents toward a standard Gaussian distribution, while REPA operates on noisy decoder-input features.
  • Autoencoder Training: A two-stage recipe first pretrains the encoder with pure decoder noise and no latent dropout, then enables diffusion timestep sampling and variable-length compression.The first stage establishes an upper bound on reconstruction ability before stochastic post-training.

4. Experiments

Experiments show that One-DVA provides strong reconstruction at standard compression, adapts latent length to video complexity, and supports downstream video generation. Diffusion scheduling and stage-wise training further affect reconstruction quality, especially under weak or variable conditioning.

  • Comparison to State-of-the-art Methods: At standard compression, One-DVA achieves the best overall PSNR and SSIM, with the second-lowest reconstruction FVD among compared autoencoders.The comparison uses 17×256×256 videos and truncates queries to obtain a standard compression ratio.
  • Comparison to State-of-the-art Methods: Adaptive per-video latent-length estimation outperforms a constant-length baseline with identical token usage.The result supports the scoring strategy used to select 1D latent lengths for reconstruction.
  • Analysis on Reconstruction: Videos with greater motion require longer 1D latents to maintain reconstruction quality, showing a steeper PSNR decline as latent length decreases.Achieving 90% PSNR requires a longer latent for videos with more motion.
  • Analysis on Reconstruction: Longer 1D latents improve fine-detail reconstruction, while motion regions become blurry when reconstructed without 1D latents.Scene text is an example of detail that benefits from longer latents.
  • Study on Training Strategy: The two-stage training pipeline outperforms end-to-end training in reconstruction after sufficient training.Deterministic pretraining encourages the encoder to capture reconstruction-relevant features before diffusion training.
  • Effectiveness of Diffusion Scheduling: More diffusion sampling steps improve rFVD when conditioning is weak, but these gains can come at the expense of PSNR.Sampling-step changes have less impact with full 1D latents; the authors hypothesize that diffusion prioritizes dataset distribution over per-sample fidelity.
  • Analysis on Generation: The full One-DVA framework achieves a gFVD of 210.9 for class-to-video generation, matching methods such as Hi-VAE.Decoder fine-tuning contributes to this result, and its error-correction capability transfers from text-to-video to class-to-video generation.

5. Conclusion

One-DVA unifies adaptive 1D video tokenization with diffusion-based generative decoding. Experiments show reconstruction comparable to advanced 3D CNN VAEs and support for downstream latent diffusion video generation.

  • 5. Conclusion: One-DVA combines query-based encoding, variable-length dropout, and pixel-space diffusion decoding for adaptive video compression and reconstruction.The framework uses latents as conditions for the diffusion decoder.
  • 5. Conclusion: Experiments validate reconstruction comparable to advanced 3D CNN VAEs and compatibility with downstream latent diffusion models for video generation.The conclusion states both reconstruction and downstream generation outcomes.

A. Autoencoder Details

The autoencoder uses transformer architectures with patch-based processing and estimates variable latent lengths from video motion. Training uses AdamW and a two-stage learning-rate schedule with specified loss-weight balancing.

  • Architecture Details: The encoder uses temporal patch size 4 for efficiency, while the decoder uses temporal patch size 2; both use spatial patch size 8.Inputs are padded when dimensions are not divisible by the patch sizes.
  • Heuristic Motion-aware Token Length Estimation: Variable latent length is determined from a motion score computed by averaging grayscale pixel differences between consecutive frames.The motion estimator aggregates absolute differences across spatiotemporal dimensions.
  • Heuristic Motion-aware Token Length Estimation: The maximum token count is denoted Nmax.
  • Training Details: Training uses AdamW with β1 = 0.9, β2 = 0.999, and weight decay 10^-4.The stage-1 loss weights are λ1 = 10, λ2 = 0.1, λ3 = 1×10^-4, and λ4 = 0.1.
  • Training Details: The learning rate is 5 × 10^-5 in stage 1 and 1 × 10^-5 in stage 2.

B. Generative Model Details

The generative model uses text-conditioned diffusion transformers, with Qwen2.5-VL supplying text features through cross-attention. Training begins on smaller structural latents to improve efficiency.

  • Text-to-video Generation: Text-to-video generation uses Qwen2.5-VL as the text encoder, injecting text conditions through cross-attention.
  • Text-to-video Generation: Each text-to-video DiT has 1.3B parameters, hidden dimension 1536, 20 blocks, and 16 attention heads.
  • Class-to-video Generation: Class-to-video generation uses a DiT with hidden dimension 1024, 24 blocks, and 16 attention heads.
  • Training Details: Text-to-video training first uses structural latents because they are much smaller than 1D latents.This stage uses 48 80G GPUs for 300K iterations.

C. Autoencoder Adaptation

The adaptation analysis identifies representation and decoding problems when diffusion models use 1D latents, then evaluates alignment and decoder fine-tuning as remedies. Structural-latent alignment supports spatial correspondence, while decoder adaptation removes generation artifacts.

  • Generative Model Results: The text-to-video latent diffusion model is trained on the autoencoder's structural latents.
  • Latent Representation Discrepancy: A diffusion model transferred directly to pure 1D latents produces structurally distorted characters despite the autoencoder reaching PSNR 33.61.The experiment excludes latent alignment loss and uses more than 13 million training samples.
  • Latent Representation Discrepancy: Hybrid latents preserve a coarse structural foundation but produce abrupt transitions and temporal discontinuity in subsequent frames.
  • Latent Space Alignment: The observed failures motivate aligning 1D latents with structural latents to enforce consistent spatial correspondence.
  • Latent Space Alignment: Self-alignment regularization matches 1D latents to best-matching structural counterparts and preserves reconstruction fidelity at regularization weight 0.01.
  • Decoder Fine-tuning: Combining structural and 1D latents can create prominent patch-like artifacts even when loss curves appear aligned.
  • Decoder Fine-tuning: Decoder fine-tuning conditions reconstruction on predicted clean latents, eliminating patch-like artifacts and improving visual smoothness.The decoder is fine-tuned with predicted latents from the latent diffusion model.

D. Further Analysis on Autoencoder

Further analysis finds little reconstruction benefit from scaling the autoencoder beyond 1B parameters and reports that GAN post-training worsens both reconstruction and perceptual metrics.

  • Scaling the Autoencoder: 1B- and 3B-parameter autoencoders have nearly overlapping loss curves under identical settings.The analysis reports little improvement in reconstruction from scaling beyond 1B parameters.
  • Post-training with GAN Loss: 5K iterations of GAN post-training worsen rFVD from 67.56 to 75.48 and PSNR from 36.02 to 35.67.The framework therefore avoids GAN-based post-training.

E. Limitation and Future Work

One-DVA’s future work centers on extending its current adaptive-compression framework to longer videos and principled token-length selection, while exploring broader diffusion-decoder and semantic-model integrations.

  • Future directions: Long-video modeling remains incompletely realized despite the architecture’s potential compatibility with streaming generation.The authors suggest overlapping spatiotemporal windows as a possible route.
  • Post-training: Adversarial post-training harms both rFVD and PSNR after pretraining.The study reports degradation on both metrics when GAN-based post-training is applied.
  • Future directions: The theoretically optimal token length for different video complexities remains unresolved because training currently samples token counts randomly.The authors identify the need to move beyond purely empirical estimations.
  • Decoder extensions: One-DVA’s decoder is intended as a pixel-space diffusion refiner during inference and could be integrated with features such as super-resolution.This positions the decoder as more than an encoder-supervision component.
  • Semantic extensions: Future work includes using foundation models such as CLIP for a more semantically grounded autoencoder with variational or multi-scale encoding.The proposed direction extends the current latent representation toward stronger semantic grounding.
  • Decoder extensions: The authors envision an all-in-one pixel-space diffusion decoder that combines reconstruction with conditional text/image-to-video generation.This could remove the need for a separate latent diffusion model.
Loading 2602.04220v1…