Source-linked AI summary
FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space
FSVideo Team, Qingyu Chen, Zhiyuan Fang, Haibin Huang, Xinwei Huang, Tong Jin, Minxuan Lin, Bo Liu, Celong Liu, Chongyang Ma, Xing Mei, Xiaohui Shen, Yaojie Shen, Fuwen Tan, Angtian Wang, Xiao Yang, Yiding Yang, Jiamin Yuan, Lingxi Zhang, Yuxin Zhang
TL;DR
Large video models offer strong generation ability but often incur high inference cost, motivating faster image-to-video generation. FSVideo addresses this with compressed video latents, a layer-memory DIT, and multiresolution refinement, achieving competitive quality with order-of-magnitude faster inference.
Problem
Large video models often have high inference costs, causing long waiting times and high GPU costs, while some acceleration methods limit speedups or degrade quality.
Method
FSVideo combines a 64 × 64 × 4 compressed video autoencoder, a layer-memory DIT, and a multiscale latent upsampler-refiner for image-to-video generation.
Results
Order-of-magnitude faster than similarly sized open-source models while generating competitive videos; it is reported as 42.3× faster than Wan2.1-14B.
Takeaways & Limitations
FSVideo indicates that reducing token amount through highly compressed latent representations can support efficient video generation without reducing the model to a lightweight architecture.
Takeaways & Limitations
Future work remains for higher-resolution generation, stronger prompt coherence and motion, longer or multiscene videos, and multimodal generation.
Abstract
from arXiv · showhide
We introduce FSVideo, a fast speed transformer-based image-to-video (I2V) diffusion framework. We build our framework on the following key components: 1.) a new video autoencoder with highly-compressed latent space ($64\times64\times4$ spatial-temporal downsampling ratio), achieving competitive reconstruction quality; 2.) a diffusion transformer (DIT) architecture with a new layer memory design to enhance inter-layer information flow and context reuse within DIT, and 3.) a multi-resolution generation strategy via a few-step DIT upsampler to increase video fidelity. Our final model, which contains a 14B DIT base model and a 14B DIT upsampler, achieves competitive performance against other popular open-source models, while being an order of magnitude faster. We discuss our model design as well as training strategies in this report.
1 Introduction
FSVideo targets the high inference cost of large video generation models with a highly compressed latent space, modified DIT, and latent upsampler. The resulting image-to-video system aims for competitive quality with substantially faster inference.
- Large video models often impose long waits and high GPU costs, while existing acceleration methods can provide limited speedups or degrade generation quality.
- FSVideo reduces per-forward-pass computation with a video autoencoder using 64 × 64 × 4 spatial-temporal compression.The highly compressed latent space is trained with a DIT from scratch rather than only adapting a pretrained DIT.
- FSAE uses 64 × 64 × 4 spatial-temporal compression with 128 latent channels, yielding 384× information reduction and competitive reconstruction performance.
- The layer-memory DIT design increases information-flow flexibility and DIT capacity utilization with negligible overhead.
- A convolutional latent upsampler and DIT refiner increase video fidelity while limiting added inference time through lightweight refiner step distillation.
- 42.3× faster than Wan2.1-14B, FSVideo generates competitive-quality videos compared with similarly sized open-source models.
2.1 Framework Overview
FSVideo is formulated as image-to-video to simplify training, work within resource constraints, and match applications that provide an initial image. The input image supplies appearance information while the model focuses on video movement.
- FSVideo uses an input image as the first-frame condition for video generation rather than targeting text-to-video directly.
- Restricting training to image-to-video supplies video-appearance information through the input image and lets diffusion training focus more on modeling movement.
- Image-to-video is more data-friendly under limited resources, and a high-quality text-to-image model can provide aesthetically controlled first frames.
- Applications such as photo reenactment and visual effects naturally provide an image as the video's first frame.
2.2 Video Autoencoder
FSAE is a highly compressed video autoencoder designed to balance reconstruction quality, generation capability, and decoding efficiency. Its architecture, training objectives, and decoder variants support this trade-off.
- Overall Design And Training Strategy: FSAE training combines multi-stage reconstruction and adversarial objectives with Video VF Loss to improve latent generation capability.Video VF Loss aligns video-autoencoder latents with framewise DINOv2 features through video feature-matching losses.
- Decoder Improvement: The autoencoder’s highly compressed latent space introduces reconstructed-video artifacts and substantial decoding cost, motivating asymmetric decoder optimization.The encoder is frozen while the decoder is changed and finetuned to improve quality and decoding performance.
- Decoder Improvement: FSAE-Standard improves video quality through non-causal decoder convolutions, first-frame feature injection, and Gaussian noise injection.Non-causal convolutions reduce temporal flickering, first-frame features provide an image-to-video condition, and noise injection targets high-frequency details.
- Decoder Improvement: FSAE-Lite reduces memory consumption and inference time by modifying FSAE-Standard’s decoder at the cost of slightly degraded performance.The lightweight variant reduces channels in the largest-feature-map blocks and uses a compromise between causal and non-causal convolution choices.
2.3 Video Diffusion Transformer
FSVideo builds its diffusion transformer on Wan2.1-14B-I2V, adding layer memory so each layer can reuse earlier representations through adaptive key–value construction. The mechanism supports selective inter-layer information flow, improves convergence, and retains lightweight compatibility with existing DIT implementations.
- DIT Architecture: FSVideo uses Wan2.1-14B-I2V’s DIT structure, with 3D-convolution patchification, transformer layers, and unpatchification for highly compressed latent tokens.The latent sequence has shape (B, T, D), with T = (1 + F/4) × H/64 × W/64.
- Layer Memory Self-Attention: Layer Memory Self-Attention lets each layer adaptively attend to partial attention features from all preceding layers, improving information flow and temporal coherence.The transformer layer is conditioned on UmT5 text embeddings and CLIP image embeddings.
- Layer Memory Self-Attention: Layer memory keeps standard query projection from the previous layer while deriving keys and values from a learned fusion of all earlier hidden representations.This extends attention to use contextual structure accumulated across the network hierarchy.
- Dynamic Router: The inter-layer dynamic router produces time-aware weights over prior representations, allowing adaptive fusion that emphasizes informative layers and reuses features across depth.The router is maintained for layers l ≥ 2 and uses representations modulated by the diffusion time embedding.
- Efficiency and Compatibility: The design introduces minimal additional trainable parameters, preserves the original DIT architecture, and remains compatible with efficient implementations such as FlashAttention.The report also states that the mechanism adds negligible overhead.
- Router Analysis: The router heatmap shows strong immediate-predecessor attention, direct reuse of the first-frame token, and selective links from later layers to much earlier representations.The later-layer links include a cluster connecting layers 13–30 with earlier representations, associated with low-level, high-frequency features.
- Training Convergence: With Layer Memory, training from scratch maintains lower loss, while fine-tuning a pretrained WAN2.1 model converges within 100 steps and gains up to 4.7% after 1,000 steps.The reported comparison is against the baseline without Layer Memory.
2.4 Video Upsampler
FSVideo adds a latent upsampling and high-resolution refinement pipeline to recover detail lost under strong compression. The refiner uses video-to-video conditioning and targeted training strategies to preserve structure, restore artifacts, and retain text-to-video capability.
- Video Upsampler: The upsample step combines a convolutional latent upsampler with a high-resolution DIT refiner to improve fidelity after base DIT generation.The latent upsampler enlarges low-resolution latents, while the refiner generates the full sequence and restores high-frequency features.
- Video Upsampler: FSVideo upsamples low-resolution latents by a factor of 2 instead of interpolating RGB frames, avoiding additional VAE encoding-decoding operations.The convolutional module uses a projection layer, pixel-shuffle upsampling, and 16 residual blocks.
- Video Upsampler: The latent upsampler is trained from low- and high-resolution latent pairs using latent L1, decoded L1, and LPIPS losses.High-resolution videos are downsampled and separately encoded before the low-resolution latent is upsampled and compared with the high-resolution target.
- High-resolution Refiner: The high-resolution refiner performs video-to-video generation, conditioning jointly on the high-resolution first frame and subsequent low-resolution frames.This setting requires faithful first-frame adherence, strong restoration of artifacts in generated inputs, and preservation of text-to-video capability.
- High-resolution Refiner: Dynamic masking differentiates genuine and generated latents by assigning mask values from the normalized difference between the original and upsampled first-frame latents.Uniform masking makes it difficult for the model to distinguish genuine latents from low-resolution latents.
- High-resolution Refiner: Deviation-based estimation deliberately perturbs the predicted clean latent so the refiner learns artifact correction rather than input replication.The perturbation maintains a controlled offset from the true latent across timesteps, while condition dropout and frame shuffling expose the model to varied conditions.
3 Experiments
Experiments evaluate FSVideo’s generation quality and inference speed against leading image-to-video models. The reported results show competitive quality and a large speed advantage, including a 42.3× duo-GPU speedup over Wan2.1 14B.
- Generation Quality: VBench 2.0 evaluates FSVideo on I2V Score and Quality Score, including criteria such as subject consistency, aesthetic quality, and motion smoothness.The experiments follow the official VBench pipeline for image-to-video evaluation.
- Generation Quality: FSVideo achieves the best total score among models based on Wan 2.1 DIT while using a higher compression rate than other deep-compression methods.The comparison includes DC-VideoGen and other Wan 2.1-based models.
- Generation Quality: Human evaluations find FSVideo drastically outperforms HunyuanVideo and LTX-Video and is on par with Wan 2.1 14B.FSVideo is less preferred than Wan 2.2 14B, which the authors attribute to limited training data and compute.
- Inference Speed: 42.3× speedup is achieved over Wan2.1 14B in the duo-GPU scenario for 5-second 720 × 1280, 24 fps video generation.FSVideo generates the target video in 76.6 seconds in the single-GPU case, while Wan cannot generate it under parameter offloading.
- Inference Speed: At least 58.7× estimated speedup over Wan2.1 14B is possible when the GPU memory constraint is avoided, such as with FP8 quantization.The reported 42.3× comparison uses a similar amount of NFE, with the evaluation conventions specified for Wan, FSVideo base DIT, and the refiner.
4 Conclusion
FSVideo combines highly compressed video latents, layer-memory DIT design, and multiscale refinement for fast image-to-video generation. The authors report competitive videos at roughly an order-of-magnitude higher speed, with further acceleration possible through distillation.
- Conclusion: FSVideo uses a high-compression autoencoder, layer-memory self-attention, and a multiscale latent-refiner strategy as its core design.These components target token reduction, feature-information reuse, and increased video fidelity.
- Conclusion: FSVideo generates competitive videos while being an order of magnitude faster than video generation models with similar parameter sizes.The conclusion frames speed and quality as the principal outcome of the combined design.
- Conclusion: Post-training step distillation and model distillation could further enhance FSVideo’s inference speed.The paper presents these methods as future ways to accelerate inference beyond the reported system.
- Future Work: Future work includes higher-resolution encoding, more efficient DIT designs, improved prompt coherence and motion, multimodal generation, and longer or multiscene videos.These directions define the stated extension scope of the method.
- Conclusion: The paper argues that reducing token count can improve efficiency while preserving the model capacity needed for complex generative distributions.This contrasts with speedups that reduce model capacity and may trade quality for speed.
5 Contributors
The report lists core contributors and additional contributors, with names alphabetized by last name and then first name. An asterisk marks contributors who have left the company.
- Contributors: Core contributors are Xinwei Huang, Minxuan Lin, Yaojie Shen, Xiao Yang*, and Yuxin Zhang.The names are presented in alphabetical order by last name and then first name.
- Contributors: Additional contributors are Qingyu Chen*, Zhiyuan Fang, Haibin Huang*, Tong Jin, Bo Liu, Celong Liu*, Chongyang Ma, Xing Mei*, Xiaohui Shen, Fuwen Tan, Angtian Wang, Yiding Yang, Jiamin Yuan, and Lingxi Zhang.Asterisks indicate people who have left the company.