Source-linked AI summary
LTX-Video: Realtime Video Latent Diffusion
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, Ofir Bibi
TL;DR
LTX-Video addresses detail loss and decoder artifacts in highly compressed latent diffusion while integrating the Video-VAE and denoising transformer. Its holistic architecture supports efficient video generation, achieving faster-than-real-time performance and reported advantages over similar-scale models, while remaining scoped to short-video generation and prompt-sensitive outputs.
Problem
High-compression latent diffusion can lose high-frequency details and produce decoder artifacts, especially when compressed representations poorly capture such signals.
Method
LTX-Video integrates the Video-VAE and denoising transformer in a compressed latent space, relocating patchification to the VAE and sharing the final denoising objective with the decoder.
Results
LTX-Video generates video faster than real time and outperforms available similar-scale models, with 121 frames at 768 × 512 pixels generated in 2 seconds on an Nvidia H100 GPU.
Takeaways & Limitations
The integrated design supports high-resolution video generation with fine-detail generation, coherence, and prompt adherence without an additional upsampling module.
Takeaways & Limitations
The model currently focuses on videos up to ten seconds, leaving longer-duration generation with temporal consistency and prompt fidelity for future work.
Abstract
from arXiv · showhide
We introduce LTX-Video, a transformer-based latent diffusion model that adopts a holistic approach to video generation by seamlessly integrating the responsibilities of the Video-VAE and the denoising transformer. Unlike existing methods, which treat these components as independent, LTX-Video aims to optimize their interaction for improved efficiency and quality. At its core is a carefully designed Video-VAE that achieves a high compression ratio of 1:192, with spatiotemporal downscaling of 32 x 32 x 8 pixels per token, enabled by relocating the patchifying operation from the transformer's input to the VAE's input. Operating in this highly compressed latent space enables the transformer to efficiently perform full spatiotemporal self-attention, which is essential for generating high-resolution videos with temporal consistency. However, the high compression inherently limits the representation of fine details. To address this, our VAE decoder is tasked with both latent-to-pixel conversion and the final denoising step, producing the clean result directly in pixel space. This approach preserves the ability to generate fine details without incurring the runtime cost of a separate upsampling module. Our model supports diverse use cases, including text-to-video and image-to-video generation, with both capabilities trained simultaneously. It achieves faster-than-real-time generation, producing 5 seconds of 24 fps video at 768x512 resolution in just 2 seconds on an Nvidia H100 GPU, outperforming all existing models of similar scale. The source code and pre-trained models are publicly available, setting a new benchmark for accessible and scalable video generation.
1 Introduction
LTX-Video is a transformer-based latent diffusion model designed for text-to-video and image-to-video generation. It combines high compression, holistic VAE–transformer design, and architectural refinements to target fast generation with visual quality, motion fidelity, and prompt adherence.
- Motivation: The VAE decoder performs the final denoising step while converting latents to pixels, addressing fine-detail limitations from compressed latent representations.This avoids a separate second-stage upsampling module.
- Architecture: RoPE with normalized fractional coordinates replaces absolute positional embeddings, while key-query normalization stabilizes attention computations.These changes are described as improving spatial and temporal coherence and increasing attention-weight entropy.
- Results: 2 seconds generates 121 frames at 768 × 512 pixels using 20 diffusion steps on an Nvidia H100 GPU.The authors report faster-than-real-time generation and performance exceeding available models of similar scale.
- Capabilities: The model supports both text-to-video and image-to-video generation, with image conditioning handled through timestep-based conditioning without additional parameters or special tokens.The image-to-video capability is positioned as useful for content creation.
- Contributions: LTX-Video integrates the Video-VAE and denoising transformer, sharing the denoising objective between the transformer and VAE decoder.This holistic design optimizes their interaction within a compressed latent space.
- Contributions: 1:192 compression uses 32 × 32 × 8 spatiotemporal downsampling and relocated patchification to enable efficient high-quality video generation.The model increases latent depth to 128 channels while prioritizing spatial and temporal dimensions.
2 Method
LTX-Video’s method combines aggressive spatiotemporal compression with a VAE decoder that also performs the final denoising step. Its design addresses token efficiency, latent redundancy, decoder artifacts, and reconstruction quality through integrated architecture and losses.
- Latent Compression: Full spatiotemporal attention becomes practical because compressed latent spaces reduce token counts and information redundancy.The attention operation is quadratic in the number of tokens.
- Video VAE: 1:192 compression with 32 × 32 × 8 downsampling and 128 channels reduces the pixels-to-tokens ratio to 1:8192 without a transformer patchifier.The patchifying operation is moved to the VAE encoder.
- Latent Compression: VAE training reduces latent-channel redundancy over time, with auto-correlations becoming near-zero by training completion.The analysis uses PCA over latent pixels from 128 video samples.
- Shared Diffusion Objective: Residual uncertainty from limited diffusion iterations can create out-of-distribution decoder inputs and pixel-space artifacts, especially at high compression and in high-frequency regions.This motivates assigning the decoder a final denoising role.
- Shared Diffusion Objective: The decoder maps noisy latents to clean pixels at varying noise levels and performs the final denoising step directly in pixel space.Pixel-space losses let it recover details inaccessible to latent-to-latent denoising.
- Reconstruction GAN: Reconstruction GAN supplies the discriminator with both input and reconstructed samples, simplifying its comparison and improving reconstruction guidance.The authors report greater GAN stability and performance, balancing fidelity and perceptual quality.
2.2 Video Transformer
LTX-Video builds a 3D transformer on Pixart-α with modified normalization and positional-encoding schemes for variable video dimensions. Experiments favor normalized fractional RoPE coordinates, exponential frequency spacing, RMSNorm, and QK normalization.
- Positional embeddings: LTX-Video replaces Pixart-α’s absolute positional embeddings with RoPE using normalized fractional coordinates for variable video sequences.Coordinates are computed in pixels and seconds relative to predefined maximum resolution and duration.
- RoPE frequency spacing: Exponential RoPE frequency spacing outperforms inverse-exponential spacing in controlled diffusion-training experiments.The inverse-exponential configuration maintains higher training loss under matched architecture, hyperparameters, and data.
- Normalization: QK normalization is applied before dot-product attention to prevent excessively large attention logits and near-zero-entropy attention weights.The transformer block also replaces LayerNorm with RMSNorm, which performs better in comparison experiments.
- Transformer architecture: The architecture integrates RoPE, QK normalization, and RMSNorm within a 3D transformer block derived from Pixart-α.These changes target accuracy and efficiency for image and video synthesis.
- Positional embeddings: Relative-fractional positional embedding works best among the three tested positional-encoding variants.The variants were absolute embeddings, fractional-coordinate RoPE, and normalized fractional-coordinate RoPE.
2.3 Text Conditioning
LTX-Video uses pretrained text encodings and cross-attention to condition diffusion transformers on textual inputs. Cross-attention is selected over MM-DiT based on the authors’ comparison.
- Text encoding: LTX-Video uses the pretrained T5-XXL text encoder to generate initial text embeddings.The choice follows prior diffusion models that use pretrained encoders for semantic understanding.
- Conditioning architecture: The model uses cross-attention to condition diffusion transformers on text embeddings.The alternative MM-DiT architecture processes text and image-patch embeddings in parallel with unified attention layers.
- Conditioning architecture: Cross-attention performs better than MM-DiT in the authors’ comparison.Both architectures are described as common approaches for conditioning diffusion transformers on text.
2.4 Image Conditioning
LTX-Video supports first-frame image conditioning by assigning diffusion timesteps independently to tokens. Conditioning tokens receive minimal noise, allowing the model to use the encoded first frame while denoising the remaining video.
- Motivation: The approach extends Open-Sora-style image conditioning without requiring a separately trained image-to-video model.The cited passage describes existing approaches as typically requiring special tokens and task-specific training.
- Per-token timesteps: The model allows each token to receive its own diffusion timestep and corresponding noise level.This relaxes the shared-timestep restriction used in DiT and Pixart-α.
- Training: During first-frame training, tokens from the first frame are sometimes assigned a small random timestep and matching noise level.The model learns to use these lightly noised tokens as a conditioning signal.
- Inference: At inference, the encoded conditioning image is concatenated with random-noise latents and flattened into the initial token sequence.The conditioning image is encoded with the causal VAE encoder and has temporal dimension 1.
- Inference: Conditioning tokens use timestep t_c, while all other tokens use t = 1 during image-to-video inference.The conditioning timestep is small, corresponding to minimally noised encoded tokens.
2.5 Rectified-Flow Training
LTX-Video trains with rectified-flow objectives and a timestep distribution shifted toward higher-noise regions. It jointly exposes the model to varied resolutions, durations, and images while controlling token counts through stochastic dropping.
- Background: Rectified flow linearly interpolates clean latent z_0 and standard-normal noise ϵ as z_t = (1 − t)z_0 + tϵ.The timestep t ranges from 0 to 1.
- Flow objective: The training target is velocity v = ϵ − z_0 rather than noise ϵ because noise-prediction difficulty varies across timesteps.During inference, predicted velocity progressively denoises pure noise toward the clean sample.
- Timestep sampling: The timestep scheduler shifts sampling toward higher-noise regions to maintain signal-to-noise ratio at higher image resolutions.Two resolution-dependent shifted distributions are shown in Figure 10, avoiding near-zero tail probabilities.
- Data configurations: Training simultaneously covers multiple resolution-duration combinations, and the model generalizes well to unseen configurations.The authors attribute this observation to exposure to diverse width, height, and duration combinations.
- Token and image training: Stochastic token dropping at rates from 0% to 20% keeps input samples near a common token count without complex packing or padding.Images are included alongside videos as one of the resolution-duration combinations.
3 Data Preparation
LTX-Video’s data preparation combines aesthetic, motion, aspect-ratio, and captioning processes to curate and enrich diverse training data.
- The training dataset combines publicly available data with licensed material to support diverse visual content.
- Quality Control and Filtering: An aesthetic model trained on manually tagged image pairs evaluates videos and images for quality control.A Siamese Network then predicts aesthetic scores, and samples below a threshold are filtered out.
- Motion and Aspect Ratio Filtering: Videos with insignificant motion are removed, and black bars are cropped to standardize aspect ratios.
- Fine-Tuning with Aesthetic Content: The most aesthetic content is selectively used for fine-tuning to improve the visual appeal of generated outputs.
- Captioning and Metadata Enhancement: An internal automatic image and video captioner re-captions the training set to improve textual metadata and visual-textual alignment.Examples appear in Figure 12, while Figure 13 visualizes captioned-data vocabulary.
4 Experiments
Experiments evaluate model preferences, reconstruction quality, positional encodings, and decoder denoising. Results support the proposed Reconstruction GAN, exponential RoPE spacing, and VAE-decoder denoising design.
- 4.2 Evaluation: LTX-Video significantly outperforms similar-sized models in human preference evaluations across text-to-video and image-to-video tasks.The survey summarized each model’s percentage of tests won, and Figure 15 reports pairwise win ratios.
- 4.3.1 Reconstruction GAN vs. traditional GAN: At 1:192 compression, standard reconstruction and GAN losses can produce temporal artifacts in high-motion, intricate-detail frames.Figure 16 contrasts standard-GAN reconstruction with the improved Reconstruction GAN result.
- 4.3.1 Reconstruction GAN vs. traditional GAN: The Reconstruction GAN loss significantly reduces visible reconstruction artifacts under high compression.
- 4.3.2 Positional Embeddings: Exponential RoPE frequency spacing produces consistently lower training loss than inverse-exponential spacing.The comparison uses the same architecture, hyperparameters, and data as LTX-Video training.
- 4.4 Denoising VAE Decoder: Videos using the VAE decoder for the final denoising step were strongly preferred over videos using transformer-only latent denoising.The benefit was particularly evident in high-motion videos, where compression artifacts were mitigated.
5 Limitations
The paper identifies limitations in prompt sensitivity, supported video duration, and domain-specific generalization.
- Model Sensitivity to Prompt Formulation: Performance can vary significantly with textual prompt quality and clarity, while ambiguous prompts may produce less coherent outputs.
- Limited Support for Long Videos: The model currently focuses on videos up to ten seconds, leaving longer-duration generation with temporal consistency and prompt fidelity as future work.
- Domain-Specific Generalization: Domain-specific capabilities such as multi-view synthesis and fine-grained editing have not been extensively tested.Further experimentation is required to assess specialized applications.
6 Social Impact
LTX-Video is presented as an efficient, accessible, open-source system with potential benefits for deployment and creative applications, alongside misuse risks.
- Accessibility and Democratization: The model is optimized for efficiency and can run on consumer-grade GPUs, broadening access to text-to-video generation.
- Open-Source Contribution: Open-source release is intended to foster innovation and collaboration across educational, creative, and enterprise applications.
- Environmental Considerations: The model’s relatively small size reduces hardware requirements and energy consumption during training and deployment.The paper presents this as contributing to more sustainable AI deployment.
- Potential Risks and Mitigation: Wider availability could enable misuse for generating misleading content, motivating documentation with responsible-use guidelines and disclaimers.
7 Conclusion
LTX-Video integrates the Video-VAE and denoising transformer to support efficient text-to-video and image-to-video generation. It reports faster-than-real-time generation, high motion fidelity, temporal consistency, and strong conditioning alignment, while identifying longer videos and advanced temporal coherence as future directions.
- Capabilities: LTX-Video supports both text-to-video and image-to-video generation while preserving motion fidelity, temporal consistency, and alignment with prompts or conditioning frames.The conclusion presents these capabilities and quality attributes as central outcomes of the model.
- Core contribution: LTX-Video integrates the Video-VAE and denoising transformer through VAE-side patchifying and a shared diffusion objective.The design combines latent-to-pixel decoding with the final diffusion step, avoiding an additional upsampling module for fine-detail generation.
- Results: LTX-Video outperforms state-of-the-art open-source models of similar scale in speed and quality for text-to-video generation.The paper describes this result as setting a new benchmark for text-to-video generation.
- Accessibility: LTX-Video’s efficient design lowers hardware requirements and supports video generation on consumer-grade GPUs.The paper connects this accessibility with broader use by researchers, developers, and creative professionals without high-end compute resources.
- Future directions: Future work includes extending the architecture to longer videos, improving temporal coherence, and adapting it to domain-specific video tasks.Examples include multi-view video synthesis and fine-grained motion editing.