Source-linked AI summary
LTX-2: Efficient Joint Audio-Visual Foundation Model
Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifschitz, Dudu Moshe, Eitan Porat, Eitan Richardson, Guy Shiran, Itay Chachy, Jonathan Chetboun, Michael Finkelson, Michael Kupchick, Nir Zabari, Nitzan Guetta, Noa Kotler, Ofir Bibi, Ori Gordon, Poriya Panet, Roi Benita, Shahar Armon, Victor Kulikov, Yaron Inger, Yonatan Shiftan, Zeev Melumian, Zeev Farbman
TL;DR
Silent text-to-video systems and specialized or decoupled audio generators do not provide integrated, synchronized audiovisual content. LTX-2 jointly generates video and audio from text through an asymmetric dual-stream model with cross-modal conditioning and modality-aware guidance. The authors report state-of-the-art open-source audiovisual quality and speed, while noting limitations in language coverage, multi-speaker assignment, long-duration coherence, and societal risk.
Problem
Existing text-to-video models omit synchronized sound, while many text-to-audio systems are domain-specialized or decoupled rather than jointly modeling audiovisual dependencies.
Method
LTX-2 combines modality-specific VAEs, refined text embeddings, an asymmetric dual-stream diffusion transformer, bidirectional cross-attention, and modality-aware classifier-free guidance for joint text-conditioned generation.
Results
LTX-2 produces coherent, expressive audiovisual content with natural speech, ambient sound, and foley realism, achieving state-of-the-art audiovisual quality and speed among open-source T2AV systems.
Takeaways & Limitations
LTX-2 provides an open-source foundation for scalable, accessible audiovisual synthesis and further research in cross-modal alignment and controllable sound-aware video generation.
Takeaways & Limitations
Performance can weaken for underrepresented languages, multi-speaker attribution, sequences longer than roughly 20 seconds, and systems requiring deeper reasoning or world modeling.
Abstract
from arXiv · showhide
Recent text-to-video diffusion models can generate compelling video sequences, yet they remain silent -- missing the semantic, emotional, and atmospheric cues that audio provides. We introduce LTX-2, an open-source foundational model capable of generating high-quality, temporally synchronized audiovisual content in a unified manner. LTX-2 consists of an asymmetric dual-stream transformer with a 14B-parameter video stream and a 5B-parameter audio stream, coupled through bidirectional audio-video cross-attention layers with temporal positional embeddings and cross-modality AdaLN for shared timestep conditioning. This architecture enables efficient training and inference of a unified audiovisual model while allocating more capacity for video generation than audio generation. We employ a multilingual text encoder for broader prompt understanding and introduce a modality-aware classifier-free guidance (modality-CFG) mechanism for improved audiovisual alignment and controllability. Beyond generating speech, LTX-2 produces rich, coherent audio tracks that follow the characters, environment, style, and emotion of each scene -- complete with natural background and foley elements. In our evaluations, the model achieves state-of-the-art audiovisual quality and prompt adherence among open-source systems, while delivering results comparable to proprietary models at a fraction of their computational cost and inference time. All model weights and code are publicly released.
1 Introduction
LTX-2 addresses the limitations of silent and decoupled generation by jointly modeling video and audio in an open, integrated framework. Its architecture combines modality-specific representations, asymmetric processing, cross-modal attention, refined text conditioning, and modality-aware guidance for synchronized audiovisual synthesis.
- Motivation: Existing text-to-video models produce visually compelling, temporally coherent outputs but omit semantic, emotional, and environmental information carried by synchronized sound.
- Motivation: Decoupled audio-video pipelines fail to model the full joint distribution because audio drives lip synchronization while visual context determines reverberation and foley.
- Architecture: LTX-2 uses separate modality-specific VAEs, enabling modality-appropriate positional embeddings, independent compression choices, and controlled capacity allocation between vision and sound.
- Architecture: An asymmetric dual-stream transformer assigns a wider high-capacity stream to video and a narrower specialized stream to audio, concentrating computation according to information density.
- Architecture: Bidirectional cross-attention with temporal positional encoding maps visual cues to auditory events and supports tight audiovisual alignment.
- Technical Contributions: LTX-2 adds refined multilingual text processing, thinking-token blocks, compact audio representations, and modality-aware classifier-free guidance for prompt understanding and controllability.
2 Related Work
Related work progresses from strong but silent text-to-video systems and specialized or sequential audio generation toward joint text-to-audio-video modeling. LTX-2 integrates separate audio and video streams while targeting lower overhead and stronger cross-modal interaction.
- Foundations: Diffusion Transformers and Rectified Flow provide scalable latent-space architectures and more efficient denoising foundations for multimodal generation.
- Text-to-Video Models: Recent text-to-video models generate visually rich, temporally coherent clips but remain silent; LTX-2 extends this architecture with a parallel synchronized audio stream.
- Decoupled Audio-Visual Synthesis: Sequential audio-video pipelines suffer a modality-first bottleneck because the first generated modality may lack cues needed for realistic soundscapes or video.
- Joint T2AV Models: Joint T2AV systems address synchronized generation from one prompt, but proprietary models are closed and some open approaches duplicate backbones, increasing overhead and limiting cross-modal synergy.
- Text Conditioning: Text-conditioning research increasingly uses pretrained encoders and refinement stages, while causal decoder-only encoders can limit global contextual awareness.
3 Method
LTX-2 jointly models video and audio using modality-specific latent representations, an asymmetric dual-stream transformer, and cross-modal conditioning. Its text pipeline extracts and projects intermediate multilingual encoder features to support semantic and phonetic grounding.
- Audiovisual Joint Generation: LTX-2 models video and audio as separate latent streams produced by modality-specific VAEs and jointly denoises them through bidirectional cross-modal exchange.Video latents use a spatiotemporal causal VAE, while audio latents come from mel spectrograms encoded by a separate causal audio VAE.
- Text Understanding: Dedicated transformer blocks refine text embeddings and predict thinking tokens, which are supplied alongside original tokens to the dual-stream transformer through cross-attention.The LLM weights remain frozen while the projection matrix W is optimized during an initial training stage.
- Audiovisual Joint Generation: Bidirectional audio-visual cross-attention exchanges information between streams, with temporal RoPE emphasizing synchronization and cross-modality AdaLN conditioning representations across diffusion timesteps.AdaLN scales and shifts cross-modal attention components based on timestep-conditioned hidden states.
- Text Understanding: The text-conditioning pipeline encodes prompts with Gemma3, extracts intermediate representations across decoder layers, and projects them into a unified embedding space for diffusion conditioning.The feature extractor mean-centers and scales layer outputs, flattens them to [B, T, D × L], and projects them to dimension D with W.
4 Inference
Inference combines textual and cross-modal classifier-free guidance, then uses a multi-scale, tiled latent process to generate synchronized high-resolution audiovisual content efficiently.
- Inference Classifier-free Guidance: Multimodal CFG adds an independently weighted cross-modal guidance direction to standard text guidance for each video or audio stream.The model receives the current modality latent, text condition, and complementary-modality features.
- Inference Classifier-free Guidance: Increasing sm promotes mutual-information refinement and improves temporal synchronization and semantic coherence between generated video and audio.Reported guidance weights are st = 3 and sm = 3 for video, and st = 7 and sm = 3 for audio.
- Multi-scale, Multi-tile Inference: The multi-scale, multi-tile strategy generates Full-HD (1080p) audiovisual content without processing all high-resolution video latents in one pass.Overlapping spatial and temporal tiles are refined independently and blended in latent space before VAE decoding.
- Multi-scale, Multi-tile Inference: Inference begins from an approximately 0.5 Megapixel base latent that establishes scene composition, motion dynamics, and foundational audio-visual synchronization.A dedicated latent upscaler then increases video resolution while maintaining temporal consistency and auditory alignment.
5 Training Data
LTX-2 training uses audio-rich video clips and detailed captions that describe both visual and auditory content, including soundscapes, dialogue, and scene dynamics.
- Training Data: The training subset selects video clips with significant audio components and balances visual and auditory content for multimodal captioning.The clips come from a subset of the dataset used for LTX-Video.
- Training Data: A dedicated captioning system describes every meaningful action, appearance, and sound across a clip’s visual and auditory tracks.Its captions are intended to be comprehensive and factual rather than emotionally interpretive.
- Training Data: Captions cover music, ambient sounds, dialogue transcriptions with speaker, language, and accent identification, plus camera motion, lighting, and subject behavior.This creates a textual interface connecting video, audio, and language domains.
6 Experiments
LTX-2 is evaluated for audiovisual quality, visual-only performance, and computational efficiency. It outperforms open-source alternatives, approaches proprietary-model human preferences, retains strong video rankings, generates longer sequences, and substantially reduces inference time.
- Audiovisual Evaluation: Human studies assess visual realism, audio fidelity, and temporal synchronization, including lip-sync and foley accuracy.
- Audiovisual Evaluation: LTX-2 significantly outperforms the open-source alternative Ovi and achieves human preference scores comparable to Veo 3 and Sora 2.
- Video-Only Benchmarks: LTX-2 ranks 3rd in Image-to-Video and 4th in Text-to-Video in Artificial Analysis public rankings as of November 6th, 2025.
- Inference Performance and Scalability: 18× faster than Wan 2.2 in time per diffusion-step on an H100 GPU, with the gap widening at higher resolutions and longer durations.The comparison uses 121 frames at 720p, a single-step Euler solver, and CFG=1.
- Inference Performance and Scalability: LTX-2 generates up to 20 seconds of continuous video with synchronized stereo audio, exceeding the temporal limits of listed proprietary and open-source alternatives.The cited alternatives include Veo 3, Sora 2, Ovi, and Wan 2.5.
7 Limitations
LTX-2 has limitations involving language coverage, multi-speaker attribution, sequence duration, and reasoning capabilities.
- Underrepresented languages or dialects may produce less accurate speech synthesis or weaker audiovisual alignment.
- Multi-speaker scenarios may inconsistently assign spoken content to characters.
- Sequences longer than roughly 20 seconds can develop temporal drift, degraded synchronization, or reduced scene diversity.
- LTX-2 lacks explicit reasoning and world-modeling capabilities, so deeper narrative coherence and factual grounding depend on external systems.
8 Social Impact
Text-to-audio+video generation may support creativity, accessibility, and communication while introducing risks from synthetic media and training-data biases.
- LTX-2 can help creators, educators, and storytellers produce expressive audiovisual material without specialized equipment or large production teams.
- Synchronized text-to-audio+video generation has promise for low-resource languages and accessibility applications, including inclusive media and educational dubbing.
- Realistic synthetic media can be misused for deceptive or manipulative content, motivating disclosure of synthetic origin and adherence to safety guidelines.
- Biases in training data may manifest in both visual and auditory modalities, motivating future work on mitigation, verification, and traceability.
9 Conclusion
LTX-2 is an open-source foundation model that jointly generates synchronized video and audio from text. Its design targets efficient multimodal generation, while experiments report state-of-the-art audiovisual quality and leading speed among open-source T2AV systems.
- LTX-2 jointly generates synchronized video and audio from text as an open-source text-to-audio+video foundation model.
- Bidirectional cross-attention, 1D temporal RoPE, cross-modality AdaLN, modality-aware guidance, and progressive joint training support efficient, coherent audiovisual generation.
- Experiments report state-of-the-art audiovisual quality and the fastest performance among open-source T2AV models.
A.1 Additional Figures
Figure A1 presents LTX-2’s training and inference pipelines, while Figure A2 shows the architecture of an individual stream shared by audio and video.
- During training, audio and video inputs are encoded into latents and optimized by matching their velocity fields with a flow-matching loss.
- During inference, the model starts from noise in both latent spaces and iteratively denoises them over N diffusion steps.
- VAE decoders and an upsampling vocoder reconstruct the denoised latents as the final waveform and video.
- Figure A2 details one stream, whose architecture is identical for the audio and video streams.