PaperlayerView exact revision on arXiv

Source-linked AI summary

TurboT2VA: Fast Large-Scale Text-to-Video-Audio Generation via Score-Regularized Consistency Distillation

Xiaoda Yang, Yuxiang Liu, Kaiwen Zheng, Yuan Liu, Yibo Lai, Shengpeng Ji, Kai Jiang, Jianfei Chen, Xiaobin Hu, Shuicheng Yan, Jintao Zhang, Jun Zhu, Zhou Zhao

arXiv:2608.24674v1cs.CV

TL;DR

Large joint text-to-video-audio models are expensive to sample, while multimodal distillation must balance modality contributions, continuous-time stability, and quality against diversity. TurboT2VA uses modality-balanced joint distillation with a progressive dCM→sCM→sCM+DMD curriculum and an architecture-aware inference stack. The resulting four-step student and optimized deployment stack substantially reduce latency while retaining competitive synchronized video-audio quality.

  • Problem

    Large-scale joint T2VA distillation faces modality imbalance, continuous-time consistency instability, and an unresolved quality–diversity trade-off.

  • Method

    TurboT2VA combines per-modality normalization, shared cross-modal distillation, a progressive dCM→sCM→sCM+DMD curriculum, and architecture-aware inference optimizations.

  • Results

    20.1× speedup reduces latency from 50.52s to 2.51s at 512×768, while 54.67× generator-only speedup reduces high-resolution latency from 318.74s to 5.83s on one H20, retaining competitive multimodal quality.

  • Takeaways & Limitations

    Four-step joint generation offers a favorable efficiency–quality trade-off, and combining distillation with systems optimization supports efficient deployment of large-scale T2VA models.

Abstract

from arXiv · show

Joint text-to-video-audio generation produces synchronized visual and acoustic content, but the long sampling trajectories and heterogeneous multimodal computation of large models make inference prohibitively expensive. We present TurboT2VA, a distillation and inference framework for accelerating a 19B-parameter joint video-audio model. Large-scale T2VA distillation is challenged by modality-imbalanced optimization, the difficulty of continuous-time consistency training at scale, and the quality--diversity trade-off. TurboT2VA addresses these issues with per-modality normalization and a progressive curriculum comprising discrete consistency warm-up, continuous consistency refinement, and joint consistency--distribution matching. The curriculum first establishes a stable, diverse generation trajectory and only then introduces distribution-level refinement. On LTX-2, four-step distillation reduces generator latency from 50.52s to 2.51s at the standard evaluation resolution of 512$\times$768, achieving a 20.1$\times$ speedup while maintaining strong visual quality, audio fidelity, diversity, and video-audio synchronization. We further develop an architecture-aware inference stack that combines guarded W8A8 and fused operators, padded-text compaction, and modality-aware sparse attention while preserving dense cross-modal and text-conditioning paths. Under the high-resolution deployment setting at 1024$\times$1792, the complete stack reduces generator latency from 318.74s to 5.83s on one NVIDIA H20, achieving a 54.67$\times$ generator-only speedup. Inference code and generation demos are available at https://github.com/thu-ml/TurboDiffusion/tree/main/turbot2va.

I. INTRODUCTION

TurboT2VA addresses the high cost and unresolved quality–diversity balance of large-scale joint text-to-video-audio generation through progressive consistency and distribution-matching distillation. A four-step student and architecture-aware inference stack substantially reduce latency while retaining competitive synchronized multimodal quality.

  • Motivation: Joint T2VA models incur high inference costs because they use multi-step sampling and process visual and acoustic modalities together.
  • Challenges: Consistency distillation preserves diversity, whereas distribution matching improves realism but reduces diversity, leaving their balance unresolved in T2VA.
  • Approach: TurboT2VA extends score-regularized continuous-time consistency distillation to a 19B-parameter joint model with per-modality normalization and progressive training.
  • Performance: 20.1× speedup reduces latency from 50.52s to 2.51s at 512×768 while maintaining a strong trade-off across visual quality, audio fidelity, diversity, and synchronization.
  • Approach: TurboT2VA jointly distills video and audio through a shared process, preserving temporal consistency rather than training independent modality branches.

III. CROSS-MODAL JOINT DISTILLATION

TurboT2VA performs joint consistency distillation on paired video-audio latents using shared conditioning and a shared cross-modal Transformer. Its continuous-time formulation couples modality directions through joint tangent estimation, while per-modality normalization balances their contributions.

  • Joint formulation: TurboT2VA distills paired video-audio latents jointly rather than treating video and audio as independent branches.
  • Joint formulation: Video and audio use independently sampled noises but share the base timestep and text condition, placing both modalities on the same generation trajectory.
  • Shared backbone: The student predicts both modalities in one forward pass through a shared T2VA Transformer with paired latents, shared conditioning, and timestep embeddings.
  • Continuous-time consistency: The continuous-time consistency formulation estimates a joint tangent through the paired teacher direction, increasing optimization difficulty for the shared model.
  • Cross-modal coupling: The joint tangent couples modalities because the video direction depends on audio tokens and the audio direction depends on video tokens.
  • Modality balancing: Per-modality normalization balances consistency directions before forming the joint objective across video and audio.
  • Distillation objective: The resulting objective transfers paired video-audio trajectories to the few-step student while preserving motion, acoustic evolution, rhythm structure, and temporal correspondence.

B. Joint Distribution Matching

TurboT2VA introduces DMD after sCM has established a stable joint trajectory, using paired student-generated video-audio samples to improve realism without disrupting cross-modal structure.

  • Joint Distribution Matching: DMD is introduced after sCM stabilizes the joint trajectory to improve few-step perceptual quality.The staged design separates trajectory preservation from distribution-level refinement.
  • Joint Distribution Matching: DMD compares teacher and fake-score predictions on perturbed student-generated samples to form a score-discrepancy signal.The fake-score network is trained on student-generated samples, while teacher predictions use classifier-free guidance.
  • Joint Distribution Matching: DMD uses residual-scale normalization and aggregates the surrogate over all latent elements before applying modality and outer-coefficient scaling.The accumulated signal is retained across the full paired latent while overall scale remains controlled.
  • Joint Distribution Matching: Unlike independent modality DMD, the objective uses paired video-audio samples from the same student condition and timestep.This preserves the cross-modal structure learned through joint sCM.

C. Modality-Balanced Joint Optimization

TurboT2VA balances video and audio contributions at the loss level while retaining a single unified generator and separate normalization schemes for sCM and DMD.

  • Modality-Balanced Joint Optimization: Modality-wise balancing prevents video dominance caused by unequal latent dimensionalities, temporal resolutions, and learning speeds.The student remains a single unified T2VA generator rather than separate modality branches.
  • Modality-Balanced Joint Optimization: sCM uses an L2 direction normalizer, whereas DMD uses a teacher-residual scale normalizer before modality-weighted combination.The different normalizers reflect the distinct roles of the two objectives.
  • Modality-Balanced Joint Optimization: The joint objective combines sCM for paired trajectory preservation with DMD for teacher-distribution matching.The coefficients λsCM and λDMD control the balance between trajectory consistency and distribution-level anchoring.

D. LTX-2 TrigFlow Adaptation

TurboT2VA adapts LTX-2’s rectified-flow backbone to TrigFlow-based sCM through a wrapper that unifies video and audio prediction interfaces without changing the backbone.

  • LTX-2 TrigFlow Adaptation: The wrapper maps TrigFlow latents and timesteps to LTX-2’s rectified-flow interface.This bridges the parameterization used by sCM and the original LTX-2 backbone.
  • LTX-2 TrigFlow Adaptation: The adaptation exposes unified video and audio predictions under one timestep interface for joint sCM and DMD training.It supports JVP computation through timestep embeddings, tokenization, positional embeddings, and the cross-modal Transformer.
  • LTX-2 TrigFlow Adaptation: Direct sCM+DMD training is feasible but yields weaker quality–diversity trade-offs, motivating staged optimization.The observed weakness is especially associated with diversity and video-audio dynamics.

A. Empirical Observation: Trainable but Suboptimal

Direct joint sCM+DMD training is trainable and produces reasonable results, but the staged curriculum performs better by ordering trajectory learning before distribution matching.

  • Empirical Observation: Trainable but Suboptimal: Direct sCM+DMD training converges reasonably, yet staged training later achieves a Javis-score advantage at the common 7,500-step endpoint.The staged route overtakes direct joint training after entering its final stage.
  • Empirical Observation: Trainable but Suboptimal: Curriculum distillation structures optimization by teaching the student to follow the teacher trajectory before stronger distribution-level supervision.This ordering is presented as the source of the curriculum’s benefit.
  • Empirical Observation: Trainable but Suboptimal: sCM better preserves trajectory diversity and temporal structure, whereas DMD provides stronger final-sample quality.The objectives therefore contribute different strengths to the quality–diversity trade-off.
  • Empirical Observation: Trainable but Suboptimal: Introducing DMD too early can weaken diversity because the student has not yet learned a sufficiently broad trajectory prior.For T2VA, diversity includes motion, rhythm, sound events, and video-audio correspondence.
  • Empirical Observation: Trainable but Suboptimal: The first curriculum stage uses dCM warm-up for basic denoising, stable timestep conditioning, and a coarse joint denoising prior.dCM avoids continuous-time trajectory-tangent estimation during warm-up.

2) Stage 2: sCM Refinement:

Stage 2 uses continuous-time consistency refinement to turn a coarse student denoiser into a trajectory-consistent generator before distribution-level supervision. This preserves trajectory fidelity, diversity, and multimodal temporal correspondence while motivating architecture-aware acceleration for heterogeneous Transformer paths.

  • Stage 2: sCM Refinement:: sCM refines the dCM-initialized student into a trajectory-consistent generator by enforcing consistency across noise levels and transferring teacher generation directions.
  • Stage 2: sCM Refinement:: Trajectory refinement preserves sample diversity and improves correspondence across visual motion, audio evolution, and video-audio timing.
  • Stage 2: sCM Refinement:: sCM and DMD are staged because sCM preserves trajectory structure and diversity, whereas DMD improves final-sample realism through teacher-distribution alignment.
  • Stage 2: sCM Refinement:: The dCM→sCM→sCM+DMD sequence builds a coarse denoising prior, expands it into a trajectory-consistent generator, and improves distribution-level quality.
  • Stage 2: sCM Refinement:: Joint Transformer acceleration must distinguish video, audio, cross-modal, and text-attention paths because their sequence lengths, masks, and matrix shapes differ.
  • Stage 2: sCM Refinement:: SageSLA sparsifies only unmasked video and audio self-attention, retaining dense bidirectional cross-modal and masked text cross-attention.

C. Fused Multimodal Transformer Operations

The inference stack fuses multimodal Transformer operations, compacts padded text conditioning, and applies guarded acceleration according to architectural role and shape. Its high-resolution evaluation targets deployment bottlenecks while preserving safe fallback behavior and valid conditioning.

  • C. Fused Multimodal Transformer Operations: Fused kernels replace repeated normalization, modulation, gated residual, output-modulation, and rotary-embedding operations in the joint Transformer.
  • C. Fused Multimodal Transformer Operations: Padded-text compaction removes only invalid prompt positions during batch-size-one inference, leaving every valid conditioning token unchanged.
  • C. Fused Multimodal Transformer Operations: The default high-resolution configuration combines ρℓ=0.3 self-attention sparsity, compatible TileLang W8A8 linear layers, fused kernels, and compacted text context.
  • C. Fused Multimodal Transformer Operations: The evaluation model is a 19B-parameter LTX-2 system with 14B video and 5B audio backbones, distilled to preserve multimodal generation properties.
  • C. Fused Multimodal Transformer Operations: Training uses 100K text-video-audio samples on 8 H20 GPUs, while standard evaluation uses a fixed 200-prompt split and eight seeds per prompt for diversity.
  • C. Fused Multimodal Transformer Operations: High-resolution latency is measured on one NVIDIA H20 at 1024×1792 and 121 frames using generator-only time, excluding loading, decoding, muxing, and I/O.

B. Main Results

TurboT2VA compresses the 40-step teacher into four steps while retaining strong multimodal quality, fidelity, diversity, and synchronization. Quantitative and qualitative comparisons show efficient generation with preserved video and audio structure.

  • B. Main Results: 50.52s to 2.51s: four-step TurboT2VA achieves a 20.1× speedup at the 512×768 standard evaluation resolution.
  • B. Main Results: The four-step student preserves coherent subjects, scene layout, temporal appearance, and audio structure across early, middle, and late examples.
  • B. Main Results: TurboT2VA maintains comparable overall JavisBench quality and improves several video-audio quality and synchronization metrics despite aggressive step reduction.
  • B. Main Results: The distilled student maintains comparable VBench video fidelity, with improved aesthetic and imaging quality and strong temporal consistency.
  • B. Main Results: Under matched prompts and seeds, staged training balances sCM-only variation and DMD-only polish, preserving coherent cat–robot interactions and cross-seed variation.
  • B. Main Results: Four-step inference maintains comparable audio quality and text-audio alignment to the teacher while substantially reducing sampling steps.

3) Diversity Evaluation:

TurboT2VA’s staged consistency–distribution matching approach improves the quality–diversity balance while an architecture-aware inference stack substantially reduces high-resolution generator latency. The evaluation also examines diversity methodology, sparse-attention trade-offs, and curriculum behavior.

  • Diversity Evaluation: The diversity score averages within-prompt pairwise distances from modality-specific ImageBind embeddings across 8 samples generated with different random seeds.The protocol produces 28 pairs per prompt and 5,600 pairs per modality over 200 prompts.
  • Diversity Evaluation: DMD-only attains competitive quality but the lowest diversity, whereas sCM-only preserves the strongest diversity at substantially lower quality.The staged sCM+DMD approach provides a more favorable quality–diversity balance.
  • Diversity Evaluation: sCM-only changes subjects and layouts substantially but less reliably preserves the requested interaction, while DMD-only produces polished yet similar centered compositions.The staged variant preserves the interaction while retaining visible cross-seed changes in appearance, robot design, and scene layout.
  • Inference Efficiency: 5.83s generator latency at 1024×1792 yields a 54.67× overall speedup over the dense 40-step teacher on one NVIDIA H20.The complete stack combines W8A8, fused operations, four-step distillation, modality-aware sparse attention, and padded-text compaction; the unoptimized student takes 16.50s.
  • Inference Efficiency: Sparse attention provides the largest single-component latency gain, while the complete inference stack reduces high-resolution generator latency to 5.83s.Fused operations and W8A8 reduce dense four-step latency from 16.50s to 13.77s and 14.73s, respectively, and jointly to 12.16s.
  • Inference Efficiency: Reducing SageSLA retention from 0.5 to 0.2 lowers latency from 6.44s to 5.57s while quality and alignment metrics remain close to the dense reference.The default ρ = 0.3 balances latency and cross-modal alignment, achieving the best CAVP score and lower desynchronization error than ρ = 0.2.

C. Ablation Study

Ablations support progressive warm-up before joint distribution refinement and show that additional sampling steps generally improve generation quality, with four-step inference outperforming two-step inference across most metrics.

  • Effect of Progressive Warm-up: The staged curriculum outperforms direct sCM+DMD optimization after joint training begins and retains a clear advantage through 7,500 global steps.dCM warm-up establishes coarse denoising, sCM refinement aligns with the continuous teacher trajectory, and DMD is introduced afterward.
  • Effect of Progressive Warm-up: After the shared dCM+sCM prefix, staged sCM+DMD reaches the strongest late-training quality, peaking at 0.1963 at 7,000 steps.DMD refinement improves quickly but plateaus below the joint objective, while sCM-only remains stable at substantially lower Javis score.
  • Sampling-Step Ablation: Generation quality generally improves as TurboT2VA uses more sampling steps.The sampling-step ablation evaluates different step counts for the distilled model.
  • Sampling-Step Ablation: Four-step inference provides further gains across most metrics compared with two-step inference, which achieves reasonable performance.This supports four steps as the stronger quality setting among the reported low-step configurations.
  • Overall Performance: At 512×768, four-step distillation reduces latency from 50.52s to 2.51s, achieving a 20.1× speedup while retaining competitive multimodal quality and diversity.The reported evaluation covers visual quality, audio fidelity, cross-modal consistency, and sample diversity.
  • Overall Performance: At 1024×1792, the complete architecture-aware stack reduces generator latency from 318.74s to 5.83s on one NVIDIA H20, achieving a 54.67× generator-only speedup.The stack combines model-level distillation with W8A8, fused operations, padded-text compaction, and modality-aware sparse attention while preserving dense conditioning paths.
Loading 2608.24674v1…