PaperlayerView exact revision on arXiv

Source-linked AI summary

TurboT2VA: Fast Large-Scale Text-to-Video-Audio Generation via Score-Regularized Consistency Distillation

Xiaoda Yang, Yuxiang Liu, Kaiwen Zheng, Yuan Liu, Yibo Lai, Shengpeng Ji, Kai Jiang, Jianfei Chen, Xiaobin Hu, Shuicheng Yan, Jintao Zhang, Jun Zhu, Zhou Zhao

arXiv:2608.24674v1cs.CV

TL;DR

Large joint text-to-video-audio models are expensive to sample, while multimodal distillation must balance modality contributions, continuous-time stability, and quality against diversity. TurboT2VA uses modality-balanced joint distillation with a progressive dCM→sCM→sCM+DMD curriculum and an architecture-aware inference stack. The resulting four-step student and optimized deployment stack substantially reduce latency while retaining competitive synchronized video-audio quality.

  • Problem

    Large-scale joint T2VA distillation faces modality imbalance, continuous-time consistency instability, and an unresolved quality–diversity trade-off.

    Source: §I · p. 1

  • Method

    TurboT2VA combines per-modality normalization, shared cross-modal distillation, a progressive dCM→sCM→sCM+DMD curriculum, and architecture-aware inference optimizations.

    Source: §I · pp. 1–2, §III · p. 3, §VII · p. 11

  • Results

    20.1× speedup reduces latency from 50.52s to 2.51s at 512×768, while 54.67× generator-only speedup reduces high-resolution latency from 318.74s to 5.83s on one H20, retaining competitive multimodal quality.

    Source: §I · p. 2, §VII · p. 11

  • Takeaways & Limitations

    Four-step joint generation offers a favorable efficiency–quality trade-off, and combining distillation with systems optimization supports efficient deployment of large-scale T2VA models.

    Source: §VII · p. 11

Abstract

from arXiv · show

Joint text-to-video-audio generation produces synchronized visual and acoustic content, but the long sampling trajectories and heterogeneous multimodal computation of large models make inference prohibitively expensive. We present TurboT2VA, a distillation and inference framework for accelerating a 19B-parameter joint video-audio model. Large-scale T2VA distillation is challenged by modality-imbalanced optimization, the difficulty of continuous-time consistency training at scale, and the quality--diversity trade-off. TurboT2VA addresses these issues with per-modality normalization and a progressive curriculum comprising discrete consistency warm-up, continuous consistency refinement, and joint consistency--distribution matching. The curriculum first establishes a stable, diverse generation trajectory and only then introduces distribution-level refinement. On LTX-2, four-step distillation reduces generator latency from 50.52s to 2.51s at the standard evaluation resolution of 512$\times$768, achieving a 20.1$\times$ speedup while maintaining strong visual quality, audio fidelity, diversity, and video-audio synchronization. We further develop an architecture-aware inference stack that combines guarded W8A8 and fused operators, padded-text compaction, and modality-aware sparse attention while preserving dense cross-modal and text-conditioning paths. Under the high-resolution deployment setting at 1024$\times$1792, the complete stack reduces generator latency from 318.74s to 5.83s on one NVIDIA H20, achieving a 54.67$\times$ generator-only speedup. Inference code and generation demos are available at https://github.com/thu-ml/TurboDiffusion/tree/main/turbot2va.

I. INTRODUCTION

TurboT2VA addresses the high cost and unresolved quality–diversity balance of large-scale joint text-to-video-audio generation through progressive consistency and distribution-matching distillation. A four-step student and architecture-aware inference stack substantially reduce latency while retaining competitive synchronized multimodal quality.

Motivation

Challenges

  • Consistency distillation preserves diversity, whereas distribution matching improves realism but reduces diversity, leaving their balance unresolved in T2VA.Source: §I · p. 1

Approach

  • TurboT2VA extends score-regularized continuous-time consistency distillation to a 19B-parameter joint model with per-modality normalization and progressive training.Source: §I · p. 1

Performance

  • 20.1× speedup reduces latency from 50.52s to 2.51s at 512×768 while maintaining a strong trade-off across visual quality, audio fidelity, diversity, and synchronization.Source: §I · p. 2

Approach

  • TurboT2VA jointly distills video and audio through a shared process, preserving temporal consistency rather than training independent modality branches.Source: §I · p. 2

III. CROSS-MODAL JOINT DISTILLATION

TurboT2VA performs joint consistency distillation on paired video-audio latents using shared conditioning and a shared cross-modal Transformer. Its continuous-time formulation couples modality directions through joint tangent estimation, while per-modality normalization balances their contributions.

Joint formulation

  • TurboT2VA distills paired video-audio latents jointly rather than treating video and audio as independent branches.Source: §III · p. 3
  • Video and audio use independently sampled noises but share the base timestep and text condition, placing both modalities on the same generation trajectory.Source: §III · p. 3

Shared backbone

  • The student predicts both modalities in one forward pass through a shared T2VA Transformer with paired latents, shared conditioning, and timestep embeddings.Source: §III · p. 3

Continuous-time consistency

Cross-modal coupling

Modality balancing

Distillation objective

  • The resulting objective transfers paired video-audio trajectories to the few-step student while preserving motion, acoustic evolution, rhythm structure, and temporal correspondence.Source: A. Joint Continuous-Time Consistency · p. 4

B. Joint Distribution Matching

TurboT2VA introduces DMD after sCM has established a stable joint trajectory, using paired student-generated video-audio samples to improve realism without disrupting cross-modal structure.

Joint Distribution Matching

  • DMD is introduced after sCM stabilizes the joint trajectory to improve few-step perceptual quality.The staged design separates trajectory preservation from distribution-level refinement.Source: B. Joint Distribution Matching · p. 4
  • DMD compares teacher and fake-score predictions on perturbed student-generated samples to form a score-discrepancy signal.The fake-score network is trained on student-generated samples, while teacher predictions use classifier-free guidance.Source: B. Joint Distribution Matching · p. 4
  • DMD uses residual-scale normalization and aggregates the surrogate over all latent elements before applying modality and outer-coefficient scaling.The accumulated signal is retained across the full paired latent while overall scale remains controlled.Source: B. Joint Distribution Matching · p. 4
  • Unlike independent modality DMD, the objective uses paired video-audio samples from the same student condition and timestep.This preserves the cross-modal structure learned through joint sCM.Source: B. Joint Distribution Matching · p. 4

C. Modality-Balanced Joint Optimization

TurboT2VA balances video and audio contributions at the loss level while retaining a single unified generator and separate normalization schemes for sCM and DMD.

Modality-Balanced Joint Optimization

  • Modality-wise balancing prevents video dominance caused by unequal latent dimensionalities, temporal resolutions, and learning speeds.The student remains a single unified T2VA generator rather than separate modality branches.Source: C. Modality-Balanced Joint Optimization · p. 4
  • sCM uses an L2 direction normalizer, whereas DMD uses a teacher-residual scale normalizer before modality-weighted combination.The different normalizers reflect the distinct roles of the two objectives.Source: C. Modality-Balanced Joint Optimization · p. 4
  • The joint objective combines sCM for paired trajectory preservation with DMD for teacher-distribution matching.The coefficients λsCM and λDMD control the balance between trajectory consistency and distribution-level anchoring.Source: C. Modality-Balanced Joint Optimization · p. 4

D. LTX-2 TrigFlow Adaptation

TurboT2VA adapts LTX-2’s rectified-flow backbone to TrigFlow-based sCM through a wrapper that unifies video and audio prediction interfaces without changing the backbone.

LTX-2 TrigFlow Adaptation

  • The wrapper maps TrigFlow latents and timesteps to LTX-2’s rectified-flow interface.This bridges the parameterization used by sCM and the original LTX-2 backbone.Source: D. LTX-2 TrigFlow Adaptation · p. 4
  • The adaptation exposes unified video and audio predictions under one timestep interface for joint sCM and DMD training.It supports JVP computation through timestep embeddings, tokenization, positional embeddings, and the cross-modal Transformer.Source: D. LTX-2 TrigFlow Adaptation · p. 4
  • Direct sCM+DMD training is feasible but yields weaker quality–diversity trade-offs, motivating staged optimization.The observed weakness is especially associated with diversity and video-audio dynamics.Source: §IV · pp. 4–5

A. Empirical Observation: Trainable but Suboptimal

Direct joint sCM+DMD training is trainable and produces reasonable results, but the staged curriculum performs better by ordering trajectory learning before distribution matching.

Empirical Observation: Trainable but Suboptimal

2) Stage 2: sCM Refinement:

Stage 2 uses continuous-time consistency refinement to turn a coarse student denoiser into a trajectory-consistent generator before distribution-level supervision. This preserves trajectory fidelity, diversity, and multimodal temporal correspondence while motivating architecture-aware acceleration for heterogeneous Transformer paths.

Stage 2: sCM Refinement:

C. Fused Multimodal Transformer Operations

The inference stack fuses multimodal Transformer operations, compacts padded text conditioning, and applies guarded acceleration according to architectural role and shape. Its high-resolution evaluation targets deployment bottlenecks while preserving safe fallback behavior and valid conditioning.

C. Fused Multimodal Transformer Operations

  • Fused kernels replace repeated normalization, modulation, gated residual, output-modulation, and rotary-embedding operations in the joint Transformer.Source: C. Fused Multimodal Transformer… · p. 6
  • Padded-text compaction removes only invalid prompt positions during batch-size-one inference, leaving every valid conditioning token unchanged.Source: C. Fused Multimodal Transformer… · p. 6
  • The default high-resolution configuration combines ρℓ=0.3 self-attention sparsity, compatible TileLang W8A8 linear layers, fused kernels, and compacted text context.Source: C. Fused Multimodal Transformer… · pp. 6–7
  • The evaluation model is a 19B-parameter LTX-2 system with 14B video and 5B audio backbones, distilled to preserve multimodal generation properties.Source: A. Experimental Setup · p. 7
  • Training uses 100K text-video-audio samples on 8 H20 GPUs, while standard evaluation uses a fixed 200-prompt split and eight seeds per prompt for diversity.Source: A. Experimental Setup · p. 7
  • High-resolution latency is measured on one NVIDIA H20 at 1024×1792 and 121 frames using generator-only time, excluding loading, decoding, muxing, and I/O.Source: A. Experimental Setup · p. 7

B. Main Results

TurboT2VA compresses the 40-step teacher into four steps while retaining strong multimodal quality, fidelity, diversity, and synchronization. Quantitative and qualitative comparisons show efficient generation with preserved video and audio structure.

B. Main Results

  • 50.52s to 2.51s: four-step TurboT2VA achieves a 20.1× speedup at the 512×768 standard evaluation resolution.Source: 1) Generation Quality · pp. 7–8
  • The four-step student preserves coherent subjects, scene layout, temporal appearance, and audio structure across early, middle, and late examples.Source: Figure 4 · p. 8, 1) Generation Quality · p. 8
  • TurboT2VA maintains comparable overall JavisBench quality and improves several video-audio quality and synchronization metrics despite aggressive step reduction.Source: 1) Generation Quality · p. 8
  • The distilled student maintains comparable VBench video fidelity, with improved aesthetic and imaging quality and strong temporal consistency.Source: 2) Video and Audio Fidelity · p. 8
  • Under matched prompts and seeds, staged training balances sCM-only variation and DMD-only polish, preserving coherent cat–robot interactions and cross-seed variation.Source: Figure 5 · p. 9
  • Four-step inference maintains comparable audio quality and text-audio alignment to the teacher while substantially reducing sampling steps.Source: 2) Video and Audio Fidelity · p. 9

3) Diversity Evaluation:

TurboT2VA’s staged consistency–distribution matching approach improves the quality–diversity balance while an architecture-aware inference stack substantially reduces high-resolution generator latency. The evaluation also examines diversity methodology, sparse-attention trade-offs, and curriculum behavior.

Diversity Evaluation

  • The diversity score averages within-prompt pairwise distances from modality-specific ImageBind embeddings across 8 samples generated with different random seeds.The protocol produces 28 pairs per prompt and 5,600 pairs per modality over 200 prompts.Source: 3) Diversity Evaluation · p. 9
  • DMD-only attains competitive quality but the lowest diversity, whereas sCM-only preserves the strongest diversity at substantially lower quality.The staged sCM+DMD approach provides a more favorable quality–diversity balance.Source: 3) Diversity Evaluation · p. 9
  • sCM-only changes subjects and layouts substantially but less reliably preserves the requested interaction, while DMD-only produces polished yet similar centered compositions.The staged variant preserves the interaction while retaining visible cross-seed changes in appearance, robot design, and scene layout.Source: 3) Diversity Evaluation · p. 9

Inference Efficiency

  • 5.83s generator latency at 1024×1792 yields a 54.67× overall speedup over the dense 40-step teacher on one NVIDIA H20.The complete stack combines W8A8, fused operations, four-step distillation, modality-aware sparse attention, and padded-text compaction; the unoptimized student takes 16.50s.Source: 4) High-Resolution Inference… · pp. 9–10, Figure 6 · p. 9
  • Sparse attention provides the largest single-component latency gain, while the complete inference stack reduces high-resolution generator latency to 5.83s.Fused operations and W8A8 reduce dense four-step latency from 16.50s to 13.77s and 14.73s, respectively, and jointly to 12.16s.Source: 4) High-Resolution Inference… · p. 10
  • Reducing SageSLA retention from 0.5 to 0.2 lowers latency from 6.44s to 5.57s while quality and alignment metrics remain close to the dense reference.The default ρ = 0.3 balances latency and cross-modal alignment, achieving the best CAVP score and lower desynchronization error than ρ = 0.2.Source: 4) High-Resolution Inference… · p. 10

C. Ablation Study

Ablations support progressive warm-up before joint distribution refinement and show that additional sampling steps generally improve generation quality, with four-step inference outperforming two-step inference across most metrics.

Effect of Progressive Warm-up

  • The staged curriculum outperforms direct sCM+DMD optimization after joint training begins and retains a clear advantage through 7,500 global steps.dCM warm-up establishes coarse denoising, sCM refinement aligns with the continuous teacher trajectory, and DMD is introduced afterward.Source: 1) Effect of Progressive Warm-up · p. 10, §VII · p. 11
  • After the shared dCM+sCM prefix, staged sCM+DMD reaches the strongest late-training quality, peaking at 0.1963 at 7,000 steps.DMD refinement improves quickly but plateaus below the joint objective, while sCM-only remains stable at substantially lower Javis score.Source: 2) Effect of Joint sCM+DMD Optimization · p. 10

Sampling-Step Ablation

  • Generation quality generally improves as TurboT2VA uses more sampling steps.The sampling-step ablation evaluates different step counts for the distilled model.Source: D. Effect of Sampling Steps · p. 10, Table VII · p. 11
  • Four-step inference provides further gains across most metrics compared with two-step inference, which achieves reasonable performance.This supports four steps as the stronger quality setting among the reported low-step configurations.Source: D. Effect of Sampling Steps · pp. 10–11

Overall Performance

  • At 512×768, four-step distillation reduces latency from 50.52s to 2.51s, achieving a 20.1× speedup while retaining competitive multimodal quality and diversity.The reported evaluation covers visual quality, audio fidelity, cross-modal consistency, and sample diversity.Source: §VII · p. 11
  • At 1024×1792, the complete architecture-aware stack reduces generator latency from 318.74s to 5.83s on one NVIDIA H20, achieving a 54.67× generator-only speedup.The stack combines model-level distillation with W8A8, fused operations, padded-text compaction, and modality-aware sparse attention while preserving dense conditioning paths.Source: §VII · p. 11

Related papers

Loading 2608.24674v1…