Source-linked AI summary
Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training
Shuyuan Tu, Qi Tian, Yinming Huang, Yue Wu, Xintong Han, Kaihang Pan, Weijie Kong, Jiangfeng Xiong, Jian-Wei Zhang, Zuxuan Wu, Yu-Gang Jiang
TL;DR
Native 2K joint video-audio training offers richer detail and sharper motion, but full attention is costly and redundant tokens can dilute informative attention. Prism adapts zone-level block shapes and per-query sparsity to visual variation and audio-visual coupling. It reports 2.5× faster training than full attention with better quality.
Problem
At 2K, full attention has quadratic cost and increasingly redundant tokens can dilute attention to informative content, while existing sparse methods overlook localized audio-visual coupling.
Source: §1 · p. 1Method
Prism assigns macro-zone-specific block shapes using directional video variation and audio-to-video cross-attention norms, then uses hybrid Top-k/Top-p selection to set per-query sparsity.
Source: §3.1 · p. 4, §3.2 · p. 6Results
Prism trains 2.5× faster than full attention while achieving better quality.
Source: §5 · p. 10Takeaways & Limitations
Prism demonstrates native 2K joint video-audio generation with stable content, including complex human–object interactions, compared with LTX-2.3 with super-resolution and MOVA using full attention.
Source: §1 · p. 3With a fixed eight-frame temporal macro-zone span, rapid camera-focus transitions can produce blurry or distorted transition frames.
Source: §A.19 · p. 31
Abstract
from arXiv · showhide
Natively training joint video-audio generation models at higher resolutions empowers them to learn richer visual details and sharper motion dynamics. However, full attention incurs quadratic cost and, as resolution increases, spreads attention over increasingly redundant tokens, diluting learning signals for informative content and disrupting pretrained priors. Existing sparse attention methods either target training-free acceleration or overlook the unique structure of joint video-audio data, where cross-modal interactions are inherently concentrated around sound-producing regions. To address this, we propose Prism, a dynamic sparse attention framework for natively training joint video-audio generation models at 2K. In particular, Prism organizes the token sequence into spatiotemporal macro-zones, enabling the attention structure to adapt to local content. For each zone, it estimates local information structure via video feature variance along the channel and feature norms from the audio-to-video cross-attention, jointly capturing how visual content varies directionally and how strongly audio influences each visual region. Based on these signals, Prism dynamically assigns a tailored block shape to each zone, applying finer partitioning along axes of rapid visual content variation and strong audio-visual coupling. This encourages tokens within each block to remain semantically coherent, allowing block-level features to capture both visual content and joint video-audio interaction patterns. Prism further adopts a hybrid block selection strategy to dynamically determine per-query sparsity. Experiments show that Prism achieves 2.5$\times$ training speedup compared to full attention, while surpassing it in generation quality.
1 Introduction
Prism targets native 2K joint video-audio training, where full attention is costly and redundant tokens can dilute attention to informative content. It adapts block shapes and per-query sparsity to local visual variation and audio-visual coupling.
1 Introduction
- At 2K, full attention incurs quadratic cost and can dilute attention to informative content as redundant tokens increasingly dominate the sequence.The paper notes that this can degrade native high-resolution learning and disrupt pretrained priors.Source: §1 · p. 1
- Existing sparse methods largely target training-free acceleration or lower-resolution video, and their visual-only, uniform criteria overlook localized audio-visual coupling.The paper identifies sound-producing regions as important sites of audio-visual synchronization.Source: §1 · p. 2
- Prism uses macro-zones to assign locally tailored block shapes, guided by visual variation and audio-to-video cross-attention norms.Finer partitions are applied along axes of rapid visual change and strong audio-visual coupling, supporting semantically coherent blocks.Source: §1 · p. 2
- Hybrid Top-k/Top-p selection dynamically sets per-query sparsity, extending coverage for queries whose attention is broadly dispersed.Top-k guarantees minimum coverage, while Top-p adapts the attended set to the query.Source: §1 · p. 3
- Prism trains 2.5× faster than full attention while achieving better quality.The introduction also reports stable native 2K video-audio generation compared with LTX-2.3 with super-resolution and MOVA using full attention.Source: §1 · p. 3
2 Related Work
Prior work has advanced joint video-audio generation and high-resolution video modeling, but the cited approaches do not address native 2K joint training with content-adaptive cross-modal attention. The related-work discussion positions Prism against these gaps.
Joint Video-Audio Generation
Efficient Attention
- Most video-generation sparse attention methods target inference at 480P or 720P, while fixed block shapes or layer-index partitions do not adapt to joint video-audio structure.The discussion cites VMoBA [78], SpargeAttention2 [92], HunyuanVideo1.5 [77], LongCat-Video [58], VSA, and LIVEditor [53] as distinct approaches.Source: §2 · p. 3
Native High-Resolution Training
3 Method
Prism replaces video self-attention in a MOVA DiT with dynamic sparse attention for native 2K joint video-audio training. Its zone-level block shapes and per-query selection adapt computation to local visual and audio-visual structure.
3 Method
- Prism replaces video self-attention in the MOVA [59] DiT with sparse attention, organizing tokens into macro-zones before block-shape assignment and hybrid block selection.It applies sparsity to video self-attention because that branch dominates the cost at 2K.Source: §3 · p. 3, §3.3 · p. 6
3.1 Dynamic Block Shape
- Each macro-zone receives a block shape guided by directional video-feature variation and audio-to-video cross-attention norms, with finer partitions along axes of stronger variation or coupling.The variation indicator is estimated independently per attention head and layer; zones share one shape, while downstream block selection captures cross-zone semantics.Source: §3.1 · pp. 4, 6
3.2 Dynamic Per-Query Sparsity
- Hybrid Top-k/Top-p selection chooses attended key/value blocks per query, expanding beyond Top-k when attention is dispersed.Sharp queries select k blocks; flatter queries may select more through Top-p.Source: §3.2 · p. 6
3.3 Training
- Training uses video/audio flow-matching loss while applying sparse attention only to video self-attention.The paper identifies video self-attention as the dominant 2K cost; the audio branch and cross-attention use shorter sequences.Source: §3.3 · p. 6
4 Experiment
The experiments evaluate Prism’s native 2K training setup, comparisons with other methods, attention ablations, and applications. They report advantages over full attention and examine how design choices affect quality, speed, and synchronization.
4.1 Implementation Details
- Training uses 100k 2K clips; evaluation includes 300 unseen 2K videos, with Prism initialized from MOVA and trained for five epochs.The reported settings include learning rate 1e−5, Top-k 95%, Top-p 0.2, τ128=0.5, and τ256=0.25.Source: §4.1 · p. 6
4.2 Comparison with State-of-the-Art Methods
- On 2K-Bench, MOVA, Ovi, and MagiHuman underperform Prism; qualitative comparisons report competitor flickering, visual melting, and human distortions while Prism remains structurally coherent.The comparison also reports that Prism outperforms competitors on 720p VABench and MOVA-Bench, though by a smaller margin.Source: §4.2 · p. 7
- Commercial models surpass Prism in visual realism and audio aesthetics, but Prism narrows the open-source gap in video-audio stability under large motions.The paper notes that commercial models use larger parameter counts and datasets.Source: §4.2 · pp. 7–8
4.3 Ablation Study
- At 2K, Full Attn performs poorly and BSA performs worst; Prism improves over SpargeAttn2 by 46% in MotionQ and 30% in DeSync.The ablation attributes Full Attn’s weakness to redundant tokens, and BSA’s to fixed blocks mixing semantically dissimilar tokens; training-free methods also perform below Full Attn.Source: §4.3 · p. 8
- Training-free 2K inference and cascaded methods have stated limitations, while T3’s native single-pass training uses content-agnostic uniform-stride sampling.Prism instead performs native 2K training with content-adaptive dynamic sparse attention in a single pass.Source: §4.3 · p. 8
- Fixed and random block shapes perform worst, while removing video guidance causes the largest visual drop and removing audio guidance degrades cpCER and DeSync.The three-tier allocation surpasses the single finest tier in both quality and speed; key features outperform query features for variance estimation, and spatial and temporal variance both matter.Source: §4.3 · pp. 8–9
- Prism is the only method reported to improve all metrics from 480P to 2K; other methods face redundancy, content-agnostic block shapes, or limited audio-aware coupling.The ablation says removing video guidance stalls visual-metric improvement beyond 1080P, while removing audio guidance prevents steady synchronization improvement.Source: §4.3 · p. 9
- Top-k(95%)+Top-p(0.2) gives the best quality-speed trade-off, and results also report backbone robustness and the lowest, most stable training loss for Prism.The visualization reports that C64 concentrates on dynamic regions and C256 covers static backgrounds, with partition granularity changing by local content.Source: §4.3 · p. 9
5 Conclusion
Prism is a dynamic sparse attention framework for native 2K joint video-audio generation. It adapts block shapes and per-query sparsity to local visual variation and audio-visual coupling, achieving faster training than full attention while surpassing it in generation quality.
5 Conclusion
- Prism organizes tokens into spatiotemporal macro-zones and assigns tailored 3D block shapes using video channel-wise variance and audio-to-video cross-attention norms.The design aims to keep blocks semantically coherent and represent both visual variation and audio-visual coupling.Source: §5 · p. 10
- Prism uses a hybrid strategy to determine per-query sparsity and concentrate computation on informative interactions.Source: §5 · p. 10
- 2.5× training speedup over full attention at 2K, while surpassing it in generation quality.Source: §5 · p. 10
A Appendix
The appendix includes quantitative comparison tables for VABench and MOVA-Bench, a native-2K methods comparison, and a dual-branch DiT architecture figure.
A Appendix
- Tables 3 and 4 present quantitative comparisons with previous open-source methods on VABench and MOVA-Bench, respectively.Both captions specify use of the methods’ original evaluation metrics.Source: Table 3 · p. 15, Table 4 · p. 15
- Table 5 compares native 2K training and inference methods.Training Time is per-step training time; GPU Mem is per-GPU memory during 4-GPU parallel inference.Source: Table 5 · p. 15
- Figure 4 depicts the architecture of Prism’s dual-branch DiT.Source: Figure 4 · p. 16
A.1 Preliminaries
The preliminaries describe a straight-line corruption from clean data to noise and a velocity-prediction objective. The predicted velocity is used to transport noise back toward the data manifold during inference.
A.1 Preliminaries
- A clean latent and a noise sample define a straight-line interpolation parameterized by timestep t from 0 to 1.Source: §A.1 · p. 16
- The velocity network estimates the direction from clean latent to noise at each noise level, and inference uses it to transport pure noise back to the data manifold.Source: §A.1 · p. 16
- Training minimizes mean squared error between predicted and ground-truth velocity.Source: §A.1 · p. 16
A.2 Evaluation Metrics
The evaluation covers video and audio quality, synchronization, high-resolution visual detail, and motion quality. The appendix also identifies ablations of DiT backbones and sparsity strategies.
A.2 Evaluation Metrics
- The benchmark assessment spans video, audio, synchronization, high-resolution visual details, and motion quality.Source: §A.2 · p. 16
- The evaluation uses Temporal Flickering and dynamic degree for video quality, aesthetic predictor v2-5 for AQ, and DINOv3 representations for identity consistency.Identity consistency compares the first synthesized frame with the remaining frames.Source: §A.2 · pp. 16–17
- The appendix includes ablations of DiT backbones and sparsity strategies, including per-query top-k sparsity and cumulative-attention top-p selection.Top-p selects the smallest set of key blocks whose cumulative attention weight exceeds the specified threshold.Source: Table 6 · p. 17, Table 7 · p. 17
- Audio quality uses AudioBox for production quality and content usefulness, while cpCER measures preservation of speaker identities and dialogue content.cpCER is computed from diarized, transcribed speech compared with ground-truth transcripts.Source: §A.2 · p. 17
- Synchronization is measured with Sync-C and Sync-D for lip-audio alignment and Synchformer for temporal misalignment between video and audio.Source: §A.2 · p. 17
- MUSIQ and MANIQA assess perceptual quality and detail fidelity in individual frames, with scores averaged across sampled frames.Source: §A.2 · p. 17
- Gemini-3.1 evaluates overall motion quality through MotionQ because conventional automatic metrics struggle to assess video motion accurately.Source: §A.2 · p. 17
A.3 Dataset Details
The dataset section describes a filtered collection of 100k 2K training clips and a 300-video unseen 2K test benchmark. The benchmark emphasizes fine detail and challenging motion, with checks for separation from training data.
A.3 Dataset Details
- The 100k-clip training set is filtered for native resolution, visual quality and detail, motion, audio quality, and speech lip-sync.Clips must have a shorter side of at least 1440 pixels; filters also remove upscaled, low-aesthetic, low-detail, near-static, low-audio-quality, or low-sync clips.Source: §A.3 · pp. 17–18
- Over 75% of the training set contains significant motion dynamics, including interactions, musical performance, and expressive speaker content.Source: §A.3 · p. 18
- 2K-Bench contains 300 unseen native-2K videos and targets fine visual details, challenging motion, human-object interaction, and audio-visual correspondence.Its reference imagery is natively captured at 2K, unlike benchmarks with lower-resolution references.Source: §A.3 · p. 18
- 2K-Bench balances six audiovisual categories at roughly 16–17% each and is released for non-commercial research use only.Source: §A.3 · pp. 18–19
- Three checks support 2K-Bench’s separation from training data: URL deduplication, perceptual-hash screening, and an I3D-feature FVD comparison.The benchmark-to-training FVD is reported as over 5.5× the FVD between random equal halves of the training set.Source: §A.3 · p. 19
A.4 Intra-Block Information Loss and Proof of Lagrange Multipliers
Prism models intra-block information loss as independent per-axis costs that combine block geometry with directional content variation. It uses this objective to choose block shapes under a fixed token budget, while accounting for approximation and discrete-gradient constraints.
A.4 Intra-Block Information Loss and Proof of Lagrange Multipliers
- The intra-block loss sums independent per-axis terms, weighting positional spread within a block by content variation along that axis.Mean-pooling replaces tokens with a block representative; longer edges increase positional deviation, while larger variation indicators imply greater feature deviation.Source: §A.4 · p. 19
- The loss derivation assumes approximately linear feature variation within each block, an assumption that may fail at sharp object boundaries.Small macro-zones and anisotropic partitioning are described as limiting the tokens affected by such discontinuities; quantizing among 16 candidate shapes also makes selection robust to moderate estimation errors.Source: §A.4 · pp. 19–20
- Prism combines video variation and audio-to-video coupling in its axis-wise variation indicators to guide macro-zone block-shape selection.The algorithm uses these indicators to estimate information density and quantize a shape from a discrete set of candidate block sizes.Source: §A.4 · p. 20
- After partitioning, mean-pooled query and key representatives score block relevance, and hybrid Top-k/Top-p selection chooses key blocks for each query block.The selected indices drive sparse attention, whose online softmax processes selected blocks and scatters outputs back to token order.Source: §A.4 · p. 20
- Under a fixed block-volume constraint, the Lagrange-derived optimum assigns the shortest edge to the axis with the largest variation indicator.The derivation minimizes intra-block loss subject to the product of the three block edges equaling the zone's token budget.Source: §A.4 · pp. 20–21
- Shape assignment and hybrid block selection are discrete, so their driving statistics are detached and gradients flow only through attention over selected blocks.The passage characterizes this as the regime used by prior trainable block-sparse attention methods.Source: §A.4 · p. 21
A.5 Dynamic Block Sparse Attention Details
Dynamic block-sparse attention assigns shapes per macro-zone, rearranges tokens for blockwise processing, and selects relevant key blocks using pooled representatives. Its kernel computes attention only over selected blocks and restores the original token order.
A.5 Dynamic Block Sparse Attention Details
- Per attention head and layer, Prism combines video channel-wise variance and audio-to-video cross-attention norms to estimate variation and assign each macro-zone a block shape.Zones use shapes with block sizes of 64, 128, or 256 tokens, mapped to contiguous 64-token tiles for the Triton kernel.Source: §A.5 · p. 21
- Prism rearranges tokens into block-contiguous layout, then mean-pools query and key blocks to score relevance and select key blocks with hybrid Top-k/Top-p.Selected block indices are stored for the downstream sparse-attention kernel.Source: §A.5 · p. 21
- For each query block, the kernel loads full query tokens and iterates over selected key blocks with online softmax, then scatters outputs back to the original token order.The kernel skips non-selected blocks, so runtime scales with the average number of selected blocks rather than the total number of blocks.Source: §A.5 · p. 21
A.6 More Implementation Details
Prism is trained by replacing video self-attention while retaining the pretrained backbone's other components. Audio and cross-modal attention remain dense, and the video sparse-attention path uses custom Triton kernels.
A.6 More Implementation Details
- Prism initializes from MOVA-720p and replaces only video self-attention, allowing the sparse attention to inherit the model's full-attention weights.Only attention modules are trainable; other parameters remain frozen during training.Source: §A.6 · p. 22
- Audio self-attention and cross-modal attention remain unmodified because audio sequences are short and prior work reports quality degradation from sparse cross-modal attention.Audio self-attention typically has under 1K tokens after VAE compression; audio-to-video guidance uses norms cached from the preceding layer, with no such norms in the first layer.Source: §A.6 · p. 22
- The video sparse-attention kernel is implemented in Triton with 64-token query and key tiles and a head dimension of 128.Separate lightweight Triton kernels perform shape assignment and token rearrangement before attention; the inverse permutation restores output order.Source: §A.6 · p. 22
A.7 More Comparison Results
The paper presents additional comparisons between Prism and state-of-the-art open-source models in Figures 7–10, and reports that Prism performs best.
A.7 More Comparison Results
- Prism is reported to have the best performance among the compared state-of-the-art open-source models.The passage identifies Figures 7–10 as showing additional comparison results.Source: §A.7 · p. 22
- Figures 7–10 contain additional comparisons of Prism with state-of-the-art open-source models.The passage does not specify metrics or individual comparison outcomes for these figures.Source: §A.7 · p. 22
- The reported best-performance conclusion applies to the set of additional open-source model comparisons described for Figures 7–10.No numerical values or metric labels are given in the passage.Source: §A.7 · p. 22
A.8 Commercial Model Comparison Results
This section compares Prism with commercial models across scaling, training, visualization, efficiency, and generation, while documenting user-study and dataset safeguards, misuse-reduction measures, and limitations including camera-transition failures and scaling limits.
A.8 Commercial Model Comparison Results
- All commercial models outperform Prism across metrics, while Prism beats LTX-2.3 at matched backbone scale and narrows the visual-detail and motion-quality gap more than the audio-quality gap.MiniMax-H3 surpasses Prism despite using an upsampling pipeline, but it has a 30B backbone and far more data; Prism is the only open-source method in this comparison that natively trains at 2K with dynamic sparse attention.Source: §A.8 · p. 22, Table 8 · p. 23
A.9 More Ablation on Dynamic Block Shape
- Dynamic block shapes outperform content-agnostic allocation: video channel-wise variance guides visual quality, while audio coupling guidance steers fine blocks toward sound-producing regions.Spatial and temporal variance are both needed to capture information density; channel-wise variance is the primary visual driver, while removing audio guidance degrades cross-modal metrics. The α=1/2 mapping balances responsiveness to directional variation against noisy estimates; more aggressive mappings destabilize shape assignment.Source: §A.9 · pp. 23–24
A.11 Native 2K Training Loss
- Prism achieves the lowest and most stable video and audio losses, whereas fixed-shape sparse methods reduce video loss but retain upward audio-loss trends.The ablations show complementary roles: video guidance primarily supports visual modeling, audio-to-video norm guidance supports cross-modal stability, and isotropic shapes produce the highest, most unstable losses on both modalities.Source: §A.11 · p. 27
A.12 Attention Visualization Results
- Prism assigns fine blocks to dynamic, sound-producing regions and coarse blocks to repetitive backgrounds, with block shapes adapting to local directional variation.Speaking faces receive finer temporal shapes, while static faces and homogeneous backgrounds can use coarser temporal or spatial grouping; block size captures information density and shape captures directional anisotropy.Source: §A.12 · p. 28
A.13 Speed and GPU Resource Comparison
- Over 3× faster inference and 47% less GPU memory than Full Attn accompany gains of 45% AQ, 68% MotionQ, and 40% DeSync.Against LTX-2.3, Prism achieves comparable speed with half the GPU memory and outperforms it across quality metrics; shape computation, token rearrangement, and block selection together account for 13.6% of Prism’s inference latency.Source: §A.13 · p. 29, Table 13 · p. 29
A.14 Multi-Speaker Results; A.15 Complex Scene Results
- Prism handles multi-speaker communication and complex 2K scenes while preserving sharp visual details, stable body structures under large motion, and precise audio-visual synchronization.The complex scenes test human-object contact, rapid movement, fine hand and finger articulation, multi-character interaction, and physically plausible environmental sounds.Source: §A.14 · p. 29, §A.15 · pp. 29–30
A.16 User Study Details
- The user study used a two-alternative forced-choice evaluation with 200 adults, comparing Prism against competitors on six criteria across 30 randomly selected videos.Participants assessed all 30 cases, with randomized left-right ordering; the videos were drawn from a dataset of 300 unseen 2K videos.Source: §A.16 · p. 30
- The user study received an exemption from full IRB review, and participants voluntarily compared pre-generated videos, took about 25 minutes, and received $5.No personally identifiable information was collected or stored beyond aggregate statistics, and participants were informed of the study purpose and their right to withdraw.Source: §A.16 · p. 30
A.17 Human Subjects Data Concern
- Training and testing videos came from public platforms, and videos were retained only after uploaders provided consent and, when needed, authorization for depicted individuals’ likenesses.Videos without the required verification were removed.Source: §A.17 · p. 30
A.18 Ethics Concerns
- The authors warn that high-fidelity 2K outputs could enable impersonation or misleading audiovisual content and recommend watermarking, detection and moderation, authenticated access with logging, and consent-based usage rules.They note that richer spatial detail and sharper motion can make generated content harder to distinguish from authentic recordings.Source: §A.18 · pp. 30–31
A.19 Limitations and Future Work
- Prism can blur or distort bodies during rapid camera-focus transitions because a fixed 8-frame temporal zone may combine heterogeneous scenes.The authors propose adaptive temporal boundaries that split at detected camera transitions; training beyond 2K and with larger DiT backbones remains future work, and 4K would quadruple the token count.Source: §A.19 · p. 31, Figure 31 · p. 48