Source-linked AI summary
All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation
Ohad Rahamim, Dvir Samuel, Idan Schwartz, Gal Chechik
TL;DR
Joint multimodal generators can be architecturally bidirectional while learning weaker modality-to-video correspondence. RecCAR aligns that weaker direction with the established video-to-modality correspondence, improving cross-modal consistency across video–motion and video–audio generation while preserving generation quality.
Problem
Joint multimodal generators permit bidirectional attention, but their pathways can develop asymmetric cross-modal correspondences, leaving modality-to-video correspondence substantially weaker.
Source: §1 · p. 2, §3.3 · p. 4Method
Results
Takeaways & Limitations
Bidirectional connectivity alone does not ensure bidirectional information flow in joint multimodal generators.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Video is a rich representation of a physical event, capturing appearance, geometry, motion, and temporal evolution. Other modalities, such as 3D body motion or audio, encode narrower aspects of the same event. We find that joint multimodal diffusion transformers exhibit a corresponding asymmetry in cross-modal correspondence: companion modalities develop strong correspondences to video, but the reciprocal correspondences through which they constrain video remain substantially weaker. We express both directions as comparable correspondence distributions over video tokens and define their disagreement as the reciprocal correspondence gap. We introduce RecCAR, standing for Reciprocal Cross-modal Attention Regularization, a KL regularizer that uses the well-established video-to-modality correspondence as a fixed reference and aligns the weaker modality-to-video correspondence toward it. Across joint video-motion and video-audio generation, RecCAR improves the Human Anatomy score from 0.69 to 0.75 and reduces audio-video desynchronization from 0.804 to 0.752, while improving overall generation
1. Introduction
Joint multimodal generators can be bidirectional in architecture yet asymmetric in cross-modal information flow. RecCAR aligns the weaker attention pathway to the stronger one, improving consistency in video–motion and video–audio generation.
1. Introduction
- Despite bidirectional cross-attention, pretrained joint generators can learn much stronger correspondences in one direction, leaving reciprocal influence weak and limiting mutual correction of inconsistencies.The standard generative objective does not directly constrain how much useful information either direction carries.Source: §1 · p. 2
- The stronger correspondence supplies an internal target for its reciprocal pathway, requiring no external reference or additional annotations; the approach is presented as agnostic to the companion modality.The paper evaluates it with structured 3D body motion and with audio carrying different temporal and semantic information.Source: §1 · p. 2
- RecCAR uses a stop-gradient KL objective to align the weaker cross-attention direction to the stronger direction as a fixed reference, using existing attention maps and lightweight LoRA fine-tuning.It adds no inference-time component and preserves the pretrained generator’s capabilities.Source: §1 · p. 2
- RecCAR raises VBench Human Anatomy from 0.69 to 0.75 on EchoMotion [48] and lowers T2AV-Compass absolute audio–video desynchronization from 0.804 to 0.752 on LTX-2 [15] [3].It also improves synchronization on AVGen-Bench [59].Source: §1 · p. 2
- Motion dynamics and visual quality are preserved, standard fine-tuning on the same data does not obtain the motion gains, and audio/video quality and semantic alignment are preserved in video–audio generation.The video–audio results also report improved synchronization on AVGen-Bench [59].Source: §1 · p. 2
2. Related Work
Prior work establishes modality imbalance and uses attention maps as optimization targets; joint diffusion enables modalities to interact, but leaves reciprocal cross-modal correspondences unconstrained. RecCAR addresses this gap by aligning the weaker modality-to-video pathway to the video-to-modality correspondence.
2.1. Modality Imbalance
- Prior work finds modalities can learn at different rates, allowing one to dominate [12] [35]; Gat et al. [13] regularize classifier preferences, while Perceptual Score measures model reliance.This paper studies a related asymmetry in joint generation: video-to-modality correspondence can be better established than the reciprocal modality-to-video correspondence.Source: §2.1 · p. 2
2.2. Attention Optimization
2.3. Joint Multi-Modal Diffusion
- Audio-video generation moved from directional approaches to joint denoising [37]; recent video-motion methods couple motion and video, enabling reciprocal interaction but generally leaving correspondence between directions unconstrained [6] [24] [25] [38] [48] [56].RecCAR regularizes these reciprocal correspondences, using the well-established video-to-modality correspondence as an internal reference for the weaker pathway back into video.Source: §2.3 · p. 3
3. Method
The method addresses asymmetric cross-modal correspondence in joint video–motion and video–audio generation by aligning the weaker modality-to-video pathway with the established video-to-modality pathway. RecCAR uses the pretrained generator’s cross-modal structure without additional correspondence supervision or an auxiliary model.
3. Method
- The method first formalizes joint generation and attention structure, then compares reciprocal correspondences and uses the better-established video-to-modality direction to strengthen the return pathway into video.For parameter-efficient adaptation, the pretrained backbone is frozen while LoRA parameters on attention projections are optimized.Source: §3 · p. 3, §3.3 · p. 4
3.1. Preliminaries: Joint Multimodal Generation
- Joint generators produce video and a companion modality from one text prompt within the same denoising or flow-matching process, with streams exchanging information in both directions.Video-to-modality updates the companion stream, while modality-to-video updates video; the pathways need not develop equally strong correspondences.Source: §3.1 · p. 3
3.2. Reciprocal Cross-Modal Correspondences
- Both directions are represented as distributions over video tokens, making their disagreement measurable as a reciprocal correspondence gap.The modality-to-video distribution is obtained by renormalizing the same attention logits over video tokens rather than using native forward-pass attention.Source: §3.2 · p. 4, §3.3 · p. 4
3.3. RecCAR: Reciprocal Cross-modal Attention Regularization
- RecCAR fixes the established video-to-modality correspondence as a target and regularizes the weaker modality-to-video pathway toward it, strengthening how the companion modality constrains video.It uses a stop-gradient target and applies the same formulation to video–motion and video–audio generation without extra correspondence supervision or an auxiliary model.Source: §3.3 · p. 4
4. Generating Video and Human Motion
This section applies RecCAR to joint video and 3D human-motion generation using EchoMotion, with curated video–motion training pairs and a separate test benchmark. On VBench, RecCAR improves anatomical correctness while maintaining dynamic degree comparable to EchoMotion.
4.3. Results
- RecCAR improves human anatomy and other VBench metrics over EchoMotion, while maintaining comparable dynamic degree; the table reports gains across most metrics against EchoMotion, CoMoVi, and FlowMo.The authors identify human anatomy as the most pronounced improvement and attribute the gains to more accurate, coherent motion rather than slower or less challenging motion.Source: Table 1 · p. 6, §4.3 · p. 7
4.1. Training data and procedure
- Filtering video–motion candidates at MPJPE and NMPJPE below 0.15 yielded 4,292 training pairs, using CameraHMR-estimated motion as a more accurate estimate of motion in the video.The process began with 1,500 human prompts and about 18,000 candidate generations; EchoMotion was fine-tuned with rank-128 LoRA for 10 epochs using RecCAR with λRecCAR = 0.01.Source: §4.1 · pp. 5–6
4.2. Evaluation Setup
- Evaluation used 1,000 diverse VidProM prompts that did not overlap the training set to compare RecCAR with EchoMotion, CoMoVi, and FlowMo.Source: §4.2 · p. 7
5. Generating Video and Audio
This section applies RecCAR to audio–video generation and reports improved synchronization across two benchmarks while preserving overall generation quality and semantic consistency.
5.1. Training data
- Audio–video training uses a random subset of about 4,300 VGGSound clips, with rank-128 LoRA on cross-attention weights for 10 epochs at RecCAR loss weight 0.01.The clips are used without additional filtering or preprocessing, and other model parameters remain frozen.Source: §5.1 · p. 7
5.3. Results
- Across AVGen-Bench, RecCAR reduces DeSync for both LTX-2 and JavisDiT++, while preserving or improving CLAP and AV-CLIP alignment.ITS yields only a marginal DeSync reduction for JavisDiT++ and none for LTX-2.Source: Table 3 · p. 7, §5.3 · p. 8
5.2. Evaluation Setup
- Evaluation uses 500-prompt T2AV-Compass and 235-prompt AVGen-Bench, comparing RecCAR with LTX-2, JavisDiT++, UniAVGen, and ITS variants.The benchmarks assess synchronization and semantic alignment, alongside audio realism, video quality, and speech quality.Source: §5.2 · pp. 7–8
5.3. Results
- For a trotting horse, RecCAR better synchronizes waveform hoofbeat transients with visible hoof contacts than baseline LTX-2.The section reports that synchronization gains across both benchmarks preserve overall generation quality and semantic consistency.Source: §5.3 · p. 8
- On T2AV-Compass, RecCAR achieves the best absolute and predicted AV Desync scores with LTX-2 and improves JavisDiT++ synchronization; LTX-2 audio realism also improves.Video quality and audio–video alignment remain unchanged or improve slightly for LTX-2.Source: §5.3 · p. 8
6. Ablation Study
The ablations attribute improved video–motion consistency and audio–video synchronization to RecCAR rather than standard fine-tuning on the same data. RecCAR preserves other motion qualities and audio–video semantic alignment.
Video–Motion
- Standard fine-tuning alone worsens N-MPJPE, MPJPE, and Human Anatomy, so curated-data training does not explain RecCAR’s video–motion gains.The comparison evaluates Human Anatomy on a subset of 50 prompts.Source: §6 · p. 8, Table 4 · p. 9
- RecCAR improves N-MPJPE, MPJPE, and Human Anatomy over original EchoMotion [48], while Dynamic Degree stays unchanged and Motion Smoothness and Aesthetic Quality are preserved or slightly improved.The reported gains are on metrics reflecting video–motion consistency and do not come from simplifying the generated motion.Source: §6 · p. 8, Table 4 · p. 9
Video–Audio
- RecCAR consistently achieves the lowest DeSync for LTX-2 [15] and JavisDiT++ [26], while standard fine-tuning has little synchronization effect.The controlled comparison uses prompts from AVGen-Bench [59].Source: §6 · p. 8, Table 5 · p. 9
- CLAP [11] and AV-CLIP [21] remain unchanged as RecCAR improves temporal coordination, showing the alignment principle generalizes across video–motion and video–audio generation.The synchronization gain does not come at the expense of semantic correspondence between modalities.Source: §6 · p. 9
7. Conclusion
Joint multimodal generators can be architecturally bidirectional while information flow remains asymmetric, with strong video-to-modality correspondence but weak reciprocal correspondence. RecCAR is a lightweight KL regularizer that aligns the weaker direction and improves cross-modal consistency while preserving generation quality.
7. Conclusion
- RecCAR improves cross-modal consistency while preserving generation quality across video–motion and video–audio generation.The conclusion reports these gains without providing quantitative results.Source: §7 · p. 9
- RecCAR uses a lightweight KL regularizer to align weaker modality-to-video attention with the established video-to-modality direction.It requires no external supervision or architectural changes beyond LoRA adaptation.Source: §7 · p. 9
- Pretrained joint multimodal generators develop strong video-to-modality correspondence while the reciprocal pathway remains weak, despite architectural bidirectionality.The conclusion notes that bidirectional connectivity alone does not ensure bidirectional information flow.Source: §7 · p. 9
8. Appendix
The appendix calibrates motion-pair filtering against independent anatomy ratings and validates RecCAR’s intended effect: it shifts cross-modal attention toward the reciprocal path and reduces the correspondence gap for most test prompts.
8.1. Anatomy-Based Calibration of the Motion Filtering Criterion
- Across 25 videos, lower MPJPE showed a strong negative monotonic relationship with independent Gemini anatomy ratings, motivating an empirical MPJPE threshold of 0.15.The threshold separates a substantial portion of higher-rated samples from those with larger motion errors; it is a filtering criterion, not a universal anatomical boundary.Source: §8.1 · p. 13, Figure 7 · p. 13
- Final video–motion training pairs must satisfy both MPJPE < 0.15 and NMPJPE < 0.15, requiring low absolute and scale-normalized pose error.Using both metrics is more conservative than applying either alone.Source: §8.1 · p. 13
8.2. Direct Validation of the Reciprocal Correspondence Gap
- RecCAR consistently shifts attention mass toward the reciprocal motion-to-video direction, with a comparatively modest shift toward video-to-motion across the test set.The reciprocal shift draws mass from video self-attention; the forward shift draws it from motion self-attention, and the two model clusters are cleanly separated.Source: §8.2 · p. 14, Table 6 · p. 14, Figure 8 · p. 14
- Most test prompts show a reduced reciprocal correspondence gap after RecCAR, as the modality-to-video path aligns more closely with the established video-to-modality correspondence.Points below the diagonal in Figure 9 indicate a reduced gap; the vast majority fall below it.Source: §8.2 · pp. 14–15, Figure 9 · p. 14