Source-linked AI summary
The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation
Yichen Liu, Quanwei Zhang, Haozhe Wang, Donghao Zhou, Xiaojie Li, Yang Shi, Jiaming Liu, Ruihua Huang, Yingtian Zou, Daquan Zhou
TL;DR
Existing joint audio-video generators can synchronize their modalities while leaving script-specified shot and dialogue timing implicit in text conditioning. TCR routes that timing independently to video and audio cross-attention, substantially improving temporal accuracy on 200 test scripts while preserving competitive quality and synchronization.
Problem
Joint generators lack explicit alignment between structured script timing and the temporal coordinates of video and audio, limiting control over shot transitions and dialogue timing.
Method
TCR maps each prompt’s script timing onto the shared video-audio timeline and adds independent routing biases to both modalities’ text cross-attention.
Results
96% lower Shot Boundary MAE, from 1.11 s to 0.042 s, and Dialogue Acc@0.5 s rising from 28.3% to 84.1% were achieved on 200 test scripts.
Takeaways & Limitations
TCR improves script-timing adherence while maintaining competitive visual quality and audio-visual synchronization, and users prefer it across all five evaluated dimensions.
Abstract
from arXiv · showhide
Joint audio-video generation models have made substantial progress in visual quality and audio-visual synchronization. However, they still provide limited control over when shot transitions occur and dialogue is spoken. This limitation constrains their application in script-driven content creation, where timing errors can undermine narrative coherence and the viewing experience. Current joint generators align video and audio representations on a shared temporal axis, yet the precise timing of shots and dialogue specified in a structured prompt is encoded only in the prompt's text representation and remains unaligned with the temporal coordinates of either modality. Consequently, video and audio may remain synchronized with each other while both fail to follow the script timeline. This mismatch motivates us to extend temporal alignment beyond video and audio to include the structured script. We therefore introduce Temporal Context Routing (TCR), which maps the script timing onto the shared temporal axis of video and audio generation and routes each prompt's guidance to the corresponding positions in both modalities. Compared with the baseline on 200 test scripts, TCR reduces Shot Boundary MAE by 96%, from 1.11 s to 0.042 s, and raises Dialogue Acc@0.5 s from 28.3% to 84.1%. TCR achieves these improvements while maintaining visual quality and audio-visual synchronization comparable to those of the baselines. A user study further shows that participants prefer TCR on all five evaluated dimensions.
1 Introduction
Script-driven generation requires shot and dialogue timing to align with the shared video-audio timeline, not merely with text conditioning. TCR addresses this gap and substantially improves temporal accuracy while preserving generation quality and synchronization.
- Joint generators can keep video and audio synchronized while both deviate from the script timeline.
- TCR addresses overlapping shot and dialogue spans, fine-grained timing supervision, and preservation of visual quality and audio-visual synchronization.
- TCR maps each prompt’s specified timing onto the shared video-audio axis and independently routes guidance to both modalities.It adds per-prompt routing biases to video-text and audio-text cross-attention.
- 96% lower Shot Boundary MAE, from 1.11 s to 0.042 s, and Dialogue Acc@0.5 s rising from 28.3% to 84.1% were achieved on 200 test scripts.
- A user study preferred TCR across shot timing, dialogue timing, script fidelity, audio-visual synchronization, and overall quality.
2 Related Work
Related work introduces structured and local prompting, joint audio-visual synchronization, and several temporal-control mechanisms. These mechanisms differ in whether timing enters through access masks, representation-based query-key interactions, or routing.
- Joint audio-visual generators couple modality-specific denoisers or align cross-modal features for multimodal video generation.
- Structured and local prompting assigns subcaptions, transition tokens, attention masks, or shot-aware encodings to localized video control.
- Temporal-control methods use access-based masks, representation-based RoPE variants, or dynamic context routing to introduce timing.
3 Method
TCR separates prompt timing from text encoding, converts each interval into normalized routing scores, and applies them independently to video and audio attention. A coarse-to-fine pipeline supplies refined shot and dialogue supervision.
- Temporal Context Routing: TCR adds duration-normalized timing scores to video-text and audio-text cross-attention logits while leaving text, query, and key representations unchanged.
- Task formulation and backbone: The structured script contains Reference, Shot, Event, and Global prompts, with experiments focusing on spoken dialogue for Event prompts.
- Task formulation and timing extraction: TCR extracts prompt time ranges, removes them from textual input, and compiles parent-prompt intervals into a token-level timing map.
- Temporal Context Routing: The routing profile peaks at each prompt’s center, reaches −β/2 at its boundaries, and uses β = 5 throughout.
- Temporal Context Routing: Each prompt receives independent routing at video and audio temporal coordinates expressed in seconds on the same clip timeline.
- Coarse-to-fine data construction: The data pipeline constructs multi-shot clips, generates coarse annotations with Gemini, corrects cuts with PySceneDetect, and aligns dialogue with WhisperX.Refined shot and dialogue timestamps are produced on a 0.1 s grid.
4 Experiments
Experiments show that TCR substantially improves script-timing control across end-to-end, qualitative, human, and controlled evaluations while preserving competitive quality and synchronization. Refined supervision and shared prompt access across both branches are important for joint temporal control.
- Experimental Setup: 200 test scripts support end-to-end evaluation against open-source joint audio-video generators and controlled comparisons using a shared backbone.The evaluation includes 640 Shot prompts and 441 dialogue prompts.
- Main Results: 96% lower Shot Boundary MAE, from 1.11 s to 0.042 s, and Dialogue Acc@0.5s rising from 28.3% to 84.1% establish TCR’s strongest temporal gains.TCR also raises Shot IoU from 0.532 to 0.957 and exact shot-count accuracy from 36.0% to 93.0%.
- Main Results: TCR achieves the highest IQ and Sync-C and the lowest WER while remaining competitive on other quality and synchronization metrics.These results indicate that improved temporal control does not come at the expense of visual quality or audio-visual synchronization.
- Qualitative Comparison: TCR produces all three requested cuts near target times and follows the four-shot structure, whereas every baseline misses or delays at least one transition.Speech-energy traces also expose discrepancies between generated and requested dialogue timing for audio-generating models.
- Human Evaluation: TCR is preferred across all five human-evaluation dimensions against both LTX-2.3 and JoyAI-Echo.With ties included, overall-preference votes are 72.3% against LTX-2.3 and 83.9% against JoyAI-Echo.
- Ablation Studies and Analysis: Among matched temporal operators, TCR reduces Shot Boundary MAE by more than 60% relative to Gaussian Interval RoPE and the hard mask, reaching 0.042 s.It also records Dialogue Acc@0.5s of 84.1%, compared with 83.7% for the hard mask and 82.8% for Gaussian Interval RoPE.
- Ablation Studies and Analysis: Coarse rather than refined annotations increase Shot Boundary MAE from 0.042 s to 0.375 s and reduce Dialogue Acc@0.5s from 84.1% to 37.6%.IQ and AES remain comparable, indicating that the degradation primarily affects temporal accuracy.
- Ablation Studies and Analysis: Providing both Shot and dialogue prompts to both branches better supports joint temporal control and audio-visual coordination than separately assigning prompt types.The Separate A/V prompts variant degrades Dialogue Acc@0.5s, WER, Sync-C, and offset accuracy.
5 Conclusion
The paper addresses missing temporal alignment between structured scripts and joint audio-video generation by introducing TCR and refined temporal supervision. On 200 test scripts, TCR sharply improves shot and dialogue timing while maintaining competitive visual quality and audio-visual synchronization, and users prefer it across all five evaluated dimensions.
- Conclusion: Existing generators align video and audio but represent script-specified shot and dialogue timing only implicitly in text, allowing both modalities to deviate from the script timeline.The paper identifies this as the missing temporal alignment addressed by its method.
- Conclusion: TCR maps script timing onto the shared video-audio temporal axis and routes each prompt’s guidance to corresponding positions in both modalities.The paper also develops a coarse-to-fine data construction pipeline for accurate temporal supervision.
- Conclusion: On 200 test scripts, TCR reduces Shot Boundary MAE by 96%, from 1.11 s to 0.042 s, and raises Dialogue Acc@0.5s from 28.3% to 84.1%.Visual quality and audio-visual synchronization remain competitive with the strongest baseline.
- Conclusion: A user study shows that TCR is preferred across all five evaluated dimensions.The conclusion also cites controlled comparisons and ablations supporting temporal routing and refined supervision.
A Temporal Operator Implementations
The ablation compares temporal operators that inject script timing into cross-attention in different ways. TCR routes timing-aware guidance per prompt, while the alternatives use interval masking or query–key rotary geometry.
- Operator comparison: TCR independently routes each prompt’s guidance using the shared timing map, whereas matched baselines differ only in how timing enters text cross-attention.The comparison uses the same backbone, timing map, training data, LoRA configuration, and optimization settings.
- Operator comparison: Intervals as text leaves numeric time_range fields in the serialized script and applies no separate attention-level temporal operator.
- Operator comparison: The hard interval mask exposes a prompt inside its assigned interval and masks it at all other temporal positions.
- Operator comparison: Gaussian Interval RoPE rotates queries at their temporal coordinates and text keys at assigned-interval centers, scaling key channels with a duration-aware Gaussian kernel.Unlike TCR, it incorporates content and timing jointly in query-key geometry.
- Training comparison: TCR achieves the lowest Shot Boundary MAE and highest Shot IoU at every evaluated checkpoint, with the advantage present at 3,000 steps and stable through 9,000 steps.The main tables use the 7,000-step checkpoint.
C Qualitative Ablation Examples
The qualitative ablation visualizes how temporal operators place cuts and dialogue relative to requested script timing. TCR matches the requested shot structure, while alternatives produce extra or prematurely placed cuts in the examples.
- Qualitative comparison: TCR matches all four requested shots in one example, while Gaussian Interval RoPE and the hard mask produce two extra cuts.Triangles mark produced cuts, and audio lanes show detected speech and energy.
- Qualitative comparison: In another example, Gaussian Interval RoPE places all three cuts early, whereas the hard mask and TCR align cuts closely with requested boundaries.The examples correspond to the ablation results in Table 2 and use identical training and evaluation settings.
D Human Evaluation Protocol
The human evaluation uses anonymous pairwise comparisons between TCR and a comparator across five timing, fidelity, synchronization, and quality dimensions.
- Study design: Twenty-eight participants complete 16 trials drawn from 16 randomly sampled test cases, comparing TCR with LTX-2.3 or JoyAI-Echo.Eight cases are used for each comparator, and videos share a portrait canvas.
- Study design: Each trial presents videos anonymously as A and B with balanced side assignment, followed by independent A/Tie/B choices.Raters view the target shot and dialogue timeline before judging the comparison dimensions.
- Evaluation dimensions: Participants evaluate shot timing, dialogue timing, script fidelity, audio-visual synchronization, and overall execution.
E Metric Details
The paper defines metrics for shot timing, dialogue timing, speech and synchronization, and visual quality, with explicit matching and detector choices. It also parameterizes routing through normalized temporal distance and endpoint retention.
- Shot timing: Shot timing uses temporal intersection-over-union between requested and detected shot intervals, alongside boundary error.
- Shot timing: Requested and detected shots are paired temporally when counts agree and otherwise by highest positive IoU, enabling Shot Boundary MAE and Shot IoU averaging.Matched-shot coverage and exact shot-count accuracy separately measure missing and additional shots.
- Dialogue timing: Dialogue metrics use WhisperX word-level timestamps matched monotonically to requested lines; Acc@0.5s requires both onset and offset errors within 0.5 s.Detection rate counts matched requested lines, while unmatched dialogue receives zero Event IoU.
- Speech and synchronization: Speech fidelity uses WER, while Sync-C and offset accuracy use SyncNet on speech-active within-shot segments; offset accuracy requires errors within one frame at 24 fps.
- Visual quality: Visual quality reports VBench Imaging Quality and Aesthetic Quality, excluding whole-clip consistency and motion smoothness because omitted cuts can score favorably.
- Routing parameterization: TCR routing depends on normalized temporal distance and contributes a multiplicative factor to unnormalized attention weights.The endpoint-retention ratio ε denotes the fraction of center weight retained at either interval endpoint.
- Routing parameterization: The fixed choice β = 5 corresponds to endpoint retention e^-5/2 = 0.082 and is used across blocks, modalities, and attention heads.It is neither learned nor tuned per prompt.
G Implementation Details
TCR uses matched temporal operators with specified training and inference settings, including LoRA adaptation, scheduled optimization, and fixed sampling configuration.
- Temporal coordinates and batching: Video query times use latent-cell midpoints, while audio query times derive from patch positions, hop length, and sample rate before conversion to seconds.
- Temporal coordinates and batching: Global prompts route across [0, T], sentinel tokens receive no routing score, and ε = 10^-4 s is used in Equation 2.
- Training and inference: The compiled routing score is shared across attention heads, and TCR adds no classifier-free-guidance-specific parameters.
- The three matched temporal operators use LoRA rank and scale 128, learning rate 10^-4, 500 warm-up steps, cosine decay, and audio loss weight 1.
- Training and inference: At inference, TCR uses 30 sampling steps, guidance 4.0, spatiotemporal guidance 1.0 at block 29, seed 42, 704 × 1280 resolution, and 24 fps.
H Structured Script and Temporal Compilation
The structured script retains shot and dialogue timing alongside semantic content, while temporal compilation aligns prompt tokens with their timing for routing. This changes the conditioning pathway without changing the requested clip duration or supplied information.
- Temporal Compilation: The compiler removes numeric ranges before text encoding for matched operators and supplies them separately through a timing map.
- Temporal Compilation: Each prompt’s character span is aligned to token positions, and every token inherits the timing of its parent prompt.
- Temporal Compilation: Repeated references are aligned independently and inherit the timing of their enclosing shot, while padding and special tokens receive sentinel timing with no routing score.
- Structured Script: Serialized scripts provide numeric time_range fields for shots and dialogue events, alongside references, visual descriptions, scene descriptions, style, and global audio.
- Temporal Compilation: The compiler changes only how script timing enters conditioning, because requested clip duration remains set by the final shot endpoint and semantic information is unchanged.