Source-linked AI summary
Prompt Relay: Inference-Time Temporal Control for Multi-Event Video Generation
Gordon Chen, Ziqi Huang, Ziwei Liu
TL;DR
Video diffusion models lack explicit control over the timing and ordering of multiple events, which can cause semantic entanglement across temporal regions. Prompt Relay adds inference-time cross-attention routing without architectural modifications or computational overhead, and the paper reports improved temporal alignment, transition naturalness, and visual quality, with fidelity still bounded by backbone capacity.
Problem
Video diffusion models lack explicit temporal control over when multiple events occur, how long they persist, and their order, while global attention can cause temporal misalignment and semantic entanglement.
Method
Prompt Relay is an inference-time, plug-and-play method that penalizes cross-attention logits so each temporal segment primarily attends to its assigned prompt.
Results
Prompt Relay consistently improves temporal alignment and transition naturalness over baseline approaches, improves visual quality over the backbone model, and remains below Kling 2.6 and Veo 3.1 in overall visual quality.
Takeaways & Limitations
Prompt Relay provides fine-grained temporal control for multi-event video generation while preserving global temporal coherence through localized prompt influence and smooth boundary transitions.
Takeaways & Limitations
Persistent elements may drift when local prompts describe them inconsistently across segments, unless a global prompt anchors shared context.
Abstract
from arXiv · showhide
Video diffusion models have achieved remarkable progress in generating high-quality videos. However, these models struggle to represent the temporal succession of multiple events in real-world videos and lack explicit mechanisms to control when semantic concepts appear, how long they persist, and the order in which multiple events occur. Such control is especially important for movie-grade video synthesis, where coherent storytelling depends on precise timing, duration, and transitions between events. When using a single paragraph-style prompt to describe a sequence of complex events, models often exhibit semantic entanglement, where concepts intended for different moments in the video bleed into one another, resulting in poor text-video alignment. To address these limitations, we propose Prompt Relay, an inference-time, plug-and-play method to enable fine-grained temporal control in multi-event video generation, requiring no architectural modifications and no additional computational overhead. Prompt Relay introduces a penalty into the cross-attention mechanism, so that each temporal segment attends only to its assigned prompt, allowing the model to represent one semantic concept at a time and thereby improving temporal prompt alignment, reducing semantic interference, and enhancing visual quality.
1. Introduction
Existing video diffusion models generate high-quality videos but lack explicit temporal control for ordering, timing, and duration of multiple events. Prompt Relay addresses this gap with inference-time attention routing that improves temporal control without training or architectural changes.
- Motivation: Existing models are optimized for single-event generation and cannot explicitly specify when events occur, how long they persist, or their order.This makes movie-grade videos composed of temporally ordered events, actions, or camera motions challenging.
- Motivation: Temporal misalignment and semantic entanglement arise because cross-attention conditions every frame on the entire prompt.Concepts intended for different temporal regions can compete for influence over the same latent queries.
- Prior limitations: Prior temporally grounded methods require annotated data, training, or model modification, while inference-time controls can restrict attention structure and introduce boundary artifacts.These trade-offs limit generality and deployment convenience.
- Contribution: Prompt Relay is an inference-time, plug-and-play attention-level routing mechanism compatible with existing video diffusion backbones.It targets fine-grained temporal control and multi-event video generation.
- Contribution: Prompt Relay uses Boundary-Attention decay to suppress cross-segment semantic interference while improving temporal prompt alignment and visual quality without computational overhead or architectural modifications.The method applies a soft Gaussian penalty to cross-attention logits.
2. Related Works
Related work establishes temporal controllability as an emerging extension of controllable video generation. Existing approaches differ in whether they modify the model through training or control attention and generation at inference time.
- Controllable Video Generation: Most controllable video-generation models address motion, viewpoint, identity, or editing, but remain limited in coherent multi-event video generation.Global attention over prompt tokens contributes to temporal misalignment and semantic entanglement.
- Attention Control: Attention manipulation is established for spatial, identity, and motion control, whereas attention-based temporal control remains largely underexplored.This positions temporal prompt routing as a distinct research direction.
- Prior Temporal Methods: MinT binds event descriptions to predefined intervals using trainable temporal cross-attention but requires architectural changes, training, and temporally annotated data.MEVG instead generates event clips sequentially, but its autoregressive design can accumulate errors and produce abrupt transitions.
3. Prompt Relay
Prompt Relay routes temporally constrained prompts through cross-attention so each instruction primarily influences its assigned interval while global coherence is preserved. Its soft boundary decay replaces abrupt prompt switching with controlled attenuation and gradual transitions.
- 3. Prompt Relay: Prompt Relay aims to realize each prompt within its designated temporal interval while preserving global video coherence.Each prompt should influence only its assigned temporal region.
- 3.1. Preliminaries: Cross-attention injects prompt-derived conditioning into latent features through learned query, key, and value projections.Attention weights determine how strongly latent queries respond to conditioning tokens.
- 3.1. Preliminaries: Figure 3 compares retained attention across latent-frame offsets for different window and decay-threshold settings.Smaller ϵ values strengthen attenuation outside the free-attention window, but small-value choices have negligible perceptual impact; ϵ = 0.1 is used by default.
- 3.1. Preliminaries: Global attention lets concepts from different temporal regions compete for the same latent queries, creating semantic interference between instructions.Prompt Relay addresses this competition through temporal routing.
- 3.2. Temporal Prompt Routing: The penalty C(Q, K) is added to cross-attention logits to suppress query-key attention when they belong to different temporal intervals.This confines each prompt’s guidance to its intended segment and reduces semantic leakage.
- 3.2. Temporal Prompt Routing: Hard masking switches prompts abruptly at boundaries, whereas Boundary-Attention decay smoothly co-activates neighboring prompts to support a gradual handoff.The soft transition avoids conflicts between changing cross-attention and continuous self-attention.
- 3.2. Temporal Prompt Routing: The local window w defines an unpenalized region around each segment midpoint, while σ controls smooth attenuation with temporal distance.The paper reports w = L −2 as the best balance between temporal isolation and intra-segment fidelity.
- 3.3. Boundary-Attention Decay: Boundary-Attention decay applies a multiplicative prior that remains 1 inside the free-attention window and decreases toward segment endpoints.This supports smooth neighboring-prompt transitions while limiting destructive cross-segment interference and maintaining temporal coherence.
4. Experiments
The experiments evaluate Prompt Relay on diverse multi-event video scenarios using human preference scores and qualitative comparisons. Prompt Relay consistently improves temporal alignment and transition naturalness, while visual fidelity remains partly limited by the backbone.
- Results: Prompt Relay preserves correct temporal structure in deliberate scene transitions by confining each semantic instruction to its intended segment while maintaining global visual coherence.The qualitative comparison focuses on whether multi-event prompts produce the intended temporal organization.
- Experimental Setup: Human preference scores assess temporal prompt alignment, transition naturalness, and visual quality because standard metrics do not capture temporal semantics or transition quality.Temporal prompt alignment measures whether each prompt appears in its intended interval, while transition naturalness evaluates smoothness between consecutive events.
- Experimental Setup: The evaluation uses 20 diverse multi-event scenarios containing 3–6 temporal events, with 30 participants ranking videos on a 1–5 scale.Scenarios include explicit scene transitions, multi-character interactions, and complex camera trajectories; model identity was withheld.
- Results: Prompt Relay consistently outperforms baseline approaches in temporal alignment and transition naturalness.The comparison is reported in Table 1, whose human preference scores use lower values to indicate better rankings.
- Results: Kling 2.6 and Veo 3.1 achieve higher visual quality overall, indicating that visual fidelity remains partly bounded by the underlying backbone capacity.Prompt Relay nevertheless improves visual quality over the Wan 2.2 backbone baseline.
5. Limitations
Prompt Relay’s segment-focused attention can leave persistent visual elements insufficiently shared across segments. Inconsistent descriptions across local prompts may therefore cause appearance drift, which a global prompt can mitigate.
- Limitations: Persistent characters, objects, or scene style are not explicitly shared across temporal segments because each segment primarily attends to its local prompt.This creates a limitation when persistent elements must remain consistent throughout the video.
- Limitations: Inconsistent descriptions of persistent elements across local prompts may cause their appearance to drift over time.The authors report that incorporating a global prompt can fully mitigate this issue by anchoring shared context.
6. Conclusion
The paper presents Prompt Relay as an inference-time, plug-and-play approach for fine-grained temporal control in multi-event video generation. It reports improved visual quality over the backbone model and frames the method as a step toward movie-grade controllable synthesis.
- Conclusion: Prompt Relay is an inference-time, plug-and-play method for multi-event video generation with fine-grained temporal control.The method is presented as the paper’s central contribution.
- Conclusion: The method improves visual quality over the backbone model.This conclusion summarizes the reported experimental outcome.
- Conclusion: The authors view Prompt Relay as a pivotal step toward movie-grade, controllable video synthesis.The conclusion states this as the work’s broader supported significance.