DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation

• Figure 1 depicts multiple text prompts as input and longer videos with multiple events, long-range coherence, and smooth transitions as output.Source: Figure 1 · p. 1
• A footnote states that the work was done during an internship at Tencent ARC Lab and notes a corresponding author.Source: §3 · p. 1

1. Introduction

• Current models produce isolated sequences from sequential prompts or fail to capture semantic transitions when prompts are consolidated, limiting depictions of dynamic multi-event scenarios.Source: §1 · p. 2
• Prior multi-prompt work faces costly from-scratch training requirements, while prior approaches are designed for UNet architectures that restrict complex motion.Source: §1 · p. 2
• DiTCtrl treats multi-prompt generation as editing and transitioning over time, using MM-DiT attention analysis, KV-sharing, and latent blending without additional training.Source: §1 · p. 2
• Experiments on MPVBench show state-of-the-art multi-prompt video-generation performance while maintaining computational efficiency.Source: §1 · p. 2
• MPVBench is introduced with diverse transition types and specialized metrics for evaluating multi-prompt transitions.Source: §1 · p. 2

Video Diffusion Model

• Video diffusion work spans motion modules, cascaded spatial-temporal layers, and latent-space generation, with recent models adopting DiT architectures.The paper builds on CogVideoX [48] to explore attention control for multi-prompt long-video generation.Source: §2 · p. 2

Long Video Generation

• Long-video approaches include autoregressive generation, which can degrade through error accumulation, and tuning-free extensions of short-video models.Gen-L-Video [43] uses overlapping clips, while FreeNoise [36] uses temporal attention fusion; MultiDiffusion [5] and Mimicmotion [49] use latent blending for smooth transitions.Source: §2 · p. 3

Image/Video Editing with Attention Control

• Prior attention-control editing methods target image edits or video-to-video translation with structure preservation; this work instead targets appearance-consistent editing over time and analyzes diffusion-transformer attention.Source: §2 · p. 3

3. Method

• Given a pretrained single-prompt T2V model and sequential prompts, DiTCtrl aims to generate a coherent longer video that follows the prompts over time.The method targets zero-shot generation without model training or optimization.Source: §3 · p. 3

3.1. MM-DiT Attention Mechanism Analysis

• MM-DiT concatenates text and video into one attention sequence, whose attention matrix separates into four text/video attention regions.The regions are video-to-video, text-to-text, text-to-video, and video-to-text attention.Source: §3.1 · pp. 3–4
• Averaged text-to-video and video-to-text attention yields token-level semantic maps reshaped per frame, enabling precise localization of textual concepts.The maps are formed in F × H × W format from attention values selected by text-token indices.Source: Figure 2 · p. 3, §3.1 · p. 4
• Text-to-text attention favors neighboring tokens and terminal special tokens, while video-to-video attention shows spatial and cross-frame diagonals resembling UNet attention patterns.The analysis describes these consistent diagonals as evidence of robust frame-to-frame correlations relevant to spatial-temporal coherence and motion fidelity.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2. Consistent Video Generation Over Time

• Mask-guided KV sharing uses thresholded token-attention masks to restrict current-video object and background queries to corresponding regions in the previous video.Masks are derived from averaged text-video and video-text attention for specified foreground-object tokens.Source: §3.2 · pp. 4–5

3.3. Latent Blending Strategy for Transition

• During each denoising step, segment pairs undergo mask-guided KV sharing, then overlapping latents are progressively blended with symmetric, position-dependent weights for smooth transitions.Frames closer to their respective segments receive higher weights, while overlap boundaries receive lower weights.Source: §3.3 · p. 5

4. Experiments

• Experiments use CogVideoX-2B with 49-frame 480 × 720 clips; latent sampling uses 13 frames with a six-frame overlap.The setup specifies 50 sampling steps and kv-sharing layers 25–30.Source: §4 · pp. 5–6

4.2. Qualitative Results

• Figure 6 contrasts prior methods’ discontinuities, jitter, or implausible motion with DiTCtrl’s smoother semantic transitions and coherent motion.FreeNoise+DiT produces more realistic object motion than the UNet-based methods described, but has discontinuities between segments.Source: Figure 6 · p. 6, §4.2 · p. 7

4.1. MPVBench

• MPVBench uses 130 long-form prompts across 10 transition modes and evaluates transition smoothness with CSCV, alongside CLIP text-image similarity and VBench motion smoothness.CSCV is designed to assess transition smoothness in multi-prompt videos.Source: §4.1 · pp. 6–7

4.3. Quantitative Results; 4.4. Human Evaluation

• DiTCtrl achieves the highest CSCV score and superior motion smoothness; users also rank it above other methods across all four evaluation criteria.The user study covers overall preference, motion pattern, temporal consistency, and text alignment.Source: Table 1 · p. 7, §4.3 · p. 7, Table 2 · p. 7

4.3. Quantitative Results

• FreeNoise and Video-Infinity score higher on text-image similarity; DiTCtrl’s KV-sharing draws on preceding segment semantics, while its continuity trade-off is reported not to harm multi-prompt visual quality.The authors point to the user study as evidence for the visual-quality claim.Source: §4.3 · p. 7

4.5. More Applications

• More than 12 times the original length is demonstrated for single-prompt videos; DiTCtrl also supports word-swap and attention-reweighting edits.Word swap uses masked-guided KV-sharing; reweighting changes attention weights for selected tokens.Source: §4.5 · pp. 7–8, Figure 8 · p. 8, Figure 9 · p. 8

4.6. Ablation Study

• Ablations show that latent blending supports smoother transitions, KV-sharing stabilizes environments and motion, and mask guidance limits cross-prompt attribute confusion.The full implementation provides the most precise content control while maintaining object consistency and desired motion patterns.Source: Figure 10 · p. 8, §4.6 · p. 8

5. Conclusion

• DiTCtrl combines mask-guided KV-sharing and latent blending to support smooth transitions and consistent object motion between semantic segments without extra training.The method is based on MM-DiT and uses its attention mechanism for semantic control across prompts.Source: §5 · p. 8
• MPVBench provides diverse transition types and specialized metrics for assessing multi-prompt transitions.Source: §5 · p. 8
• The authors identify attribute-binding errors from weaker conceptual composition and DiT inference overhead as limitations.These are presented as directions for improving semantic understanding and architectural efficiency.Source: §5 · p. 8

Overview

• The supplement provides experimental details, qualitative analyses, technical implementations, and evaluations across baselines, applications, and ablations.Source: Overview · p. 11
• Sections A–D cover experimental setup, evaluation protocols and t-SNE discussion, qualitative comparisons, and applications including video editing.Source: Overview · p. 11
• Sections E–G cover prompt-generator usage, ablation studies of masking, and inference times for alternative methods.Source: Overview · p. 11

A. Implementation Details

• DiTCtrl is implemented using CogVideoX-2B, an open-source text-to-video diffusion model based on MM-DiT.The hyperparameters and implementation details are listed in Table 3.Source: A. Implementation Details · p. 11, Table 3 · p. 11
• Baselines include Kling, Gen-L-Video, FreeNoise, Video-Infinity, and FreeNoise+DiT, with methods aligned to the same stride ratio.The comparisons use six overlapping frames for CogVideoX-2B’s 13 latent frames, approximately a 1/2 stride ratio.Source: A. Implementation Details · p. 11
• For KV sharing, token-specific Text-video and Video-text attention values are averaged into semantic maps and thresholded into foreground-background masks.The previous and current prompt branches supply attention maps; the previous mask guides KV-sharing attention, and the current mask is used for final fusion.Source: A. Implementation Details · pp. 11–12, Figure 11 · p. 12

B. Evaluation details

• MPVBench evaluation uses 130 long-form prompts spanning 10 transition modes, three random seeds, and CSCV, Motion Smoothness, and Text-Image Similarity metrics.The user study included 28 participants rating overall preference, motion pattern, temporal consistency, and text alignment.Source: B. Evaluation details · p. 12
• Real-video t-SNE trajectories are described as continuous due to semantic continuity, and the selected example is a classic car video from DAVIS.The paper reports that DiTCtrl’s multi-prompt video trajectories are comparable to those of real videos.Source: B. Evaluation details · pp. 12–13, Figure 12 · p. 13, Figure 13 · p. 13

C. More Qualitative Results

• DiTCtrl produces multi-prompt videos with temporal consistency, prompt following, and cinematographic-style transitions in a boy’s riding sequence.The section also presents comparisons with multi-prompt methods, FreeNoise+DiT, and Kling.Source: C. More Qualitative Results · p. 13
• Gen-L-Video shows temporal jitter, while Video-Infinity and FreeNoise lack physically plausible motion; FreeNoise+DiT improves motion but has semantic discontinuities.DiTCtrl is reported to preserve DiT motion capabilities while providing smooth semantic transitions and motion coherence.Source: C. More Qualitative Results · p. 13

D. Applications

• The authors report that their attention-control method could also apply to single-prompt longer-video generation and video editing.They describe these as promising applications.Source: D. Applications · p. 13

D.1. Single-prompt Longer Video Generation

• For single-prompt longer videos, DiTCtrl is shown maintaining consistent motion patterns and environmental coherence.The section illustrates this capability with longer-video results in Figure 20.Source: §D.1 · p. 13

D.2. Video Editing

• Video editing uses attention reweighting to attenuate or enhance token semantics, and KV sharing to swap prompt tokens while preserving other content.The examples include reducing “pink” attention, increasing “snowy” attention, and replacing “a large bear” with “a large lion.”Source: §D.2 · pp. 13–14

E. Prompt Generator

• The prompt generator uses GPT-4 to create long-form prompts covering 10 transition modes, including background, subject, camera, style, and action changes.The listed modes also include lighting, location, speed, emotion, and clothing transitions.Source: E. Prompt Generator · p. 14

F.1. Quantitative Results of Components

• DiTCtrl achieves optimal performance on Motion Smoothness, with a slight decrease in Text-Image similarity and higher user-study scores.The paper attributes the similarity decrease to incorporating keys and values from preceding segments into later segments.Source: §F.1 · p. 14
• Latent blending improves CSCV over isolated clips, and KV sharing further increases CSCV.Table 4 presents the component ablation comparison; the supplied text gives no specific CSCV values.Source: Table 4 · p. 14, §F.1 · p. 14
• Mask-guided and unmasked variants have comparable CSCV scores, while qualitative analysis favors the mask-guided method visually.The reported contribution of foreground masking to overall frame-transition smoothness is modest.Source: Table 4 · p. 14, §F.1 · p. 14

F.2. Mask-guided Generation Analysis

• Mask-guided KV-sharing preserves scene coherence for a horse-to-zebra-herd transition and better preserves a white SUV’s appearance, color, and shape through a snow transition.Figure 15 compares generations without mask-guided KV-sharing in the first row and with it in the second.Source: §F.2 · p. 14, Figure 15 · p. 15

G. Inference Time

• Despite a sixfold runtime increase, the mask-guided method remains Pareto optimal in inference-time comparisons.The comparison uses approximately 100 frames for two prompts and 150 frames for three prompts.Source: G. Inference Time · pp. 14–15, Table 5 · p. 16