Source-linked AI summary
AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction
Zhen Xing, Qi Dai, Zejia Weng, Zuxuan Wu, Yu-Gang Jiang
TL;DR
Text-guided video prediction must align initial frames with instructions and produce consistent future frames, while text-to-image-based approaches lack video priors and may struggle on small domain datasets. AID transfers a pretrained Image2Video model with MLLM-guided conditioning and temporal and spatial adapters; it reports over 50% FVD improvement across multiple datasets and significant gains over prior state of the art.
Problem
Text-guided video prediction must align initial frames with instructions and generate consistent futures, while adapting text-to-image models can lack video priors and perform poorly on small domain-specific datasets.
Source: §1 · p. 1Method
AID transfers a pretrained Image2Video model and uses an MLLM, a dual-branch DQFormer for multimodal and frame-level controls, and temporal and spatial adapters.
Source: §1 · pp. 2–3, §5 · p. 9Results
AID reports over 50% FVD improvement across multiple datasets and significantly surpasses prior state of the art [23] on four mainstream TVP datasets.
Source: §1 · pp. 2–3Takeaways & Limitations
Abstract
from arXiv · showhide
Text-guided video prediction (TVP) involves predicting the motion of future frames from the initial frame according to an instruction, which has wide applications in virtual reality, robotics, and content creation. Previous TVP methods make significant breakthroughs by adapting Stable Diffusion for this task. However, they struggle with frame consistency and temporal stability primarily due to the limited scale of video datasets. We observe that pretrained Image2Video diffusion models possess good priors for video dynamics but they lack textual control. Hence, transferring Image2Video models to leverage their video dynamic priors while injecting instruction control to generate controllable videos is both a meaningful and challenging task. To achieve this, we introduce the Multi-Modal Large Language Model (MLLM) to predict future video states based on initial frames and text instructions. More specifically, we design a dual query transformer (DQFormer) architecture, which integrates the instructions and frames into the conditional embeddings for future frame prediction. Additionally, we develop Long-Short Term Temporal Adapters and Spatial Adapters that can quickly transfer general video diffusion models to specific scenarios with minimal training costs. Experimental results show that our method significantly outperforms state-of-the-art techniques on four datasets: Something Something V2, Epic Kitchen-100, Bridge Data, and UCF-101. Notably, AID achieves 91.2% and 55.5% FVD improvements on Bridge and SSv2 respectively, demonstrating its effectiveness in various domains. More examples can be found at our website https://chenhsing.github.io/AID.
1 Introduction
Text-guided video prediction requires aligning initial frames with instructions while generating consistent futures, but methods adapted from text-to-image models lack video priors and can struggle with stability on small domain datasets. AID transfers a pretrained Image2Video model, adding MLLM-guided DQFormer conditioning and temporal and spatial adapters; the paper reports significant gains over prior state of the art.
1 Introduction
- TVP predicts future frames from initial frames and an instruction, supporting long-video generation and applications such as robotic-arm planning and first-person VR.Its challenges include understanding initial frames, aligning them with text, and generating consistent future frames.Source: §1 · p. 1
- Methods extending text-to-image models to TVP lack video priors, and training on small domain-specific video datasets can yield poor consistency and stability.Tutorial, first-person cooking, and robotic-arm videos are cited as settings where limited data constrains generation quality.Source: §1 · p. 2
- SVD provides a pretrained Image2Video base, but its public version supports image-to-video generation without textual control; adapting it requires instruction injection and lower-cost target-dataset training.The figure contrasts SVD's duplicated initial image and noise with AID's initial frames, future-frame mask, and text conditions.Source: §1 · p. 2, Figure 1 · p. 2
- AID uses an MLLM to predict future video states, a dual-branch DQFormer to integrate text and visual controls and decompose state conditions into frame-level controls, and temporal and spatial adapters for transfer with few parameters and computational costs.The combined multi-condition embedding is injected into the UNet through cross-attention.Source: §1 · pp. 2–3
- Over 50% FVD improvement across multiple datasets is reported for AID, which also significantly surpasses prior state of the art [23] on four mainstream TVP datasets.The paper presents these results as evidence for transferring general video generation models to specific video prediction tasks.Source: §1 · pp. 2–3
2 Related Work
Related work spans video diffusion, video prediction, and language-model control of video generation. AID is situated among methods that extend image models, adapt video models, and use language models to plan or structure video content.
Video Diffusion Models
Video Prediction
- Video prediction moved from GANs [36] [10] to autoregressive Transformers [17] [30] and diffusion models [23] [76]; Seer [23] adds text-based prediction but is described as limited in stability and continuity by small datasets.AID is framed as transferring a large video diffusion prior [3] to this task.Source: §2 · p. 3
3 Method
AID adapts pretrained Stable Video Diffusion for text-guided video prediction by combining MLLM-predicted future states with DQFormer conditioning and trainable spatial and temporal adapters.
3.1 Preliminaries of Diffusion Model
- Diffusion preliminaries perturb data with Gaussian noise and denoise from high noise toward zero using learned score and denoiser parameterizations.The score model approximates the score function, while the denoiser predicts clean data.Source: §3.1 · pp. 3–4
3.2 Overview
- Given the first K frames and an instruction, AID adapts pretrained SVD to complete the remaining N − K frames through masked-latent denoising and video reconstruction.Conditioned and noisy latents are concatenated before the U-Net denoises the sequence; the VAE decoder reconstructs the video.Source: §3.2 · p. 4, Figure 2 · p. 4
3.3 Text Condition Injection
- An MLLM predicts future video states from the initial frame and instruction; DQFormer aligns frame and instruction globally, decomposes prompts into frame-level conditions, and injects MCondition through cross-attention.Its upper branch aligns the visual feature with the instruction, while its lower branch decomposes instruction and MLLM-aided prompt features for each frame.Source: §3.3 · pp. 5–6, Figure 4 · p. 5
3.4 Adapter Modeling
- The spatial adapter targets dataset spatial distributions, short-term 3D convolution models adjacent-frame relations, and temporal self-attention targets global relations; training updates adapters while freezing the original U-Net.The spatial adapter is added alongside spatial self-attention; the two temporal adapters use different mechanisms for short- and long-range modeling.Source: §3.4 · pp. 6–7
4 Experiments
AID is evaluated on text-guided video prediction across SSv2, Bridge Data, and Epic100, with additional UCF-101 evaluation. Results and ablations assess its performance against prior methods and the contribution of its conditioning and adapter components.
4.1 Datasets
- Evaluations use SSv2, Bridge Data, and Epic100, spanning everyday actions, robotic-arm operations, and first-person kitchen activities.SSv2 and Epic100 videos are reduced to 12 frames and Bridge videos to 16 frames for training and evaluation.Source: §4.1 · p. 7
4.2 Implementation Details
- The model initializes its VAE and U-Net blocks from Stable Video Diffusion, freezes the VAE and 3D U-Net, and trains added components and cross-attention layers.Videos are resized to 256 × 256; inference uses separate guidance scales for frame and text conditions.Source: §4.2 · p. 7
4.3 Evaluation Setting
- The evaluation compares eight video-generation baselines and reports FVD and KVD on 2,048 SSv2, 5,558 Bridge, and 9,342 Epic100 validation samples.UCF-101 class-conditioned results are reported separately in the appendix.Source: §4.3 · pp. 7–8
4.4 Main Results
- Over 50% FVD improvement on all three datasets, with over 90% on Bridge Data, where AID’s FVD is 21.57.Table 1 reports FVD and KVD for SSv2, Bridge, and Epic100; AID significantly surpasses Seer across all three.Source: Table 1 · p. 8, §4.4 · p. 8
- In qualitative comparisons on SSv2 and Bridge, AID better follows the instructions and maintains stable video generation than Seer, Gen-2, and Pika.Examples include moving a box closer to a car and putting corn in a pot among sink distractors.Source: Figure 6 · p. 8, §4.4 · p. 9
4.5 Ablation Study
- Removing either DQFormer branch worsens FVD and KVD, while removing both leaves the model without text guidance and significantly worsens FVD.Removing LLava-predicted state prompts also decreases performance.Source: §4.5 · p. 9, Table 2 · p. 9
- Removing any adapter worsens performance, and training only DQFormer with the U-Net fixed causes the largest decrease.The authors interpret the adapter ablations as evidence that conditional injection alone cannot transfer the model to the target video distribution.Source: §4.5 · p. 9, Table 3 · p. 9
5 Conclusion
The conclusion presents AID as a transfer approach for text-guided video prediction, combining MLLM prompting, DQFormer conditioning, and spatial and temporal adapters. Its experiments support the feasibility and potential of transferring pretrained video diffusion models to domain-specific video generation.
5 Conclusion
- AID combines MLLM-predicted video states, a dual-branch DQFormer for multimodal control, and spatial and temporal adapters for transfer with few parameters and training costs.The authors describe it as an exploration of transferring a pretrained video diffusion model to domain-specific video generation.Source: §5 · p. 9
A.1 Additional Results on UCF-101
The appendix evaluates class-conditioned video prediction on UCF-101 using one- and five-reference-frame settings. AID outperforms other models on FID with one reference frame, has FVD results comparable to Make-A-Video, and performs much better with five reference frames.
A.1 Additional Results on UCF-101
- UCF-101 is evaluated as a class-conditioned video-prediction benchmark, although the authors note it is not particularly suited to the TVP task.They nevertheless conduct two settings to assess video-generation performance.Source: §A.1 · p. 15
Settings
- The evaluation fine-tunes on 256 × 256, 16-frame videos and tests prediction from one or five reference frames, reporting FVD and FID on 2,048 test samples.Each class is given a sentence caption, following the PYoCo method.Source: §A.1 · p. 15
Results
- With one reference frame, AID significantly outperforms other video-generation models on FID and achieves FVD results comparable to Make-A-Video; with five frames, it performs much better than other methods.The reported UCF-101 evaluation uses 16-frame videos and includes qualitative results for both settings.Source: §A.1 · p. 15, Table 4 · p. 15
B Additional Visualization
Additional visualizations cover Epic-Kitchen and Bridge predictions, long-video prediction, and instruction-based video manipulation. The examples include iterative clip extension and different predicted futures from the same reference frame under different instructions.
B Additional Visualization
- For iterative long-video examples, the last two frames of one clip become the initial frames for the next clip.Other examples use the same reference frame with different instructions to produce different future frames.Source: B Additional Visualization · p. 16, Figure 11 · p. 20
- The appendix provides additional prediction visualizations for Epic-Kitchen and Bridge.Figure 10 shows Epic-Kitchen examples; Figure 9 shows Bridge examples.Source: B Additional Visualization · p. 16, Figure 9 · p. 18, Figure 10 · p. 19
- The visualizations also include long-video prediction and text-guided video manipulation examples.These are presented in Figure 11.Source: B Additional Visualization · p. 16, Figure 11 · p. 20