VideoStudio: Generating Consistent-Content and Multi-Scene Videos

1 Introduction

• Multi-scene video generation must arrange events logically and preserve the appearance of shared entities, challenges left underexplored by work focused on single-scene videos.VideoStudio uses LLMs to organize scenes and entity reference images to link content across them.Source: §1 · p. 2
• VideoStudio converts prompts into multi-scene scripts, generates reference images for common entities, and uses two diffusion models to produce scene images and video clips.The script specifies each scene’s event, foreground, background, and camera movement; VideoStudio-Img creates a scene-reference image, and VideoStudio-Vid generates the clip.Source: §1 · pp. 2–3
• The authors report that VideoStudio outperforms state-of-the-art video-generation models in visual quality, content consistency, and user preference.Source: §1 · p. 3

2 Related Work

• VideoStudio explicitly generates entity reference images, unlike the most related work [26], which aligns entity appearance across scenes using LLM-provided bounding boxes.The reference images serve as a link across scenes to enhance content consistency.Source: §2 · p. 4

3 VideoStudio

• VideoStudio has three stages: multi-scene script generation, entity reference-image generation, and video scene generation.Source: §3 · p. 4

3.1 Multi-Scene Video Script Generation

• The LLM creates a script with an event prompt, foreground entities, background, and camera movement for each scene; common entities are matched by name and described through multi-round dialogue.The number of scenes is determined by the LLM, and camera directions and speeds are selected from specified sets.Source: §3.1 · p. 5

3.2 Entity Reference Image Generation

• Entity descriptions are sent to Stable Diffusion to generate reference images, which U2-Net segments so foreground and background pixels can be separated to reduce visual interference.Source: §3.2 · p. 6

3.3 Video Scene Generation

• VideoStudio-Img conditions scene-reference image generation on the event prompt and foreground and background reference images, using added cross-attention modules for the entity images.The model optimizes the added cross-attention modules while freezing the other network parts.Source: §3.3 · pp. 6–7
• VideoStudio-Vid generates a clip from the scene-reference image, action category, and camera movement, using spatial and temporal self-attention and added temporal convolutions.Source: §3.3 · p. 7
• Intermediate frames are warped after Tm DDIM steps to reflect scripted camera movement; Tm=5 is reported as an empirical trade-off between camera control and visual quality.Source: §3.3 · p. 8

4.1 Datasets

• Training uses LAION-2B text-image pairs for VideoStudio-Img and WebVid-10M plus a randomly chosen 20M HD-VG-130M subset for VideoStudio-Vid; evaluation prompts come from MSR-VTT, ActivityNet Captions, and Coref-SV.MSR-VTT supplies single-scene prompts, while ActivityNet Captions and Coref-SV supply multi-scene prompts.Source: §4.1 · p. 8

4.2 Evaluation Metrics

• Evaluation measures video quality with FID and FVD, prompt alignment with CLIPSIM, intra-scene consistency with frame consistency, and cross-scene consistency with scene consistency.Scene consistency compares detected common objects across scenes using CLIP similarity.Source: §4.2 · pp. 8–9

4.3 Implementation Details

• VideoStudio is implemented with Diffusers and PyTorch, with VideoStudio-Img trained from Stable Diffusion v2.1 and VideoStudio-Vid built on Stable Diffusion XL with temporal layers.The image model adds cross-attention modules; the video model adds temporal attentions and convolutions.Source: §4.3 · p. 9

4.4 Experimental Analysis of VideoStudio

• Using both foreground and background references yields 0.8102 FG-SIM and 0.5861 BG-SIM, while reference-guided VideoStudio-Img variants outperform IP-Adapter or variants without the corresponding references.The authors attribute alignment gains to pre-segmented regions and local image tokens that improve spatial discrimination.Source: §4.4 · pp. 9–10, Table 1 · p. 10
• On WebVid-10M, RF+VideoStudio-Vid achieves 116.5 FVD and 98.8 frame consistency; on MSR-VTT, VideoStudio-Vid attains the best FVD with and without a real-frame reference.On VBench, action conditioning raises the action score from 90.3% to 96.5%.Source: Table 2 · p. 10, §4.4 · pp. 11–12, Table 3 · p. 11

4.5 Evaluations on Multi-Scene Video Generation

• VideoStudio reaches 77.3 scene consistency on Coref-SV, improving by 39.4 over ModelScopeT2V and 34.5 over VideoDirectorGPT; on ActivityNet Captions, it also exceeds VideoStudio w/o Ref. by 24.3.Coref-SV has prompts but no corresponding videos, so FID and FVD are unavailable there.Source: §4.5 · pp. 12–13, Table 4 · p. 12, Table 5 · p. 13

4.6 Human Evaluation

• A user study found that LLM-generated scripts improve logical coherence and entity reference images improve content consistency; evaluators preferred GPT-4-generated scripts over ChatGLM3-6B, while preferences for corresponding videos were comparable.Source: §4.6 · p. 14

• VideoStudio uses LLMs to rewrite prompts as multi-scene scripts and generates reference images for common entities to support appearance consistency across scenes.VideoStudio-Img creates scene-reference images from event prompts and entity references; VideoStudio-Vid conditions video clips on the scene image, action, and camera movement.Source: §5 · p. 14
• The framework uses VideoStudio-Img and VideoStudio-Vid to turn event prompts and entity references into scene images and video clips.Source: §5 · p. 14
• Evaluations on four video benchmarks report superior visual quality and content consistency over state-of-the-art models.Source: §5 · p. 14

• The authors are Fuchen Long, Zhaofan Qiu, Ting Yao, and Tao Mei, affiliated with HiDream.ai Inc.Source: VideoStudio: Generating… · p. 19
• The supplement includes LLM instructions, implementation details for both VideoStudio models, performance contribution, more generated examples, and a video demo.Source: VideoStudio: Generating… · p. 19

• Figures 2 and 3 present LLM instructions, output examples, and in-context examples for video-script and entity-description generation.Source: §1 · p. 19
• Figure 4 shows the multi-round dialogue used for entity-description generation.Source: §1 · p. 19

• VideoStudio-Img is built on Stable Diffusion v2.1 and incorporates two additional cross-attention modules.Source: §2 · p. 19
• CLIP ViT-H/14 encodes text prompts and foreground/background reference images into text and local image features.Source: §2 · p. 19
• Text features have sequence length 77, while foreground and background image features each have length 256.Source: §2 · p. 19
• The text and image cross-attention dimensions use the default number in each block of the original diffusion model.Source: §2 · p. 19

Implementation details

• VideoStudio-Vid extracts scene-reference image features with CLIP ViT-H/14 and concatenates their latent code with noisy video latents along the temporal dimension.The resulting input is processed by a 3D UNet built by adding temporal transformer and convolution layers to SD-XL's 2D UNet.Source: §3 · p. 19

Action condition

• Action conditioning uses a VideoMAE classifier trained on Kinetics-400, while inference maps prompt action phrases to Kinetics-400 categories by CLIP cosine similarity.Categories below a cosine similarity of 0.2 are dropped; retained similarities are normalized into a 400-entry indicator vector.Source: §3 · pp. 19, 21

Camera movement

• After Tm sampling steps, VideoStudio-Vid blends the estimated clean video with a flow-warped version and re-noises it to maintain scripted camera movement.The prose identifies sampling as DDIM and says the warp uses optical flow for the required camera movement.Source: §3 · p. 21
• Tm=5 provides an empirical trade-off: earlier modification may not control camera movement effectively, while later modification may affect visual quality.The study compares generated and target optical-flow MSE across Tm settings on MSR-VTT.Source: §3 · pp. 21–22, Figure 1 · p. 21

Performance contribution

• The full VideoStudio version uses entity reference images and improves both video quality and cross-scene consistency.This is the reported outcome for the full version in the ActivityNet Captions ablation.Source: §4 · p. 22
• The LAION- and WebVid-10M-trained variant improves video quality over ModelScopeT2V, attributed to the deeper Stable Diffusion backbone and two-stage T2I+I2V framework.The comparison uses FID and FVD as video-quality metrics.Source: §4 · p. 22
• Adding HD-VG training data to the second variant leads to slightly better video quality.The paper reports this change in the ActivityNet Captions ablation.Source: §4 · p. 22

More video examples

• Figures 5 and 6 show MSR-VTT multi-scene video examples using single and multiple foreground reference images, respectively.Each example also presents its input prompt, camera movement, foreground and background reference images, and generated video.Source: §5 · p. 22
• Figure 7 shows three generated multi-scene video examples using real foreground and background entity-reference images.These examples are presented alongside their generated videos.Source: §5 · p. 22
• The examples pair generated videos with the input prompt, camera movement, and foreground and background reference images.This information is given for each example.Source: §5 · p. 22

Video demo

• The video demo illustrates VideoStudio-generated videos in various scenarios, and the offline project page provides further details.The demo is named VideoStudio.mp4 and the project page VideoStudio.html.Source: §6 · p. 22
• Figures 2 and 3 show LLM instructions, output examples, and in-context examples for generating scene prompts, common entities, and camera movements..Source: Figure 2 · p. 23, Figure 3 · p. 23
• Figures 5 and 6 show generated multi-scene videos on MSR-VTT with single and multiple foreground reference images, respectively; Figure 7 shows three generated multi-scene videos using real images as entity reference images.Source: Figure 5 · p. 24, Figure 6 · p. 25, Figure 7 · p. 26