PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning

Han Lin, Abhay Zala, Jaemin Cho, Mohit Bansal

arXiv:2309.15091v2cs.CVcs.AIcs.CLcs.LG

TL;DR

VideoDirectorGPT targets the difficulty of generating long, multi-scene videos with changing events and consistent entities. It uses GPT-4 to create structured video plans and Layout2Vid to generate videos with layout and consistency control, achieving stronger layout and movement control while producing consistent multi-scene videos. The method also remains competitive on open-domain single-scene T2V generation.

  • Problem

    Most T2V methods focus on short single-event clips, creating a need for temporally consistent generation of longer videos with multiple changing scenes.

  • Method

    GPT-4 expands one prompt into a plan of scenes, entities, layouts, backgrounds, and consistency groupings, while Layout2Vid generates the planned video with grounded spatial and temporal control.

  • Results

    The framework improves object layout and movement control, generates visually consistent multi-scene videos, and remains competitive with SOTAs on open-domain single-scene T2V generation.

  • Takeaways & Limitations

    LLM-generated video plans can guide grounded generation toward controllable layouts, object movements, and entity consistency across multiple scenes.

  • Takeaways & Limitations

    Bounding-box control can degrade when many boxes overlap, and background motion is constrained by the video generator’s prior knowledge.

Abstract

from arXiv · show

Recent text-to-video (T2V) generation methods have seen significant advancements. However, the majority of these works focus on producing short video clips of a single event (i.e., single-scene videos). Meanwhile, recent large language models (LLMs) have demonstrated their capability in generating layouts and programs to control downstream visual modules. This prompts an important question: can we leverage the knowledge embedded in these LLMs for temporally consistent long video generation? In this paper, we propose VideoDirectorGPT, a novel framework for consistent multi-scene video generation that uses the knowledge of LLMs for video content planning and grounded video generation. Specifically, given a single text prompt, we first ask our video planner LLM (GPT-4) to expand it into a 'video plan', which includes the scene descriptions, the entities with their respective layouts, the background for each scene, and consistency groupings of the entities. Next, guided by this video plan, our video generator, named Layout2Vid, has explicit control over spatial layouts and can maintain temporal consistency of entities across multiple scenes, while being trained only with image-level annotations. Our experiments demonstrate that our proposed VideoDirectorGPT framework substantially improves layout and movement control in both single- and multi-scene video generation and can generate multi-scene videos with consistency, while achieving competitive performance with SOTAs in open-domain single-scene T2V generation. Detailed ablation studies, including dynamic adjustment of layout control strength with an LLM and video generation with user-provided images, confirm the effectiveness of each component of our framework and its future potential.

1 Introduction

VideoDirectorGPT addresses the difficulty of generating long videos with multiple changing events by combining LLM-based video planning with grounded generation. Its two-stage design produces multi-scene videos with controllable layouts, movements, and entity consistency.

  • Motivation: Most T2V systems generate short clips centered on one event, while existing long-video methods often repeat or continue a single action instead of modeling changing events.The motivation contrasts short single-event generation with the limited multi-event dynamics of longer videos.
  • Framework: VideoDirectorGPT expands one text prompt into a multi-scene plan containing scene descriptions, entities and layouts, backgrounds, and consistency groupings.The plan specifies where entities appear and which entities should remain visually consistent across scenes.
  • Framework: The framework then uses Layout2Vid to render videos from the plan with explicit layout control and visually consistent entities across scenes.Layout2Vid is the grounded video-generation stage of the pipeline.
  • Training: The method uses image-level layout annotations to train Layout2Vid while adding spatial and consistency control to a pretrained video generator.Only a small set of trainable parameters is added to the pretrained backbone.
  • Results: Experiments report improved layout and movement control over several T2V/I2V systems, consistent multi-scene videos, and competitive open-domain single-scene performance.The reported evaluation covers both single-scene and multi-scene generation.
  • Framework: GPT-4 serves as the video planner, while Layout2Vid serves as the grounded video generator for multi-scene synthesis.The framework separates content planning from video rendering.

2 Related Works

Related work motivates VideoDirectorGPT by highlighting the tension between efficient pretrained T2V systems and long-range consistency. The paper combines parameter-efficient temporal consistency with LLM-generated structured video plans.

  • Text-to-video generation: Pretrained T2I-to-T2V strategies enable high-resolution video generation but generally remain limited to short clips because long-video consistency is difficult.The related work contrasts short-clip generation with approaches targeting longer videos.
  • Layouts and planning: Layout2Vid extends layout-guided generation toward multi-scene video by combining layout control with entity-level temporal consistency.The framework connects layout-based control with consistency across scenes.
  • Text-to-video generation: The paper introduces temporal consistency into a pretrained T2V model through data- and parameter-efficient training while preserving its original visual quality.This positions Layout2Vid as an adaptation rather than a model trained entirely from scratch.
  • Layouts and planning: Prior layout research increasingly uses pretrained LLMs to generate image layouts, but this work applies LLMs to structured video plans for controllable long-video generation.The claimed distinction is the use of LLM-generated structure for video rather than image layouts alone.

3 VIDEODIRECTORGPT

VideoDirectorGPT first converts a prompt into a structured video plan, then uses Layout2Vid to generate videos under layout and consistency constraints. The generator is adapted efficiently from a pretrained T2V model using image-level supervision.

  • Video planning: GPT-4 expands a single prompt into a video plan with scene descriptions, entity names and boxes, backgrounds, and consistency groupings.The plan is generated through separate prompting steps for scene structure and layouts.
  • Video planning: Entity consistency groupings record the scenes in which each entity should remain visually consistent.The grouping is formed by matching entities across scenes, such as a chef appearing in scenes 1–4.
  • Video planning: Entity layouts are generated as bounding boxes for sampled frames and linearly interpolated to provide layouts for denser frames.The procedure generates layouts for nine frames per scene before interpolation.
  • Layout2Vid: Layout2Vid adds explicit spatial control to a pretrained ModelScopeT2V backbone through guided attention and trainable parameters.The original spatio-temporal generation backbone is retained while layout control is injected.
  • Layout2Vid: Entity grounding tokens fuse image embeddings, text embeddings, and Fourier features of bounding boxes for layout-guided generation.Learnable projections are applied to the visual and textual features before fusion.
  • Layout2Vid: At inference, entity image embeddings can be obtained from text through unCLIP Prior or from user-provided exemplars.Shared entity representations support temporal consistency across scenes.
  • Training: The training procedure updates only guided 2D attention parameters, enabling image-level layout supervision while retaining multi-scene consistency through shared grounding tokens.The approach freezes the remaining pretrained components during this efficient adaptation.

4 Experimental Setup

The experiments evaluate VideoDirectorGPT across single-scene and multi-scene video generation using automated metrics, human judgments, and step-by-step analysis. The setup measures video quality, text-video alignment, layout and movement control, and temporal consistency of objects across scenes.

  • Evaluated models: The study compares VideoDirectorGPT primarily with ModelScopeT2V and additionally with AnimateDiff, I2VGen-XL, and SVD on MSR-VTT and UCF-101.
  • Single-scene evaluation: Single-scene evaluation measures layout skills, object dynamics, and open-domain generation using VPEval, ActionBench-Direction, MSR-VTT, and UCF-101 prompts.
  • Multi-scene evaluation: Multi-scene evaluation uses ActivityNet Captions, Coref-SV, and HiREST prompts covering event lists, coreference-based descriptions, and single-sentence video generation.
  • Automated metrics: Automated evaluation uses FID, FVD, IS, and CLIPSIM for video quality and text-video alignment, alongside skill-specific layout measures.
  • Automated metrics: Object movement direction accuracy evaluates whether detected target objects move in the prompted direction using their first- and last-frame locations.
  • Automated metrics: Multi-scene object temporal consistency averages CLIP image-embedding similarities between target-object crops from adjacent scenes.The target object is detected in each scene’s center frame before embedding similarities are computed.
  • Human and error analysis: Human evaluation compares generated multi-scene videos on quality, text-video alignment, and object consistency, while error analysis examines descriptions, layouts, groupings, and final videos.

5 Results and Analysis

VideoDirectorGPT improves controllable video generation across single-scene and multi-scene settings, while preserving entity consistency across scenes. Its ablations and error analysis identify benefits from LLM planning and remaining limitations in layout-guided synthesis.

  • Single-scene video generation: VIDEODIRECTORGPT significantly outperforms ModelScopeT2V and other T2V/I2V models on all VPEval layout-control skills.The results include object placement, count, spatial, and scale control, with examples requiring physics understanding such as gravity and perspective.
  • Single-scene video generation: VIDEODIRECTORGPT outperforms competing T2V/I2V models in movement-direction accuracy on ActionBench-Direction prompts.The benchmark evaluates four directions, with 25% movement accuracy corresponding to random guessing.
  • Single-scene video generation: On MSR-VTT, the method improves FVD while remaining competitive in FID and CLIPSIM, and on UCF-101 it reduces FVD from 1093 to 748 versus ModelScopeT2V.Its open-domain performance is competitive with models trained using larger video datasets or higher resolution.
  • Multi-scene video generation: VIDEODIRECTORGPT outperforms ModelScopeT2V in multi-scene visual quality and object temporal consistency on ActivityNet Captions and Coref-SV.On Coref-SV, the advantage remains when co-reference pronouns are replaced with ground-truth object names.
  • Multi-scene video generation: On HiREST, VIDEODIRECTORGPT achieves better visual quality than ModelScopeT2V, while Layout2Vid further improves over a benchmark using GPT-4-generated scene descriptions alone.Examples show consistent step-by-step videos, whereas ModelScopeT2V repeatedly generates the final cake and becomes visually inconsistent.
  • Additional analysis: LLM choice affects planning quality: zero-shot LLaMA2-13B successfully parses only 82 of 600 layouts and performs worse across reported metrics, while fine-tuning improves scores.GPT-3.5-turbo is similar to GPT-4 on MSR-VTT but significantly worse on ActionBench-Directions, indicating weaker handling of strong layout-control prompts.
  • Limitations: The main generation limitation is that overlapping bounding boxes can weaken object control, while layout-guided video generation scores drop from 4.52 to 3.61 in error analysis.The authors suggest stronger T2V backbones could improve final video accuracy.

6 Conclusion

VideoDirectorGPT combines LLM-based multi-scene planning with Layout2Vid for grounded video generation with explicit spatial and temporal entity control. The framework uses GPT-4-generated plans and modifies a ModelScopeT2V backbone with guided attention.

  • Framework: VideoDirectorGPT decomposes generation into video planning and grounded video generation for consistent multi-scene videos.GPT-4 creates a multi-component script, while Layout2Vid renders videos with layout and consistency control.
  • Video planning: The video planner expands a single prompt into scene descriptions, entity names and layouts, backgrounds, and consistency groupings.Planning uses two prompting steps with distinct in-context examples.
  • Grounded generation: Layout2Vid uses a frozen ModelScopeT2V backbone and adds trainable control parameters for layout-guided generation.The supplied passages describe the added control modules but do not specify the complete training setup here.
  • Grounded generation: Layout2Vid adds explicit spatial control through guided 2D attention within ModelScopeT2V’s spatio-temporal diffusion blocks.The blocks combine spatial and temporal convolution with spatial and temporal attention.

D.3 Training and Inference Details

The supplementary details specify Layout2Vid’s architecture and inference setup, alongside human-in-the-loop video-plan editing. The appendix also documents model settings, samplers, and editable generation examples.

  • Implementation details: Layout2Vid’s architecture and training/inference settings are summarized in Table 7.The passage identifies the table as the source of model architecture and parameter details.
  • Training: Layout2Vid is trained using image-level data with bounding-box annotations rather than video annotations.Training uses 0.64M GLIGEN images, 50k steps, and 8 A6000 GPUs over two days.
  • Inference: Inference uses CLIP image embeddings to preserve the visual consistency of the same object across scenes.The same image embedding is reused across scenes for an entity.
  • Human-in-the-loop editing: Users can edit video plans by adding, deleting, or replacing entities, changing backgrounds, and modifying entity bounding-box layouts.Examples use the prompt “A horse running” with altered object sizes and backgrounds.

F Experiment Setup

The experiments evaluate VideoDirectorGPT against established video-generation models across layout control, object dynamics, open-domain quality, and multi-scene consistency. The setup uses dedicated single-scene and multi-scene prompt collections.

  • Evaluated models: The study compares VideoDirectorGPT with six popular T2V models, primarily using ModelScopeT2V as the closest baseline.Layout2Vid reuses ModelScopeT2V’s frozen weights while adding spatial control and temporal consistency.
  • Single-scene evaluation: Single-scene evaluation covers object, count, spatial, and scale layout skills, movement directions, and open-domain video generation.The benchmarks include VPEval, ActionBench-Direction, and MSR-VTT.
  • Single-scene evaluation: ActionBench-Direction uses 400 prompts with balanced movement directions and diverse objects.The prompts are sampled from ActionBench-SSV2 captions.
  • Multi-scene evaluation: Multi-scene evaluation uses ActivityNet Captions, Coref-SV, and HiREST to test event sequences and cross-scene entity consistency.Inputs include both event-caption lists and single sentences expanded into multi-scene videos.
  • Comparison protocol: ModelScopeT2V is matched to the number of scenes generated by the planner for comparative multi-scene evaluation.An additional ablation supplies ModelScopeT2V with LLM-generated prompts instead of generic step labels.

G Human Evaluation Details

The evaluation details define human-preference criteria, intermediate accuracy measures, and ablations of planning and grounding choices. Error analysis identifies layout-guided video generation as the largest accuracy drop.

  • Human evaluation: Human evaluation compares VideoDirectorGPT and ModelScopeT2V on Quality, Text-Video Alignment, and Object Consistency.The study uses Coref-SV videos and evaluates whether target entities remain visually consistent across scenes.
  • Human evaluation: Ten crowd-annotators rate each of 50 videos, with model identities hidden and quality controls applied to participants.The study reports 28 unique annotators and randomized video presentation.
  • Error analysis: The error analysis scores scene descriptions, layouts, consistency groupings, and final videos on a 1-5 Likert scale.An expert annotator evaluates 50 prompts/videos, with sampled scenes used for layout and final-video scoring.
  • Error analysis: 4.52 →3.61 is the largest step-wise drop, occurring between layout-guided planning outputs and final video generation.The planning components reach scores up to 4.52, while the final layout-guided generation score is 3.61.
  • Ablations: Ablations vary the planning LLM, denoising steps, grounding embeddings, and layout representation formats.The LLM comparison includes GPT-3.5-turbo and LLaMA2 models in addition to GPT-4.
  • Ablations: The grounding ablation finds CLIP image-text joint embeddings outperform image-only and text-only embeddings.The joint embedding is selected for the default configuration.

H.3 Layout Control: Bounding Box v.s. Center Point

Bounding-box layouts provide richer guidance than center-point layouts, improving visual quality and text-video alignment. Training only the added gated self-attention layers outperforms unfreezing the entire guided 2D attention module.

  • Layout representation: Bounding-box guidance improves visual quality and text-video alignment over both no-layout and center-point representations.Bounding boxes encode object shape, size, and aspect ratio.
  • Trainable layers: Unfreezing the entire guided 2D attention module significantly worsens performance relative to training only the additional gated self-attention layers.The decline is attributed to misalignment between spatial and temporal layers when temporal components are activated during inference.

I Additional Experiments

Additional experiments evaluate UCF-101 performance, denoising trade-offs, physical-motion planning, and movement across object types. The framework improves UCF-101 FVD over ModelScopeT2V while balancing layout adherence against visual quality.

  • UCF-101: 748 FVD versus ModelScopeT2V’s 1093 shows a substantial UCF-101 improvement, while Inception Score remains competitive at 19.42 versus 19.49.The evaluation uses 2048 randomly sampled UCF-101 test prompts.
  • Layout-control strength: Higher α increases layout control but can reduce visual quality during two-stage denoising.Layout-guided denoising is applied for α*N steps, with the remaining steps using unguided denoising.
  • Layout-control strength: An α value between 0.1 and 0.3 usually offers the best trade-off between visual quality and adherence to bounding-box layouts.The figure compares objects generated with different layout-control strengths.
  • Physical and object movement: The GPT-4 video planner generates movements requiring physical understanding, including gravity and perspective.Examples include a stone thrown into the sky and a car approaching from a distance.
  • Layout-control strength: A small α such as 0.1 guides movement without forcing strict bounding-box adherence, supporting greater creativity, diversity, and visual quality.The analysis describes this as a balance between movement guidance and flexible generation.
  • Physical and object movement: For static objects, movement is often conveyed through the camera, whereas naturally movable objects visibly move in the generated videos.Examples include a bottle, airplane, and boat prompted to move from left to right.
  • Multi-scene generation: The framework includes a multi-scene example generated from the prompt “A boy playing in sand and flying kites.”

J.2 Additional Qualitative Examples

Qualitative examples show the planner and Layout2Vid handling physical motion, custom entity grounding, spatial and count relations, directional actions, and cross-scene consistency. Across these examples, the framework produces detailed plans and preserves entities or scene attributes more reliably than ModelScopeT2V.

  • Physical motion: The planner generates physically informed movements involving gravity and perspective, extending control beyond static spatial placement.
  • Object movement: For movement prompts, static-object motion is often represented through camera movement, while airplanes and boats show object motion directly.
  • Custom entity grounding: Text-only and image-plus-text inputs both support custom entity grounding with identities preserved across multiple scenes.
  • Cross-scene consistency: The framework preserves the boy’s appearance across scenes in a multi-scene video involving human appearance.
  • Spatial and count skills: LLM-generated plans guide correct spatial relations and object counts, while ModelScopeT2V misses a pizza and overproduces frisbees.
  • Directional action: The planned layouts place the stuffed animal and pear correctly and guide their movement toward target directions, unlike ModelScopeT2V’s stationary or random motion.
  • Cross-scene consistency: The framework maintains the same brown dog and snow across scenes, whereas ModelScopeT2V changes dogs and loses the snow after the first scene.
  • Detailed multi-scene plans: The planner expands a strawberry-surprise prompt into a step-by-step plan, while Layout2Vid generates the complete consistent process instead of repeatedly generating strawberries.

K Limitations

The framework shares limitations with its underlying T2V backbone and introduces practical concerns around misuse and LLM inference cost. Real-world deployment therefore requires caution and human supervision.

  • Safety and deployment: Real-world use requires caution and human supervision because the framework could create false information or misleading videos.The paper points to human-in-the-loop video-plan editing as one supervision approach.
  • Practical cost: Generating video plans with the strongest LLM APIs can be costly, although quantization, distillation, and open-source models may reduce inference costs.
  • Model scope: Layout2Vid inherits backbone limitations, including training-distribution deviations, imperfect generation quality, and understanding only the English corpus.
Loading 2309.15091v2…