LLM-grounded Video Diffusion Models

1 INTRODUCTION

• Text-to-video models struggle to produce realistic spatial layouts and temporal dynamics that align with prompts.Source: §1 · p. 2
• Prior text-to-image work uses LLM-generated spatial layouts, but whether LLMs can generate video layouts across both spatial and temporal dimensions had not been demonstrated.Such motion depends on physical properties and object attributes.Source: §1 · p. 2
• LVD uses LLM-generated Dynamic Scene Layouts as an intermediate representation and attention guidance to control object relations and motion without training.Source: §1 · p. 2
• LVD significantly improves text-video alignment over several strong baselines and consistently improves the base diffusion model on common datasets and evaluator assessment.The evaluation includes a five-task benchmark, UCF-101, and MSR-VTT.Source: §1 · p. 2
• The paper contributes a benchmark for prompt-video alignment and reports that text-only LLMs generate layouts that generalize to unseen spatiotemporal sequences.Source: §1 · p. 2

2 RELATED WORK

• Controllable diffusion methods add spatial or other control signals, but complex video dynamics from text alone remain unresolved.Examples include ControlNet, GLIGEN (Li et al., 2023), and Control-A-Video.Source: §2 · p. 2
• Prior text-to-video models still struggle with reasonable dynamics, with limited paired training data and substantial training costs identified as challenges.Source: §2 · pp. 2–3
• LLM-based layout methods show that generated boxes can guide text-to-image models, while LVD investigates spatial and temporal video dynamics.VideoDirectorGPT (Lin et al., 2023) instead explores training diffusion adapters; LVD uses inference-time guidance.Source: §2 · p. 3

3 CAN LLMS GENERATE SPATIOTEMPORAL DYNAMICS?

• The LLM prompt combines task instructions with examples of desired outputs and rules, and asks for a brief reasoning statement before box generation.Instructions specify the coordinate system, canvas size, frame count, and frame speed.Source: §3 · p. 4
• A Dynamic Scene Layout represents objects as bounding boxes linked across frames, with numerical locations and sizes, text descriptions, and IDs matching boxes across frames.The method generates visible boxes sequentially and does not use a box tracker.Source: §3 · p. 4
• The investigation tests gravity and elasticity as world properties and perspective projection as a camera property, using GPT-4 and one in-context example for each.The authors note potential to generalize to other uninvestigated properties.Source: §3 · p. 4, Figure 3 · p. 4
• LLMs generate realistic layouts aligned with both seen and unseen scenarios, including distinguishing an elastic ball that bounces from a non-elastic rock that falls.The examples do not explicitly describe the rock’s properties.Source: §3 · p. 4
• The paper reports that videos generated from the layouts align with their prompts, including objects thrown right-to-left and objects viewed from different viewpoints.Source: Figure 5 · p. 5
• LLMs extend sparse examples to unmentioned properties, accounting for air friction in a paper airplane’s motion and applying perspective geometry when objects move relative to a stationary camera.Source: §3 · p. 5

• LVD uses an off-the-shelf text-to-video diffusion model without fine-tuning, avoiding the need for instance-annotated training data.The pipeline is implementation-agnostic, and its LLM and layout-conditioned video generator stages can be swapped for counterparts.Source: §4 · pp. 5–6
• DSL guidance alternates with denoising and uses gradient descent on the partially denoised video to steer generation toward layout constraints.The guidance minimizes an energy function E.Source: Figure 6 · p. 6, §4 · p. 6
• Etopk encourages high cross-attention inside each box and low attention outside, while ECoM encourages matching box and attention positions and velocities across frames.The overall energy is a weighted sum of Etopk and ECoM; ECoM addresses imprecise control when consecutive boxes overlap.Source: §4 · p. 6
• The Table 2 caption reports that LVD follows prompt spatiotemporal keywords better than its underlying ModelScope diffusion model at 256 × 256 resolution.Source: Table 2 · p. 7

5 EVALUATION

• The benchmark covers five prompt tasks with 500 prompts and 1,000 video generations, alongside UCF-101, MSR-VTT, and evaluator-based assessments.The setup compares LVD with ModelScope without DSL guidance and a retrieval-based baseline that builds and tracks layouts from retrieved videos.Source: §5 · p. 7

5.1 EVALUATION OF GENERATED DSLS

• GPT-3.5 DSLs achieved 77% average accuracy and GPT-4 DSLs achieved 98%, substantially exceeding the retrieval-based layout baseline.Only three in-context examples were used throughout the evaluation; adding retrieved examples led to similar performance.Source: §5.1 · p. 7

5.2 EVALUATION OF GENERATED VIDEOS

• Adding ECoM further boosts alignment on sequential tasks, whereas Eratio and Ecorner do not improve alignment over Etopk alone.The ablation uses the same GPT-4-generated layouts across energy-function variants.Source: Table 3 · p. 8, §5.2 · p. 8
• LVD significantly outperforms ModelScope and the retrieved-layout baseline by a large margin in video-text alignment.The Table 2 caption identifies ModelScope as the underlying diffusion model and reports better adherence to prompt spatiotemporal keywords.Source: §5.2 · p. 8
• GPT-3.5 layouts yield better video-text alignment on three tasks, which the authors attribute to less motion being easier to render; the results indicate video generation is the bottleneck.Source: §5.2 · p. 8
• LVD improves FVD video quality over ModelScope on both UCF-101 and MSR-VTT, though both methods outperform Zhou et al. on MSR-VTT and trail it on UCF-101.Lower FVD scores indicate better quality.Source: §5.2 · p. 8, Table 4 · p. 8
• LVD more consistently generates the correct object count, introduces objects mid-video, follows sequential directions, and handles multiple objects than ModelScope.In one example, LVD depicts a blue dog and brown ball correctly, while ModelScope assigns the dog’s color to the ball.Source: §5.2 · p. 8
• Evaluators found LVD better or at least on par with the baseline in 96.8% of alignment cases and 85.5% of visual-quality cases.The assessment compared randomly shuffled pairs of videos, with 10 participants evaluating each of 20 pairs.Source: §5.2 · pp. 8–9

• LVD handles complex prompts without updating LLM or diffusion-model parameters, and the paper aims to demonstrate LLM spatiotemporal modeling and DSL-based alignment improvements.Source: §6 · p. 9
• LVD inherits base-model limitations in synthesizing objects or artistic styles that the underlying model struggles with.The authors suggest LoRA fine-tuning as a possible mitigation and leave exploration for future work.Source: §6 · p. 9
• The DSL-to-video stage remains simple; the authors anticipate that a more advanced DSL-conditioned generator may improve control precision and video quality.They also identify finer-grained control, including human poses, as a potential extension.Source: §6 · p. 9

• GPT-4’s statements invoke bird flight, barrel buoyancy, a cat’s reaction, and the rock’s non-bouncing motion to explain generated DSLs.Source: §A.1 · p. 14
• For the rock example, an increasing y-coordinate is consistent with upward motion because the paper’s coordinate system runs top-to-bottom.Source: §A.1 · p. 14
• GPT-4 reasoning statements from the ChatGPT interface are often longer than those from the API; the authors hypothesize that interface prompting encourages more explanation.Source: §A.1 · p. 14

Handling ambiguous prompts

• For “a walking dog,” the LLM assumes a direction missing from the prompt and generates a reasonable video from that layout.The assumption is included in the LLM’s reasoning statement.Source: §A.2 · p. 14

Handling overlapping boxes

• The video stage produces realistic raccoon-on-barrel imagery despite the raccoon’s layout box overlapping about half of the barrel.The resulting motion also remains plausible.Source: §A.2 · p. 14

Handling dynamic background

• LVD generates a changing background for a lightning-and-rain scene despite inheriting ModelScope’s preference for backgrounds with little motion.Rain appears even though only lightning was mentioned, an assumption inherited from the base model.Source: §A.2 · p. 14

A.3 OUR PROMPTS AND IN-CONTEXT EXAMPLES FOR DSL GENERATION

• Table 10 presents the prompts used for DSL generation in the quantitative evaluation.The prompt template is populated with the text query before being passed to an LLM.Source: §A.3 · p. 14
• The appendix includes additional LVD visualizations and a table of reasoning statements from GPT-4.Figure 8 illustrates ambiguous prompts, overlapping boxes, and dynamic backgrounds.Source: Table 6 · p. 15, Figure 8 · p. 16
• The API setup assigns system, user, and assistant roles to the instructions, queries, and example outputs, while the web interface merges them into one message.Benchmark queries use API calls; the visualizations in Figure 1 use GPT-4 through the ChatGPT web interface.Source: §A.3 · p. 16

Setup

• Inference uses 40 DPMSolverMultiStep denoising steps, with DSL guidance applied five times per step during only the first ten steps.The energy is scaled by 5, with background and foreground weights of 4.0 and 1.0, respectively.Source: §A.4 · pp. 16–17

Proposed benchmark

• The proposed benchmark covers generative numeracy, attribute binding, visibility, spatial dynamics, and sequential actions across 500 prompts and 1,000 generated videos.Each task has 100 programmatically generated prompts, and each prompt receives two videos.Source: §A.4 · p. 17

UCF-101 and MSR-VTT benchmarks

• UCF-101 evaluation compares FVD for 10,000 generated 16-frame videos against the training set, while MSR-VTT uses 2,048 generated videos against its test set.For MSR-VTT, layouts are generated from captions in the 2,990-video test set before 2,048 are sampled.Source: §A.4 · p. 17

A.5 FAILURE CASES

• LVD can guide apple motion in cartoon videos but inherits the base model’s poor cartoon quality, including malformed apples and unsuitable background textures.Domain-specific diffusion fine-tuning is outside this work’s scope.Source: §A.5 · p. 17, Figure 10 · p. 18
• Cross-attention energy minimization can hide a barrel with leaves and pop another from the water instead of moving the drifting barrel rightward.Reducing ambiguity in cross-attention-based control is left for future work.Source: §A.5 · p. 18, Figure 11 · p. 18
• LLM-generated layouts can miss long-horizon motion: a ball intended to reach the upper left stops only slightly above the frame’s middle.Independent LLM queries account for most fluctuations in the layout-generation stage’s performance.Source: §A.5 · p. 18, Figure 9 · p. 18

A.6 ADDITIONAL QUALITATIVE RESULTS WITH HIGHER VIDEO RESOLUTION

• At 512 × 512, the full method surpasses the unguided baseline by a large margin, while removing spatial normalization from Ecorner significantly degrades alignment versus Etopk alone.The reported performance gap over baselines grows with resolution; the authors suggest easier motion control and more accurate detection boxes as possible reasons.Source: §A.6 · pp. 18–19, Table 7 · p. 19

A.7.1 ABLATIONS ON THE LLM-GROUNDED DSL GENERATION STAGE

• GPT-4 retains high-quality DSL generation with one in-context example, but performance degrades slightly and fluctuates more; GPT-3.5 can fail to produce parseable layouts.Three examples provide stable performance, while adding more yields no additional benefit.Source: §A.7.1 · p. 19, Table 8 · p. 19

A.7.2 ABLATIONS ON THE DSL-GROUNDED VIDEO GENERATION STAGE

• Average accuracy improves from one to five DSL guidance steps and plateaus beyond five, making five the selected performance-efficiency trade-off.The higher-resolution results also report that adding Ecorner gives similar text-video alignment, whereas removing its spatial normalization degrades alignment versus Etopk alone.Source: §A.6 · pp. 18–19, §A.7.2 · p. 19, Table 9 · p. 19