Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs

1. Introduction

• Existing diffusion-based T2V models can generate high-resolution frames but may still produce unsmooth transitions, crude motion, and disordered action occurrence.The paper identifies a mismatch between succinct, abstract language and the specific, often redundant frames needed to depict actions.Source: §1 · p. 1
• Effective complex-action T2V requires ordering events chronologically, enriching scenes beyond prompt details, structuring scene semantics, and modeling fine-grained spatio-temporal features.These principles are motivated by a human-like process of arranging an event sequence and imagining specific scenes.Source: §1 · pp. 1–2
• Dysen uses an LLM to extract and order key actions, convert them into dynamic scene graphs, and enrich graph scenes before integrating their features into a video diffusion backbone.The paper describes ChatGPT-based action planning and scene imagination using in-context learning.Source: §1 · p. 2, Figure 2 · p. 2
• Dysen-VDM consistently outperforms existing state-of-the-art methods on UCF-101, MSR-VTT, and ActivityNet in automatic and human evaluations.The reported gains are especially pronounced for complicated actions.Source: §1 · p. 2
• The paper presents LLM-based action planning and scene imagination, dynamic-scene controllability through scene graphs, and benchmark performance as its contributions.It characterizes its LLM use for T2V dynamics understanding as a first attempt.Source: §1 · p. 2

2. Related Work

• Prior T2V methods include GANs, VAEs, flow-based models, and autoregressive models, followed more recently by diffusion methods.The passage describes GANs as vulnerable to mode collapse and diffusion models as offering high-quality samples with wide mode coverage.Source: §2 · p. 3
• Diffusion-based T2V methods are described as providing more stable training, while latent diffusion models reduce computation by learning in a low-dimensional latent space.The paper adopts latent diffusion as its backbone synthesizer.Source: §2 · p. 3
• Prior dynamics-aware T2V approaches are characterized as limited to coarse operations, while much diffusion-based work prioritizes frame resolution over fine-grained action dynamics.The paper proposes fine-grained spatio-temporal feature modeling based on dynamic scene graphs.Source: §2 · p. 3

3.1. Text-to-video Latent Diffusion Model

• A latent diffusion model encodes video into a lower-dimensional latent space, progressively adds Gaussian noise, and learns a text-conditioned reverse denoising process.A text encoder conditions the denoising network, commonly implemented as a 3D-UNet.Source: §3.1 · p. 3

3.2. Dynamic Scene Graph Representation

• A dynamic scene graph is a time-ordered list of per-frame scene graphs whose object, attribute, and relation nodes describe spatial semantics and temporally consistent actions.Relations form subject-predicate-object triplets, while attributes modify objects.Source: §3.2 · p. 3

4. Methodology

• The framework uses an LDM backbone guided during denoising by Dysen’s capture of the input text’s spatiotemporal characteristics.Dysen-VDM’s training then pre-trains the backbone, further pre-trains it for text-conditioned video generation with DSG annotations, and updates the overall system with Dysen.Source: §4 · p. 3, §4.3 · p. 5

4.1. Dynamic Scene Manager

• Dysen uses ChatGPT with in-context learning to plan key actions, convert their timed events into a frame-wise DSG skeleton, and enrich scenes through local and global refinement.Action scenes are represented by an agent, event predicate, target, and start/end interval; scene enrichment uses neighboring contexts to support coherent transitions.Source: §4.1 · pp. 3–5

4.2. Scene Integration for T2V Generation

• A recurrent graph Transformer encodes the enriched DSG’s spatial and temporal features, which are fused into the 3D-UNet decoder through cross-attention alongside raw text features.The raw text representation is encoded by CLIP.Source: §4.2 · p. 5

5.1. Setups

• The setup uses UCF-101 action labels, MSR-VTT caption prompts, and ActivityNet videos with at least three actions; sampling uses 16 keyframes per two-second clip.The experiments include zero-shot and fine-tuned settings, use ChatGPT GPT-3.5 turbo by default, and evaluate with dataset-specific automatic metrics and human ratings.Source: §5.1 · pp. 5–6

5.2. Main Comparisons and Observations

• 95.23 IS and 255.42 FVD on fine-tuned UCF-101 establish Dysen-VDM as a new state-of-the-art; zero-shot results also lead on UCF-101 IS/FVD and MSR-VTT metrics.Table 1 reports zero-shot results on UCF-101 and MSR-VTT; Table 2 covers fine-tuning on UCF-101 without pre-training.Source: Table 2 · p. 6, §5.2 · p. 6, Table 1 · p. 6

5.3. Results on Action-complex T2V Generation

• Dysen-VDM outperforms baselines across three ActivityNet tests, with the advantage clearer for more co-occurring actions, longer prompts, and longer videos.The comparisons include CogVideo, VDM, and Latent-VDM; the paper notes that Dysen-VDM surpasses pre-trained CogVideo without additional pre-training.Source: Figure 5 · p. 6, §5.3 · p. 7

5.4. Human Evaluation

• On ActivityNet human evaluations, Dysen-VDM scores 92.4 for scene richness, 17.4 above CogVideo, and exceeds Latent-VDM in action faithfulness by 15.9.The evaluation rates action faithfulness, scene richness, and movement fluency; the paper cautions that standard automatic metrics may not fully assess temporal dynamics.Source: Table 3 · p. 7, §5.4 · p. 7

5.5. System Ablations

• Removing Dysen causes the largest ablation loss, increasing FVD by 90.98 on UCF-101 and 141.82 on ActivityNet; removing scene imagination is the next most damaging change.The remaining ablations also report harm from removing sliding-window context, RL optimization for in-context learning, or RGTrm.Source: Table 4 · p. 7, §5.5 · p. 7

5.6. Qualitative Results

• In qualitative comparisons, Dysen-VDM produces smoother frames and correctly ordered motions, while LVDM shows jumpy transitions and less delicate scenes with erroneous actions.The paper also notes that a baseline can miss predicates in the prompt, including VDM missing “man walks” in prompt #1.Source: Figure 6 · p. 7, §5.6 · p. 8

5.7. In-depth Analyses

• The TriRec. analysis finds Dysen-VDM has the highest subject-predicate-object triplet recall among two baselines, with clear margins.TriRec. measures recall of fine-grained subject-predicate-object structures between input-text scene graphs and video-frame scene graphs.Source: Figure 7 · p. 8, §5.7 · p. 8
• Scene imagination increases both object-attribute and subject-predicate-object structures; GPT-3.5 and GPT-4 also show satisfactory action-annotation accuracy, with GPT-4 slightly ahead.Object-attribute structures reflect static content, while subject-predicate-object structures reflect dynamic scenes; the action-parsing comparison uses 100 MSR-VTT samples.Source: §5.7 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8

• Dysen extracts key actions in time order, converts them into dynamic scene graphs, and enriches their scenes with reasonable details.Source: §6 · p. 8
• ChatGPT-based in-context learning supports Dysen's temporal-dynamics understanding, with its prompting optimized via reinforcement learning.Source: §6 · p. 8
• A recurrent graph Transformer learns fine-grained spatiotemporal features from the resulting DSG and integrates them into the backbone video diffusion model.Source: §6 · p. 8
• Experiments on three T2V datasets report new best results, with stronger performance especially for scenarios with complex actions.Source: §6 · p. 8

• This appendix section provides extended technical details of the proposed methods.Source: A. Extended Technical Specification · p. 14

• A video encoder maps frames into a lower-dimensional latent space, and a decoder maps the latent representation back to video.Source: §A.1 · p. 14
• The forward diffusion process gradually corrupts the compressed video latent into Gaussian noise through a Markov chain over T steps.Source: §A.1 · p. 14
• The text-conditioned reverse process progressively denoises the latent, while training estimates added noise from a noisy latent, timestep, and text conditioning.Source: §A.1 · p. 14
• The 3D-UNet extracts spatial features with shared 3D convolutions, then applies temporal attention across frames; a lower-triangular mask restricts each token to previous tokens.Source: §A.1 · p. 14

A.2. Specification on Dynamic Scene Graph Representation

• A dynamic scene graph is a time-ordered list of frame-level scene graphs containing objects, attributes, and subject-predicate-object relations, with structure consistent across frames.Source: §A.2 · p. 14

A.3. Prompting ChatGPT with In-context Learning

• ChatGPT prompts combine an instruction, input-output demonstrations, and a test instance; the described ICL setup uses five demonstrations per task.Source: §A.3 · pp. 14–15
• Action-planning demonstrations use ChatGPT-extracted major events as gold annotations and a video moment-localization model to assign event start and end positions.Source: §A.3 · p. 15
• Step-wise imagination replaces raw scene-graph triplets with outputs covering additions and changes; global-polishment demonstrations use externally parsed DSG annotations, filtering labels occurring more than 500 times in training.Source: §A.3 · pp. 15–16

• The section extends the description of the experimental settings.Source: B. Experiment Specification · p. 16

B.1. Evaluation

• Automatic evaluation uses IS and FVD on UCF-101, and FID and CLIPSIM on MSR-VTT; Triplet Recall measures relation-triplet recall between input-text and generated-video scene graphs.Figure 14 illustrates in-context learning for global scene polishing.Source: §B.1 · pp. 16, 18, Figure 14 · p. 17
• Human evaluation on ActivityNet uses trained participants’ ratings of action faithfulness, scene richness, and movement fluency.The study samples 50 text-video pairs and averages ratings from ten participants using a 10-point Likert scale.Source: §B.1 · p. 17

B.2. Baseline Specification

• Baselines are grouped into diffusion-based and non-diffusion-based T2V methods, including VDM and LVDM, and VideoGPT and TGANv2.Source: §B.2 · p. 18

C.1. Visualization of DSG-guided Controllable Video Synthesis

• Figure 15 visualizes generated video frames together with the dynamic scene graphs induced and enriched by Dysen.The illustrated prompts include a woman helping a boy cross obstacles and a cat falling from a table.Source: §C.1 · pp. 18–19, Figure 15 · p. 19

• LLM hallucinations can cause scene-understanding errors and lower-quality imagined DSGs, which in turn affect generated-video quality.Source: §C.2 · p. 20
• Structured triplet representations struggle with abstract scenes such as cartoons and may not improve artistic styles in some frames.Source: §C.2 · p. 20
• Despite these failures, the authors report that Dysen-VDM advances most T2V scenarios and that detailed DSG representations support controllable content generation and high-quality video dynamics.Source: §C.2 · p. 20