PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphVid: Interactive Graph-Controllable Video Generation

Vedant Shah, Onkar Susladkar, Tushar Prakash, Kiet Nguyen, Tianjio Yu, Adheesh Juvekar, Muntasir Waheed, Ismini Lourentzou

arXiv:2607.21580v1cs.CVcs.AI

TL;DR

Precise control of multi-object interactions remains limited by low-level motion cues, cumbersome trajectories, and costly annotation requirements. GraphVid uses directed interaction graphs with a lightweight graph-to-token adapter and LoRA conditioning for image-to-video generation, achieving substantial gains over Motion-I2V, including 39.9% lower FID and 37.6% lower FVD.

  • Problem

    Existing controllable video methods provide limited object-level interaction control and often depend on costly, difficult-to-scale motion or physics annotations.

  • Method

    GraphVid represents scene dynamics as directed interaction graphs and conditions a frozen video diffusion transformer through graph-to-token adaptation and parameter-efficient LoRA.

  • Results

    39.9% lower FID and 37.6% lower FVD, with PSNR improving from 9.87 to 15.98 and SSIM from 0.38 to 0.61 versus Motion-I2V.

  • Takeaways & Limitations

    The results support directed interaction graphs as a structured interface for controllable multi-object video generation.

Abstract

from arXiv · show

Controllable video generation remains challenging due to the difficulty of specifying precise multi-object interactions using text prompts or motion-control inputs that primarily constrain pixel movement. In practice, trajectory-based control often requires users to draw accurate tracks for multiple objects, which scales poorly with scene complexity and becomes ambiguous under occlusion or overlap. To enable flexible yet precise multi-subject control, we introduce $\textbf{GraphVid}$, a graph-conditioned image-to-video generation model that enables interactive control through structured interaction graphs. We further curate $\textbf{GraphVid-Bench}$, a large-scale interaction-centric video dataset with structured relational annotations to enable training of interaction-aware video generation models. Despite using substantially less training data and fewer trainable parameters than prior motion-control methods, GraphVid delivers strong controllability and video quality. Compared with Motion-I2V, GraphVid reduces FID by up to 39.9% and FVD by 37.6%, while improving PSNR (9.87=>15.98) and SSIM (0.38=>0.61). Our results highlight the potential of structured semantic interfaces as a powerful paradigm for controllable video generation.

1 Introduction

GraphVid addresses the limitations of trajectory- and physics-conditioned controllable video generation with interactive, graph-based multi-object control from a single image. It achieves strong controllability and video quality while outperforming prior baselines on FID, FVD, PSNR, and SSIM.

  • Motivation: Trajectory- and physics-conditioned methods rely on costly dense motion, trajectory, optical-flow, or physics annotations and large curated datasets.Wan-Move, Motion Prompting, and Motion-I2V use 2M, 2.2M, and 10M training samples, respectively.
  • Method: GraphVid enables interactive multi-object dynamics control through directed interaction scene graphs constructed from a single input image.Users edit relational interactions over detected entities, and the edited graph is translated into conditioning for video generation.
  • Results: Compared with WISA, GraphVid reduces FID by 34.5% (25.98 →17.02) and FVD by 7.8% (107.89 →99.42), while increasing PSNR by 5.9% (15.08→15.98) and SSIM by 15.0% (0.53→0.61).WISA uses physics-centric text inputs.
  • Results: FID −39.9%, FVD −37.6%, PSNR +61.9%, and SSIM +60.5% are GraphVid’s improvements over Motion-I2V.These results establish GraphVid’s advantage over a trajectory-based baseline across all four reported metrics.
  • Results: FID −30.3%, FVD −10.0%, PSNR +41.7%, and SSIM +12.9% are GraphVid’s improvements over Tora.GraphVid outperforms this trajectory-based baseline on FID, FVD, PSNR, and SSIM.

2 Related Work

Video generation has progressed from temporally extended image diffusion models to latent, motion-conditioned, and large-scale transformer architectures. GraphVid builds on this trajectory by using interaction-grounded graphs and lightweight modules to provide structured control while retaining a video backbone’s learned priors.

  • Video Generation: Video generation evolved from image diffusion models with temporal modules to latent video diffusion, motion-conditioned architectures, and large-scale diffusion transformers.These developments target improved motion modeling, temporal consistency, and scalability.
  • Graph-Conditioned Control: Objects are encoded as node embeddings, relational edges are extracted to form an interaction graph, and edge-aware reasoning produces tokens that guide video generation.A frozen video diffusion transformer uses these conditioning tokens to generate interaction-consistent dynamics.
  • Graph-Conditioned Control: GraphVid uses an interaction-grounded graph and lightweight modules to translate graph cues into conditioning tokens compatible with a video-generation backbone.This design enables structured control while retaining the backbone’s strong priors.

3 Method

GraphVid generates controllable videos from a conditioning image and structured interaction graph by combining edge-aware graph reasoning with a frozen pretrained video diffusion transformer. Lightweight graph-to-token adapters and LoRA modules translate object relationships into semantic conditioning while preserving pretrained generative capabilities.

  • Interaction Graph: GraphVid models video generation as p(I1:T | I0, G), where graph nodes represent detected entities and directed edges encode relations such as push, pull, lift, and hold.The attributed interaction graph captures how entities influence one another over the generated frames.
  • Graph Representations: Object nodes fuse visual-region features, text labels, and normalized bounding boxes, while directed edges use open-vocabulary textual interaction descriptors.These representations jointly encode appearance, semantic identity, spatial grounding, and intended interactions.
  • Edge-Aware Graph Reasoning: The edge-aware GNN injects relational attributes into message passing so node embeddings encode object identity and the physical relationships governing dynamic interactions.GINEConv layers propagate relational information, unlike standard GCNs that primarily treat edges as binary connectivity.
  • Efficient Adaptation and Training: GraphVid keeps the pretrained DiT and VAE frozen while training the GNN, graph-to-adapter MLP layers, and LoRA modules injected into transformer attention projections.LoRA modules adapt Q, K, V, and output projections using only a small number of trainable parameters, and training uses the pretrained model’s flow-matching objective.
  • Graph-to-Token Conditioning: Interaction-aware node embeddings become semantic transformer tokens, concatenated with edge text tokens as encoder hidden states for self-attention over entity interactions.Node sequences are padded to a maximum length Nmax to support scenes with varying numbers of objects.

4 GraphVid-Bench Dataset

GraphVid-Bench is introduced to support structured control of relational scene dynamics beyond text prompts and low-level motion cues. It pairs approximately 27K standardized video clips with directed interaction graphs describing entities and their interactions.

  • Motivation: GraphVid-Bench addresses the limited object-level interaction control offered by text-prompt and low-level-motion video benchmarks.Real-world dynamics are relational, emerging from how entities interact with one another.
  • Dataset Composition: Approximately 27K interaction-centric video clips comprise GraphVid-Bench.The dataset statistics describe approximately 27K curated videos.
  • Dataset Composition: 81 frames at 16 fps standardize every clip for consistent training and evaluation.All clips use the same frame count and frame rate.
  • Graph Annotations: Each video is paired with a directed interaction graph whose nodes represent scene entities and whose edges encode interaction cues.Node attributes include visual features, semantic labels, and spatial information; edges capture interaction type and direction.
  • Inference-Time Control: Users can add interaction edges at inference time to model how scene behavior changes when new interactions are introduced.Such samples help the model learn behavior changes associated with newly introduced interactions.

5 Experiments

GraphVid achieves strong controllability, perceptual quality, temporal consistency, and motion accuracy on interaction-centric benchmarks while using fewer resources than prior methods. Qualitative results and ablations further support interaction graphs, sparse graph conditioning, and higher-rank adapters as effective design choices.

  • Quantitative comparison: GraphVid achieves competitive performance across interaction-centric MoveBench metrics while requiring substantially fewer training samples and trainable parameters than prior methods.The comparison also reports faster inference and identifies WISA as the closest prior method incorporating structured physical knowledge.
  • Quantitative comparison: 39.9% lower FID and 37.6% lower FVD than Motion-I2V, with PSNR improving from 9.87 to 15.98 and SSIM from 0.38 to 0.61.These results indicate better frame reconstruction and perceptual fidelity than trajectory-based control.
  • Multi-object evaluation: 49.45 FID and 291 FVD are GraphVid’s best multi-object results among comparable-scale methods, improving over FlashMotion by 10.1% and 6.4%, respectively.GraphVid also achieves the lowest EPE among comparable-scale methods in the multi-object setting.
  • Qualitative results: Graph interaction conditioning enables precise, interpretable control over object-object, human-object, and articulated human motion while preserving object identity, appearance, and background geometry.The interface binds motion to the correct entities and combines multiple primitives into coherent multi-step dynamics.
  • Graph sparsity ablation: Capping graph size at 30 nodes achieves the best FVD (99.42), outperforming larger capacities because excessive zero-padding dilutes attention and degrades generation quality.As the maximum node budget increases from 30 to 256, FID degrades, consistent with attention dilution from padded tokens.
  • LoRA rank ablation: Increasing LoRA rank from 16 to 128 reduces FID from 18.34 to 17.02 and FVD from 103.18 to 99.42, while increasing PSNR from 13.70 to 15.98 and SSIM from 0.58 to 0.61.Rank 128 is used in all reported experiments.

6 Conclusion · A GraphVid-Bench Details and Curation Pipeline

GraphVid enables controllable image-to-video generation by representing scene dynamics as directed interaction graphs for intuitive multi-object control from a single image. GraphVid-Bench aggregates three complementary open-source datasets covering physics-driven dynamics, commonsense interactions, and complex real-world motion.

  • 6 Conclusion: GraphVid represents scene dynamics through directed interaction graphs.This structured representation supports controllable image-to-video generation.
  • 6 Conclusion: Unlike approaches using dense motion signals or low-level physics annotations, GraphVid models motion as structured interactions between entities.The framework is designed for structured multi-object control rather than low-level motion specification.
  • 6 Conclusion: GraphVid enables intuitive multi-object control directly from a single image.Its interaction-graph formulation connects scene structure with image-to-video control.
  • 6 Conclusion: GraphVid integrates Edge-Aware Graph Reasoning with a frozen video diffusion transformer.The supplied passage identifies this integration as part of the framework architecture.
  • A GraphVid-Bench Details and Curation Pipeline: GraphVid-Bench training data combines WISA-80K, Something-Something v2, and Magic-Data to broaden action and interaction coverage.The datasets are described as complementary open-source sources.
  • A GraphVid-Bench Details and Curation Pipeline: WISA-80K provides rigid-body physics-driven dynamics, while Something-Something v2 covers commonsense human-object and object-object interactions.These sources contribute distinct interaction categories to the training data.
  • A GraphVid-Bench Details and Curation Pipeline: Magic-Data contributes complex real-world foreground motion and multi-interaction scenarios to GraphVid-Bench.The passage identifies Magic-Data as the source for these complex real-world cases.

A.1 Test Set Distillation

The test sets are distilled through dataset-specific filtering and a shared spatio-temporal standardization pipeline. DAVIS 2017 clips are motion-centered and camera-motion filtered, while MoveBench retains discrete object-level interactions and is standardized to 512 × 288 resolution and 81 frames.

  • DAVIS 2017: DAVIS 2017 sequences are converted to video tensors and edge-padded when slightly shorter than 81 frames.Sequences of 78–79 frames are padded using the final frame to satisfy the temporal requirement.
  • DAVIS 2017: DAVIS 2017 sequences longer than the temporal requirement are processed with motion-energy windowing to extract the most dynamic segment.The resulting clips are filtered by a VLM motion classifier to remove sequences dominated by camera or ego-motion.
  • MoveBench: MoveBench videos are filtered by a VLM to retain clips exhibiting discrete object-level physical interactions.Retained videos are resized and standardized to match the downstream model’s visual input requirements.
  • MoveBench: 512 × 288 resolution and 81 frames are used to standardize the retained MoveBench videos.This preprocessing matches the visual input requirements of the downstream model.

B Additional Experimental Details

GraphVid uses a frozen LTX 2B backbone with graph adaptation through an Edge-Aware GNN, a 3-layer MLP adapter, and LoRA modules. Sampling uses continuous time-step FlowMatchEulerDiscreteScheduler inference, with FlashAttention disabled for fair comparison.

  • Model Training: GraphVid trains an Edge-Aware GNN with hidden dimension 512 and projects its graph representation into the LTX 2B backbone’s 4096-dimensional latent space.The projection uses a 3-layer MLP adapter, while the backbone remains frozen throughout training.
  • Model Training: LoRA modules are inserted into to_q, to_k, to_v, and to_out.0 attention projections with rank r = 128 and scaling factor α = 32.This provides parameter-efficient adaptation while keeping the backbone frozen.
  • Sampling: Sampling uses the FlowMatchEulerDiscreteScheduler with continuous time-step sampling, and FlashAttention is disabled during inference for fair comparison with prior methods.These settings define the reported inference procedure and comparison protocol.
  • Environmental Impact: Training uses 8 NVIDIA A100 GPUs for approximately 2.5 days (60 hours), with carbon footprint estimated under a uniform hardware-level accounting protocol.The estimate assumes a maximum thermal design power of 400 W per NVIDIA A100 GPU and compares GraphVid with Wan-Move.

C Additional Experiments

Additional experiments evaluate GraphVid’s efficiency on an interaction-centric DAVIS subset and examine the robustness of its edge-aware graph architecture. The results emphasize competitive performance with reduced resources and isolate semantic edge messaging as an architectural factor.

  • Evaluation on the filtered DAVIS benchmark: 0.6B trainable parameters and 27K training videos support GraphVid’s evaluation on the interaction-centric DAVIS subset.The DAVIS evaluation reports highly competitive performance while using a fraction of the trainable parameters and training data.
  • Edge-Aware GNN Training Robustness: GraphVid’s full model additionally incorporates semantic edge embeddings through message passing, providing the reference configuration for the training-time graph representation ablation.Table 9 compares this full model with variants using only node features.
  • Evaluation on the filtered DAVIS benchmark: An order-of-magnitude reduction in model capacity and training scale still yields strong temporal coherence and structural fidelity among parameter-efficient baselines.The comparison is made against WISA and other parameter-efficient baselines on the filtered DAVIS benchmark.
  • Edge-Aware GNN Training Robustness: The architectural ablation removes semantic edge connections and message-passing layers, retaining only isolated node features from bounding boxes and visual object embeddings.This baseline isolates the contribution of relational edge messaging within GraphVid’s GNN.

D Human Preference Study

A human preference study evaluated whether generated videos captured intended interaction semantics across diverse scenarios. GraphVid received strong user preference, indicating faithful prompt realization with visual fidelity.

  • Study setup: 60 participants evaluated 10 distinct interaction scenarios using prompts, conditioning images, motion-guidance videos, and four anonymized candidate videos.The study assessed how well generated videos satisfied the intended interaction semantics described in each prompt.
  • User preference results: GraphVid achieved the highest Selection Rate (Wins) across the evaluated dimensions compared with all baselines.“Wins” denotes the percentage of times a model was selected as the absolute best for a metric; “Ties” indicates equal preference, and “Losses” indicates non-selection.
  • User preference results: 77% compared to the strongest baseline, reflecting strong user preference for GraphVid’s generated videos.The reported preference indicates that users perceived GraphVid as better capturing the prompt’s core intent.
  • Interpretation: GraphVid’s semantic interaction representation enabled more faithful prompt realization across diverse contexts without sacrificing visual fidelity.The passage characterizes this result as bridging abstract control signals and visually realistic video generation.

E Qualitative Results

GraphVid demonstrates stronger qualitative control of multi-object interactions than prior methods. It generates coherent, stable motion and preserves object structure under challenging interaction cues.

  • Qualitative comparisons: GraphVid outperforms prior methods qualitatively by following user-specified interaction cues with more coherent multi-object motion and fewer visual artifacts.Prior approaches often fail to faithfully follow intended interaction dynamics across scenarios.
  • Ship scenario: In the ship scenario, GraphVid produces coherent diverging trajectories, while baselines fail to separate the ships’ motion paths or introduce artifacts.The trajectories remain consistent with the specified interaction cues.
  • Handshake scenario: In the handshake scenario, GraphVid maintains stable contact and smooth temporal evolution, unlike competing models with abrupt termination or blurred hand dynamics.These baseline failures indicate difficulty sustaining consistent multi-entity interactions.
  • Metro scenario: In the metro scenario, GraphVid avoids premature termination and structural hallucinations such as incorrect wheel formations seen in baseline generations.The baseline failures reflect weak grounding between motion generation and object structure.

F Failure Case Analysis · G Broader Impacts

GraphVid’s remaining weakness is semantic grounding of small or symbolic elements, which can misassign user interactions and distort generated motion. Its structured interaction interface may broaden video-creation access and interpretability, but realistic video synthesis also raises misuse risks involving deceptive media.

  • F Failure Case Analysis: GraphVid can fail when the VLM does not isolate small or symbolic elements during scene-graph construction.This limitation concerns semantic granularity and object grounding.
  • F Failure Case Analysis: Prior methods may show abrupt motion termination or blurry hand dynamics, while GraphVid produces a consistent handshake with smooth temporal continuity.The comparison is illustrated in a handshake scenario.
  • F Failure Case Analysis: A missed “Zzz” node causes rotation intended for the symbol to be applied rigidly to the sleeping bear, producing severe geometric distortion across frames.The error associates the interaction vector with the nearest high-confidence object.
  • G Broader Impacts: Graph-based dynamics specification may lower the technical barrier to high-quality animation and simulation by replacing complex motion annotations with intuitive interaction graphs.The paper identifies potential applications in creative media, virtual prototyping, education, robotics, and embodied-AI simulation.
  • G Broader Impacts: Structured interactions may improve interpretability in generative systems by making motion control more explicit and modular.This benefit follows from representing dynamics through structured interactions.
  • G Broader Impacts: Realistic video generation from minimal input could enable deceptive or manipulated media, including misinformation and impersonation, if deployed irresponsibly.The paper notes that GraphVid emphasizes structured interaction control rather than photorealistic identity synthesis, but similar methods may still be misused.
Loading 2607.21580v1…