PaperlayerView exact revision on arXiv

Source-linked AI summary

VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformer

Xinyu Liu, Ailing Zeng, Wei Xue, Harry Yang, Wenhan Luo, Qifeng Liu, Yike Guo

arXiv:2502.05979v4cs.CV

TL;DR

Controllable VFX generation remains underexplored, and access to proprietary VFX resources is restricted. This paper introduces Open-VFX and VFX Creator, which combines a Video Diffusion Transformer with spatial and temporal controls. Experiments report strong performance in both controllability dimensions, while the authors note that unified training performs worse than category-specific training.

  • Problem

    Controllable VFX video generation remains underexplored, and proprietary platforms restrict access to their VFX resources.

    Source: §1 · p. 2

  • Method

    The paper presents Open-VFX, annotated for spatial and temporal control, and VFX Creator, a Video Diffusion Transformer framework with mask and timestamp controls.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Experiments report state-of-the-art performance in both spatial and temporal controllability, and the introduction says the system surpasses existing methods on Open-VFX.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    VFX Creator is presented as a route to efficient, user-friendly VFX generation that makes advanced effects more accessible.

    Source: §7 · p. 9

    Open-VFX still needs greater breadth and depth, and unified training performs worse than category-specific training because it tends to confuse effects.

    Source: §6 · p. 9

Abstract

from arXiv · show

Crafting magic and illusions is one of the most thrilling aspects of filmmaking, with visual effects (VFX) serving as the powerhouse behind unforgettable cinematic experiences. While recent advances in generative artificial intelligence have driven progress in generic image and video synthesis, the domain of controllable VFX generation remains relatively underexplored. In this work, we propose a novel paradigm for animated VFX generation as image animation, where dynamic effects are generated from user-friendly textual descriptions and static reference images. Our work makes two primary contributions: (i) Open-VFX, the first high-quality VFX video dataset spanning 15 diverse effect categories, annotated with textual descriptions, instance segmentation masks for spatial conditioning, and start-end timestamps for temporal control. (ii) VFX Creator, a simple yet effective controllable VFX generation framework based on a Video Diffusion Transformer. The model incorporates a spatial and temporal controllable LoRA adapter, requiring minimal training videos. Specifically, a plug-and-play mask control module enables instance-level spatial manipulation, while tokenized start-end motion timestamps embedded in the diffusion process, alongside the text encoder, allow precise temporal control over effect timing and pace. Extensive experiments on the Open-VFX test set demonstrate the superiority of the proposed system in generating realistic and dynamic effects, achieving state-of-the-art performance and generalization ability in both spatial and temporal controllability. Furthermore, we introduce a specialized metric to evaluate the precision of temporal control. By bridging traditional VFX techniques with generative approaches, VFX Creator unlocks new possibilities for efficient and high-quality video effect generation, making advanced VFX accessible to a broader audience.

1 INTRODUCTION

Controllable VFX video generation remains underexplored, while existing open-source models struggle with complex effects and motion control. The paper introduces the Open-VFX dataset and VFX Creator, a framework for generating effects from text and reference images with spatial and temporal control.

1 INTRODUCTION

  • VFX Creator uses a Video Diffusion Transformer with a spatial-temporal controllable LoRA adapter to generate effects from text and static reference images.Mask conditioning supports instance-level spatial manipulation, while timestamp tokens support control of effect timing and pace.Source: §1 · p. 2
  • Open-VFX is a VFX video dataset spanning 15 effect categories, with text prompts, instance masks, and start–end timestamps for spatial and temporal conditioning.The dataset is presented as the first high-quality resource of its kind.Source: §1 · p. 2
  • The system surpasses existing methods on the Open-VFX evaluation, and the authors introduce a metric for temporal-control precision.The introduction reports the comparison without giving metric values.Source: §1 · p. 2

2 RELATED WORK

Prior work develops general video-generation and LoRA-based fine-tuning methods, but the supplied review identifies controllable VFX generation and combined spatial-temporal controllability as unexplored. The section also situates the proposed dataset among diverse image inputs and VFX prompts.

2.1 General Video Generation

  • Diffusion Transformers and controllable video-generation methods address temporal consistency or image conditioning, while LAMP is constrained by fixed motion patterns in few-shot settings.The review describes architectures and methods including CogVideoX, Hunyuan-DiT, LAMP, and ControlNet.Source: §2.1 · p. 2
  • The review identifies controllable visual-effect video generation as an unexplored area despite advances in general video generation.The overview figure depicts varied input images, descriptions for 15 VFXs, and an Explode it example.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 LoRA-Based Video Generation

  • Prior LoRA-based video methods fine-tune selected components or separate appearance and motion, but the review says they had not combined temporal and spatial controllability with LoRA.Examples include Tune-A-Video, MotionDirector, and VMC.Source: §2.2 · p. 3

3 DATASET

Open-VFX comprises videos across 15 VFX categories and reference images, with annotations for motion timing and object masks. Its annotation process uses tracked-point displacement for temporal boundaries and masks for spatial control.

3.1 Definition of Visual Effects

  • The dataset includes 15 effects, from Explode and Dissolve to character transformations, illustrating the range of visual effects considered.The paper describes VFX as encompassing compositing, digital effects, motion capture, and digital makeup.Source: §3.1 · p. 3

3.2 Source Videos

  • Open-VFX contains 675 videos from Pika 1.5 and PixVerse 3.0 across 15 categories, plus 245 Pexels reference images; videos are at least 700 × 1000 pixels and 24 fps.Fig. 3 shows examples of 10 VFXs and diverse reference images.Source: §3.2 · p. 3, Figure 3 · p. 4

3.3 Data Annotation

  • Co-Tracker labels motion start and end from tracked-point displacement, while SAM2 produces instance mask sequences and inference-time binary region masks for spatial conditioning.Motion begins when tracked coordinates shift and ends when displacement ceases.Source: §3.3 · pp. 3–4

4 NETWORK

VFX Creator uses a Video Diffusion Transformer with spatial and temporal controllable LoRA adapters. Mask guidance supports instance-level spatial manipulation, while temporal signals control effect timing and pace.

4.1 Preliminary

  • The preliminary introduces CogVideoX and LoRA as the baseline architecture and adaptation method for the spatial and temporal controllable adapter.LoRA is described as helping explain the controllable adapter.Source: §4.1 · p. 4

4.1.1 Baseline Diffusion Transformer Network.

  • The baseline jointly processes sequential video latents and T5 text embeddings in a stacked Expert Transformer with Adaptive Layer Normalization and 3D RoPE.A causal 3D VAE compresses video with temporal and spatial factors of 4 and 8, respectively.Source: §4.1.1 · pp. 4–5

4.1.2 Low-rank Adaptation(LoRA).

  • LoRA updates the original weight matrix with a product of two low-rank matrices, reducing training computation and memory costs.The update is ΔW = AB^T, with d smaller than n and m.Source: §4.1.2 · p. 5

4.2 VFX Creator

  • Training augmentation randomly shifts moving clips while constraining start and end timestamps to [0,T−D] and [D,T], respectively.T is the maximum training-video frame count and D is the duration of effect motion.Source: §4.2.1 · p. 5
  • Temporal masks can instead be projected into timestep embeddings and added to each frame, applying motion timing and pace uniformly across frames.The timestamp encoder projects the temporal masks into timestep-embedding space for incorporation into DiT blocks.Source: §4.2.1 · p. 5
  • Start- and end-time timestamps are encoded as tokens and concatenated with text prompt tokens, so DiT cross-attention conditions generation on temporal signals.The timestamps are projected into the intermediate text representation space.Source: §4.2.1 · p. 5
  • The plug-and-play mask control module enables instance-level spatial manipulation; its mask sequences condition a trainable spatial ControlNet branch connected to the partly frozen main branch.The branch uses a zero-initialized convolution to integrate spatial conditions while minimizing interference with the base model.Source: §4.2.2 · p. 6

5 EXPERIMENT

The experiments evaluate VFX Creator across 15 effect types, comparing generation quality and controllability with existing methods. Results favor VFX Creator in quantitative and qualitative comparisons, while ablations examine temporal integration, spatial control, and training-set size.

5 EXPERIMENT

  • The evaluation compares VFX Creator with DynamiCrafter, CogVideoX, LTX-Video, and Pika or PixVerse, including focused tests of spatial and temporal control.All compared approaches accept text prompts and reference images.Source: §5 · p. 6

5.1 Implementation Details

  • Training uses rank-128 low-rank matrices in the baseline 3D Transformer, with 49 sampled frames at 480 × 720 resolution and 3,000 training steps.The dataset is split 9:1; training averages 40 videos per effect and testing uses 5 per effect.Source: §5.1 · p. 6

5.2 Evaluation Metrics

  • Generation quality and dynamics are evaluated with FID-VID, FVD, and Dynamic Degree, while temporal control uses frame- and second-level errors and Temporal IoU.The timestamp evaluation samples five start-end pairs per reference; starts are constrained to [0, 2/T] and ends to [2/T, T], where T is the clip’s frame count.Source: §5.2 · p. 6

5.3 Quantitative Comparison Results

  • VFX Creator outperforms three open-source methods across all reported metrics, particularly for large-motion effects, with more accurate motion generation.DynamiCrafter and CogVideoX produce minimal or no motion, while LTX-Video can produce large but semantically incorrect motion; Table 1 reports the comparisons.Source: §5.3 · p. 6, Table 1 · p. 7

5.4 Qualitative Comparison Results

  • Across qualitative comparisons, VFX Creator produces more reliable, semantically aligned, temporally consistent effects when other methods struggle.Examples include Pika failing on “Deflate it” and “Eye-pop it”; the figures also show spatial-control results using user-specified instance masks.Source: §5.4 · p. 7, Figure 5 · p. 7, Figure 6 · p. 8

5.5 Ablation Study

  • Text-space timestamp integration outperforms the alternative, and increasing training shots generally improves performance, with 10-shot often yielding balanced results.The spatial module also achieves accurate object manipulation with photorealistic videos and strong consistency with user interactions.Source: §5.5 · p. 8, Table 2 · p. 8, Table 3 · p. 8

5.6 User Study

  • In a 20-user study of 30 comparison sets, VFX Creator and Pika/PixVerse received superior preferences across four evaluation dimensions, with VFX Creator slightly ahead.Dimensions were text alignment, subject fidelity, motion fluency, and overall quality; results are shown in Fig. 7.Source: §5.6 · pp. 8–9, Figure 7 · p. 9

6 LIMITATION

Open-VFX covers 15 effect types, but the dataset and unified-training approach have scope and performance limitations.

6 LIMITATION

  • Open-VFX’s 15 effect types leave room to expand dataset breadth and depth, while unified training performs worse than category-specific training and can confuse effects.The authors identify Mixture of Experts as a possible future strategy for improving mixed training.Source: §6 · p. 9

7 CONCLUSION

The paper introduces Open-VFX as a resource for controllable VFX generation and presents VFX Creator, a Video Diffusion Transformer framework. The authors report realistic, dynamic effects and state-of-the-art spatial and temporal controllability using minimal training videos.

7 CONCLUSION

  • Open-VFX offers diverse effect categories and detailed annotations intended to support training VFX generation models.VFX Creator is a Video Diffusion Transformer framework for controllable VFX generation.Source: §7 · p. 9
  • Experiments demonstrate realistic, dynamic effects with state-of-the-art spatial and temporal controllability using minimal training videos.The authors connect these capabilities with efficient, user-friendly VFX generation and broader access to advanced visual effects.Source: §7 · p. 9

A DEFINATIONS OF VISUAL EFFECTS

The Open-VFX dataset contains 15 visual-effect categories and a range of subjects; Table 4 explains the meanings assigned to the effect types.

A DEFINATIONS OF VISUAL EFFECTS

B QUANTITATIVE RESULTS OF MIXED TRAINING

Mixed training generally underperformed category-specific training, though about one-third of effects improved slightly. The authors suggest Mixture of Experts as a possible future strategy.

B QUANTITATIVE RESULTS OF MIXED TRAINING

C MORE CONTROLLED QUALITATIVE RESULTS

Additional qualitative results illustrate VFX Creator's spatial instance manipulation and temporal rhythm control across generated effects.

C MORE CONTROLLED QUALITATIVE RESULTS

D MORE QUALITATIVE COMPARISON RESULTS

Qualitative comparisons show LTX-Video and CogVideoX producing minimal actions when they fail to interpret effect prompts accurately. VFX Creator and Pika perform better in the described examples.

D MORE QUALITATIVE COMPARISON RESULTS

Loading 2502.05979v4…