PaperlayerView exact revision on arXiv

Source-linked AI summary

VFXMaster: Unlocking Dynamic Visual Effect Generation via In-Context Learning

Baolu Li, Yiming Zhang, Qinghe Wang, Liqian Ma, Xiaoyu Shi, Xintao Wang, Pengfei Wan, Zhenfei Yin, Yunzhi Zhuge, Huchuan Lu, Xu Jia

arXiv:2510.25772v1cs.CV

TL;DR

VFX generation is challenging for generative models, and per-effect LoRAs require dedicated training without generalizing to unseen categories. VFXMaster uses reference-based in-context learning, with an attention mask and one-shot token adaptation for difficult OOD effects. The authors report diverse effect imitation and generalization to unseen effects.

  • Problem

    Pretrained video models struggle with abstract dynamic effects, while one-LoRA-per-effect approaches require dedicated training and do not generalize to unseen categories.

    Source: §1 · p. 2

  • Method

    VFXMaster transfers effects from a reference prompt-video pair to target content using in-context conditioning, an attention mask, and one-shot concept-enhancing tokens for OOD effects.

    Source: §1 · pp. 2–3, §3.2 · p. 4, §3.3 · p. 5

  • Results

    The authors report high-quality effect imitation and strong generalization to unseen effects, with extensive experiments across effect categories and OOD data.

    Source: §1 · pp. 2–3, §5 · p. 8

  • Takeaways & Limitations

    The framework supports unified reference-based VFX imitation, with one-shot adaptation intended to improve generalization on tough OOD effects.

    Source: §1 · p. 3, §5 · p. 8

Abstract

from arXiv · show

Visual effects (VFX) are crucial to the expressive power of digital media, yet their creation remains a major challenge for generative AI. Prevailing methods often rely on the one-LoRA-per-effect paradigm, which is resource-intensive and fundamentally incapable of generalizing to unseen effects, thus limiting scalability and creation. To address this challenge, we introduce VFXMaster, the first unified, reference-based framework for VFX video generation. It recasts effect generation as an in-context learning task, enabling it to reproduce diverse dynamic effects from a reference video onto target content. In addition, it demonstrates remarkable generalization to unseen effect categories. Specifically, we design an in-context conditioning strategy that prompts the model with a reference example. An in-context attention mask is designed to precisely decouple and inject the essential effect attributes, allowing a single unified model to master the effect imitation without information leakage. In addition, we propose an efficient one-shot effect adaptation mechanism to boost generalization capability on tough unseen effects from a single user-provided video rapidly. Extensive experiments demonstrate that our method effectively imitates various categories of effect information and exhibits outstanding generalization to out-of-domain effects. To foster future research, we will release our code, models, and a comprehensive dataset to the community.

1. Introduction

VFX generation remains difficult because pretrained models struggle with abstract, dynamic effects, while per-effect LoRAs require dedicated training and do not generalize to unseen categories. VFXMaster instead uses reference-based in-context learning, an attention mask, and one-shot adaptation to imitate effects in a unified model.

1. Introduction

  • Pretrained video models struggle with abstract, surreal effects and spatially aligned controls do not capture their intricate, unstructured dynamics and textures.Even detailed text prompts may not produce the desired effects accurately.Source: §1 · p. 2
  • One-LoRA-per-effect training requires dedicated data and training for each effect, and its closed-set scope excludes unseen categories.Joint LoRA-MoE learning also does not generalize to unseen effects [27].Source: §1 · p. 2
  • VFXMaster treats a reference prompt-video pair as an in-context example and a target prompt and first frame as a query for unified effect imitation.Its attention mask focuses on the reference effect while preventing information leakage.Source: §1 · pp. 2–3
  • A one-shot adaptation strategy uses learnable concept-enhancing tokens to capture fine-grained VFX from one user-provided sample for tough OOD cases.The authors report extensive experiments showing effect imitation and generalization to OOD effects; they also plan to release a curated dataset and metric.Source: §1 · pp. 2–3

2. Related Work

Prior controllable video work adds control signals and editing capabilities, while VFX-specific approaches remain limited in effect range and generalization. VFXMaster is presented as a unified framework addressing that gap.

2.1. Controllable Video Generation

  • Controllable video methods use 3D signals, motion planning, style transfer, and reference-video motion transfer, while other systems support frame transitions and unified editing.The cited examples include 3D control signals [8] [9] [40], VLM motion planning [41], StyleMaster [44], ControlNet [45], FlexiAct [46], Wan-FLFV [37], and VACE [21].Source: §2.1 · p. 3

2.2. Visual Effects Generation

  • VFX-specific systems range from green-screen overlays to per-effect LoRAs [24] and narrow in-domain LoRA-MoE combinations [27], with limited OOD generalization.The section positions VFXMaster as a unified framework for diverse effects.Source: §2.2 · p. 3

3. Method

VFXMaster adapts image-to-video generation into a reference-based in-context framework for transferring effects to a target image. It combines paired reference-target conditioning and controlled attention with single-example adaptation for difficult OOD effects.

3. Method

  • VFXMaster frames controllable VFX generation as reference-based in-context learning, using one reference VFX video to reproduce an effect on a target image.The method section introduces the base model, in-context conditioning, and one-shot adaptation for OOD cases.Source: §3 · p. 3

3.1. Preliminary

  • CogVideoX-5B-I2V supplies the image-to-video backbone, using a 3D VAE, DiT, and T5 encoder to predict video noise from latent inputs.The target's first image is padded and encoded with the video; training minimizes the gap between predicted and standard Gaussian noise.Source: §3.1 · p. 3

3.2. In-Context Conditioning for VFX Generation

  • Training pairs two prompt-video examples with the same effect, concatenates their shared-space latents, and fine-tunes spatial-temporal attention using loss on the target video.Identical 3D RoPE is applied to reference and target videos; no additional trainable parameters or modules are introduced for this conditioning stage.Source: §3.2 · p. 4, Figure 2 · p. 4
  • The in-context attention mask suppresses irrelevant reference information while routing clean reference features to noisy target tokens for effect transfer.Target prompt tokens can attend to all contexts; target video tokens are restricted to their corresponding prompt and reference video tokens.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
  • After training on diverse effects, the model is reported to imitate training-set effects and generalize to unseen visual effects.The section attributes this capability to in-context conditioning.Source: §3.2 · p. 5

3.3. Efficient One-shot Effect Adaptation

  • For a tough OOD effect, one-shot adaptation freezes the base model and trains learnable concept-enhancing tokens from a single user-provided example.Augmentations include random cropping, flipping, shearing, and sharpening; the learned tokens serve as a semantic proxy for the new effect.Source: §3.3 · p. 5

3.4. Training and Inference Pipeline of VFXMaster

  • The pipeline fine-tunes spatial-temporal attention on a broad dataset, then trains concept-enhancing tokens on one OOD example with DiT parameters frozen.Inference uses the tuned attention layers for in-domain effects and can additionally load the corresponding tokens for OOD effects.Source: §3.4 · p. 5

4. Experiment

Experiments assess VFXMaster on OpenVFX and out-of-domain effects through quantitative and qualitative comparisons, ablations, and a user study. Results favor VFXMaster in-domain and show improved out-of-domain metrics with one-shot adaptation.

4.1. Experiment Setup

  • Training uses 10k samples across 200 effect categories, with a separate OOD test set and both established video metrics and VFX-Cons.The training sources include Open-VFX, commercial platforms, and other online resources.Source: §4.1 · p. 5

4.3. Qualitative Evaluation

  • In-domain comparisons report stronger motion dynamics, texture fidelity, material realism, and overall visual fidelity; the Harley Quinn makeup example better preserves subject identity.The qualitative comparison covers four effects.Source: §4.3 · pp. 6–7, Figure 3 · p. 6

4.2. Quantitative Evaluation

  • Across OpenVFX averages, VFXMaster outperforms competing methods on all metrics and leads on VFX-Cons.The evaluation reports advantages in visual quality, temporal coherence, and dynamic range, particularly for complex or fast-moving effects.Source: §4.2 · p. 6, Table 1 · p. 7

4.4. Ablation Study

  • Removing the in-context attention mask drops EFS to 0.11 and CLS from 0.79 to 0.24, and sometimes prevents the intended effect from appearing.The ablation attributes the degradation to entanglement of irrelevant reference content with effect attributes.Source: §4.4 · p. 7

4.3. Qualitative Evaluation

  • For unseen effects, one-shot adaptation captures distinctive textures and dynamics more effectively than in-context conditioning alone.Conditioning alone already produces effects consistent with the reference in content, motion patterns, and overall visual style.Source: Figure 4 · p. 7, §4.3 · p. 7

4.4. Ablation Study

  • Removing the reference prompt lowers metrics but retains basic imitation, while larger training subsets correlate with stronger performance, especially on OOD metrics.The dataset-size comparison spans 2k, 4k, 6k, and 10k videos.Source: §4.4 · pp. 7–8

4.2. Quantitative Evaluation

  • Comparing the model trained solely with in-context learning against its one-shot effect-adapted variant, EFS improves from 0.47 to 0.70 and CLS from 0.79 to 0.87.The comparison assesses OOD generalization.Source: §4.2 · p. 6, Table 2 · p. 8

4.5. User Study

  • Among 50 participants in a 2AFC study, VFXMaster was preferred over Omini-Effect and VFXCreator for effect consistency and aesthetic quality.The study presents participants with a reference video and two generated videos to compare.Source: §4.5 · p. 8, Table 3 · p. 8

5. Conclusion

VFXMaster is a unified in-context learning framework for imitating diverse visual effects. It combines reference conditioning and one-shot adaptation, and the authors report strong in-domain performance, OOD generalization, and data scalability.

5. Conclusion

  • VFXMaster introduces a unified in-context learning framework for efficiently imitating diverse visual effects.The framework uses reference information to condition effect generation.Source: §5 · p. 8
  • Its conditioning strategy uses an attention mask to separate effect attributes from irrelevant reference content and prevent information leakage.A one-shot adaptation mechanism uses learnable concept-enhancing tokens to capture a new effect from a single example.Source: §5 · p. 8
  • The authors report that VFXMaster surpasses state-of-the-art methods on in-domain effects across multiple metrics and generalizes on a dedicated OOD benchmark.They also report excellent data scalability.Source: §5 · p. 8

A.1. Detailed Experimental Details of Attention Implementation

VFXMaster reformulates CogVideoX full attention as multiple cross-attentions while preserving pretrained parameters. The authors report that this implementation accelerates optimization and inference while mitigating content leakage.

A.1. Detailed Experimental Details of Attention Implementation

  • The implementation decomposes long-sequence 3D full attention into multiple cross-attentions while keeping pretrained parameters unchanged.The authors describe the reformulation as equivalent to the original attention architecture.Source: §A.1 · p. 12
  • The original full-attention mechanism incurs substantial optimization overhead, which the added attention mask further exacerbates.Source: §A.1 · p. 12
  • Controlling information flow across the cross-attention modules accelerates optimization and inference while mitigating content leakage.Source: §A.1 · p. 12

A.2. Training Details

The appendix describes training adjustments for paired videos, inference conditioning, and ablations of reference-based generation. The ablation reports leakage when the attention mask is removed and degraded content and effect dynamics without the reference prompt.

A.2. Training Details

  • Reference videos are padded to the training-video resolution before VAE encoding when paired video resolutions differ.Inference follows a similar procedure.Source: §A.2 · p. 12
  • Single-video adaptation augments samples with random combinations of three image transformations and horizontal flips applied with 50% probability.Source: §A.2 · p. 12

A.3. Inference Details

  • Inference generates an effect-specific caption from the effects video, then uses it with the reference video to create an effect-aware description of the first frame for I2V generation.Source: §A.3 · p. 12

A.4. Ablation Details

  • Removing the attention mask leaks irrelevant reference elements, while removing the reference prompt degrades generated content and effect dynamics.The authors present visualizations of these ablations in Fig. 12.Source: §A.4 · p. 12

B.1. Datasets

The dataset contains 10k VFX videos across 200 categories with fine-grained captions. The proposed VFX-Cons. assesses effect occurrence, fidelity to the reference, and leakage, with evaluations linked by dependencies.

B.1. Datasets

  • The dataset comprises 10k high-quality VFX videos across 200 effect categories, each with a fine-grained caption covering subjects, environment, style, and effect progression.The categories include character transformation, environment alteration, and style transition.Source: §B.1 · p. 12

B.2. Metric

  • VFX-Cons. evaluates generated videos on effect occurrence, fidelity to the reference effect, and leakage of irrelevant reference content.EOS checks whether effects occur; EFS assesses consistency of effect presentation; CLS checks for irrelevant content leakage.Source: §B.2 · pp. 12–13
  • The metric uses progressive dependencies: later evaluations are skipped if EOS finds no effect, and CLS is meaningful only when EFS is True.Source: §B.2 · p. 13
  • The final VFX-Cons. score averages its three dimensions, and the VLM provides a concise rationale alongside each decision.Source: §B.2 · p. 13

C. Experiment Result Details

OOD generalization is evaluated on a manually constructed VFX dataset, with detailed results presented in Table 4.

C. Experiment Result Details

  • OOD generalization is evaluated through extensive experiments on a manually constructed VFX dataset, with detailed results in Table 4.The supplied passage does not report the results themselves.Source: C. Experiment Result Details · p. 13

D. More Qualitative Results

The supplementary results show in-domain and out-of-domain effect examples, along with ablation, captioning, and assessment visuals.

D. More Qualitative Results

Loading 2510.25772v1…