PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-As-Prompt: Unified Semantic Control for Video Generation

Yuxuan Bian, Xin Chen, Zenan Li, Tiancheng Zhi, Shen Sang, Linjie Luo, Qiang Xu

arXiv:2510.20888v1cs.CVcs.AI

TL;DR

Semantic-controlled video generation lacks a unified, generalizable approach, with existing methods relying on pixel-wise priors or condition- and task-specific designs. VAP uses reference videos as in-context prompts, and reports strong open-source results, including a 38.7% user preference rate competitive with leading commercial models.

  • Problem

    Semantic-controlled video generation lacks a unified, generalizable framework for conditions without pixel-aligned correspondence, while existing approaches can rely on inappropriate pixel-wise priors or condition- and task-specific designs.

    Source: §1 · p. 1

  • Method

    VAP uses a reference video with wanted semantics as an in-context prompt, with a plug-and-play Mixture-of-Transformers expert augmenting a frozen video diffusion transformer.

    Source: §1 · p. 2

  • Results

    38.7% user preference rate makes VAP competitive with leading closed-source commercial models; it surpasses condition-specific methods and shows zero-shot generalizability.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    VAP demonstrates unified semantic control across diverse conditions and strong generalization within the reported experiments.

    Source: §1 · p. 3, §5 · p. 11

    VAP-Data's semantic conditions are relatively limited, synthetic, and derived from other generative models; inaccurate semantic descriptions or large subject mismatch can degrade generation quality.

    Source: §5 · p. 11

Abstract

from arXiv · show

Unified, generalizable semantic control in video generation remains a critical open challenge. Existing methods either introduce artifacts by enforcing inappropriate pixel-wise priors from structure-based controls, or rely on non-generalizable, condition-specific finetuning or task-specific architectures. We introduce Video-As-Prompt (VAP), a new paradigm that reframes this problem as in-context generation. VAP leverages a reference video as a direct semantic prompt, guiding a frozen Video Diffusion Transformer (DiT) via a plug-and-play Mixture-of-Transformers (MoT) expert. This architecture prevents catastrophic forgetting and is guided by a temporally biased position embedding that eliminates spurious mapping priors for robust context retrieval. To power this approach and catalyze future research, we built VAP-Data, the largest dataset for semantic-controlled video generation with over 100K paired videos across 100 semantic conditions. As a single unified model, VAP sets a new state-of-the-art for open-source methods, achieving a 38.7% user preference rate that rivals leading condition-specific commercial models. VAP's strong zero-shot generalization and support for various downstream applications mark a significant advance toward general-purpose, controllable video generation.

1 Introduction

Semantic-controlled video generation lacks a unified, generalizable framework, while existing approaches rely on unsuitable pixel-wise priors or condition- and task-specific designs. VAP uses reference videos as prompts for in-context control, with a plug-and-play expert, temporally biased position embedding, and VAP-Data supporting its approach.

1 Introduction

  • Semantic-controlled generation lacks pixel-aligned conditions and a unified, generalizable framework, limiting applications such as visual effects, stylization, motion imitation, and camera control.The cited examples include concept, style, motion, and camera semantics.Source: §1 · p. 1
  • Existing approaches either impose inappropriate pixel-wise mapping priors, costly per-condition finetuning, or task-specific designs that hinder unified models and zero-shot generalization.VAP instead treats a reference video with the wanted semantics as a prompt, avoiding those design choices.Source: §1 · pp. 1–2
  • A plug-and-play Mixture-of-Transformers expert augments a frozen video diffusion transformer, interpreting the reference prompt while the backbone generates the target video.Independent feed-forward and layer-normalization paths communicate through full attention for synchronous layer-wise reference guidance.Source: §1 · p. 2
  • Temporally biased RoPE places the reference before the current video while keeping spatial positions fixed, removing shared-RoPE pixel-mapping priors and supporting robust context retrieval.This arrangement matches the temporal order expected by in-context generation and preserves spatial consistency for semantic changes in the reference.Source: §1 · pp. 2–3
  • 38.7% user preference rate makes VAP competitive with leading closed-source commercial models; it also surpasses condition-specific methods and generalizes zero-shot.Experiments report coherent, semantically aligned videos across diverse semantic conditions and downstream tasks.Source: §1 · p. 3
  • VAP-Data provides over 100K curated paired samples across 100 semantic conditions as a data foundation for unified semantic-controlled video generation.The paper presents it as the largest dataset for this task to date.Source: §1 · p. 3

2 Related Works

The related work describes video generation's shift to DiTs, then distinguishes pixel-aligned structure control from semantic control without pixel mapping. VAP frames the latter as in-context generation using reference-video prompts.

2.1 Video Generation

  • The standard video-generation pipeline encodes Gaussian noise into latents, processes latent patches with a DiT, and decodes them; pretrained DiTs typically support text or first/last-frame control.New controllable tasks commonly add task-specific modules or special inference procedures.Source: §2.1 · p. 3

2.2 Controllable Video Generation

  • Structure-controlled methods exploit pixel alignment by injecting condition signals through an adapter or branch, while semantic control covers conditions without pixel-wise correspondence.Structure conditions include trajectory, pose, depth, optical flow, and masks; unified methods combine these as spatial conditions.Source: §2.2 · p. 4
  • Semantic-control methods use per-condition finetuning or task-specific modules, limiting unified and zero-shot generation; a concurrent LoRA mixture-of-experts also fails on unseen conditions.The section poses unified semantic-controlled generation as an open question.Source: §2.2 · p. 4
  • VAP treats videos with wanted semantics as in-context prompts, unifying multiple semantic-controlled tasks without task-specific designs and with strong zero-shot abilities.The paper presents this as its approach to the open question of a unified framework.Source: §2.2 · p. 4

3 Methods

VAP unifies semantic-controlled video generation by using a reference video as an in-context prompt, rather than relying on pixel alignment, per-condition finetuning, or task-specific designs. A trainable expert guides a frozen video DiT, with temporally shifted position indices to avoid false reference-target mapping priors.

3.1 Preliminary

  • Flow matching trains the video model to predict the velocity along a path from Gaussian noise to a video latent, using mean squared error against the ground-truth velocity.At inference, an ODE solver uses discrete denoising steps to produce the video latent.Source: §3.1 · p. 4

3.2 Reference Videos as Task-Agnostic Prompts

  • VAP uses videos sharing target semantics as unified prompts across concept, style, motion, and camera conditions, avoiding per-condition finetuning and per-task designs.The paper evaluates these four representative condition types in one unified model.Source: §3 · p. 4, §3.2 · p. 5
  • The method evaluates concept, style, motion, and camera as four distinct condition types, represented in VAP-Data’s 100-condition taxonomy.The overview groups the conditions into four categories and shows diverse reference images and a word cloud of conditions.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 Plug-and-Play In-Context Control

  • A trainable expert processes reference-video prompts alongside a frozen DiT that generates the target, exchanging information bidirectionally through full attention at each layer.The two branches retain separate query, key, value, feed-forward, and normalization paths, while full attention enables two-way fusion.Source: §3.3 · p. 6

3.4 Temporally Biased Rotary Position Embedding

  • Shifting reference temporal indices by a fixed offset while keeping spatial indices unchanged removes a false pixel-level mapping prior and improves performance.The shift places reference prompt indices before the noisy target-video tokens.Source: §3.4 · p. 6

4 Experiments

Across comparisons, VAP performs comparably to closed-source commercial models and on average surpasses open-source baselines, while operating as one unified model. The experiments also describe VAP-Data, evaluation procedures, zero-shot generation, and ablations of model structure, position embeddings, scale, and backbone.

4.1 Implementation Details

  • The experiments train VAP on CogVideoX-I2V-5B and Wan2.1-I2V-14B, using roughly 20K training steps and 48 NVIDIA A100s.Videos are resized to 480×720 (832) and sampled at 49 frames and 16 fps.Source: §4.1 · p. 7

4.2 Metrics

  • The evaluation covers text alignment, video quality, and semantic alignment, using CLIP similarity, three video-quality metrics, and an automatic reference-generation consistency score.Gemini-2.5-pro scores each video pair for semantic alignment under detailed evaluation rules.Source: §4.2 · p. 7

4.3 Dataset

  • VAP-Data contains over 100K paired samples across 100 semantic conditions, constructed from 2K curated reference images and compatible video-generation templates and LoRAs.Evaluation evenly samples 24 test conditions from four categories, with two samples per condition.Source: §4.3 · p. 7

4.4 Comparison with Previous Methods

  • VAP performs comparably to closed-source models and on average surpasses open-source baselines; researchers also ranked it among the highest in overall preference alongside Kling and Vidu.Qualitative comparisons report stronger temporal coherence, visual quality, and semantic consistency than structure-controlled baselines, DiT backbones, and condition-specific finetuning.Source: §4.4 · p. 8, Table 1 · p. 9
  • VAP uses a single unified model, whereas LoRA baselines require a separate model per condition and do not generalize to unseen references; VAP also generates from unseen semantic references.The paper reports that LoRA improves semantic alignment but harms base quality relative to CogVideoX-I2V.Source: §4.4 · pp. 8–9

4.5 Ablation Study

  • The MoT ablation preserves the base DiT’s generation ability, while bidirectional layer-wise exchange improves semantic alignment; single-branch LoRA has limited capacity for complex in-context generation.Residual-addition injection degrades performance, which the authors attribute to its rigid pixel-to-pixel mapping prior.Source: §4.5 · p. 10
  • Ablations find that identical reference and target RoPE imposes an unrealistic pixel-wise alignment prior, while adding a width bias makes spatial referencing harder and degrades performance.The tested alternative adds a width bias by placing the reference video to the left of the target video.Source: §4.5 · p. 10
  • Performance improves across all metrics as training data grows, and a Wan2.1-I2V-14B backbone improves dynamic degree and aesthetic score but slightly weakens reference alignment.The Wan2.1 version uses only one-quarter in-context interaction compared with CogVideoX.Source: §4.5 · p. 10

5 Conclusion

VAP uses reference videos as prompts for unified semantic control and reports strong results across open-source and commercial comparisons. The authors identify limits in the dataset and captioning setup that remain for future work.

5 Conclusion

  • VAP achieves state-of-the-art performance among open-source models, comparable performance to commercial models, and strong generalization.The framework treats reference videos as prompts and uses a plug-and-play mixture-of-transformers expert for in-context control.Source: §5 · p. 11
  • VAP-Data's semantic conditions are relatively limited, synthetic, and derived from other generative models, potentially inheriting their biases, artifacts, and conceptual limitations.The authors leave construction of larger-scale, real semantic-controlled video data to future work.Source: §5 · p. 11
  • Inaccurate semantic descriptions or large subject mismatch in captions can degrade generation quality.The authors suggest instruction-style captions may better capture intended semantics and improve control.Source: §5 · p. 11

6 Ethics Statement

VAP is intended for research, education, and creative prototyping, with specified misuse boundaries and mitigations. The appendix points readers to additional demonstrations and implementation, dataset, and ablation material.

6 Ethics Statement

  • VAP is intended for research, education, and creative prototyping, not surveillance, impersonation, political persuasion, or other high-risk deployments.Artifact releases will include a research-only license and an acceptable-use policy prohibiting abusive or unlawful scenarios.Source: §6 · p. 11
  • Potential misuses include identity impersonation, targeted harassment, deceptive political messaging, and sexualized or violent media; mitigations include research-only release and default content filters.The stated filter examples include sexual content, explicit violence, and hate symbols.Source: §6 · p. 11

Appendix

  • The appendix covers qualitative results, downstream applications, implementation details, VAP-Data details and limitations, reference and caption quality, multiple references, and ablations.The listed material is organized in Sections A–F.Source: Appendix · p. 16

A Gallery

  • Additional gallery cases show VAP generation examples in Figures 8 and 9.The authors encourage readers to view the webpage for better visualization.Source: A Gallery · p. 16

B Application

VAP applies semantics from a source video to new subjects and demonstrates multiple downstream controls. The reported application cases include prompt-based attribute editing while preserving identity and motion.

B Application

  • VAP supports semantic transfer across subjects, with applications spanning different semantics, reference images, and fine-grained prompt edits.The application demonstrations cover the cases described in the following points.Source: B Application · p. 16
  • With the same reference image, VAP generates videos aligned to different semantics from different reference videos, and follows the provided semantics when those videos share a semantic.These two demonstrations vary either the semantics or the source video while keeping the reference image fixed.Source: B Application · p. 16
  • With one reference video and different reference images, VAP transfers the video’s semantics to each image and generates corresponding videos.Source: B Application · p. 16
  • Changing a single prompt word can adjust a generated attribute while preserving identity and motion.The demonstration fixes the reference inputs and changes one word, such as “black” to “white.”Source: B Application · p. 16

C.1 Hyperparameters

  • Tab. 3 summarizes hyperparameters for two VAP variants based on CogVideoX-5B and Wan2.1-14B [70], showing transferability across different DiT architectures.Source: §C.1 · p. 16

C.2 Metrics

The semantic-alignment score evaluates consistency between a reference condition and a generated video using VLM scoring and semantic-specific criteria. The appendix reports close agreement across VLM evaluations and user-preference trends, alongside further visualizations and a hyperparameter table.

C.2 Metrics

  • The semantic-alignment score measures consistency between the reference semantic condition and generated video because standard video-quality metrics do not reliably capture semantic adherence.Each evaluation supplies the VLM with semantic-specific criteria, the reference condition, and the generated video.Source: §C.2 · p. 16
  • GPT-5 scores match closely with Gemini-2.5-Pro scores and follow human-preference trends, supporting the reported stability of the semantic-alignment score.The comparison is summarized in Table 5.Source: §C.2 · p. 16

D Dataset

  • Figures 8 and 9 provide additional visualizations of entity transformation and interaction, style, motion, and camera semantic categories.Source: Figure 8 · p. 17, Figure 9 · p. 18
  • Figures 10 and 11 show semantic control with a fixed reference image: outputs vary with different semantics and align with shared semantics across reference videos.Source: Figure 10 · p. 19, Figure 11 · p. 19
  • Figures 12 and 13 illustrate semantic transfer across reference images and fine-grained attribute changes while preserving semantics and identity.Source: Figure 12 · p. 20, Figure 13 · p. 20
  • Table 3 summarizes hyperparameter selection for VAP variants based on CogVideoX-I2V-5B and Wan2.1-I2V-14B.Source: Table 3 · p. 21

D.1 Dataset Details

VAP-Data bootstraps a large paired dataset from curated images and specialist generators, enabling training of a single generalist model. Its test subset samples semantic conditions across four categories.

D.1 Dataset Details

  • VAP-Data contains over 100K paired samples across 100 semantic conditions, covering concept, style, motion, and camera movement.The data was bootstrapped from curated high-quality real images using commercial APIs and community LoRAs.Source: §D.1 · p. 21
  • VAP-Data serves as proof of concept that one generalist model can learn unified semantic control from examples produced by disparate specialist models.Source: §D.1 · p. 21
  • The test subset includes 24 semantic conditions across four categories, with two samples per condition for 48 total test samples.Source: §D.1 · p. 21

D.2 Dataset Limitations

VAP-Data is synthetic and may inherit limitations from its source models; collecting a large real-world semantic-controlled dataset is left to future work. The reported experiments nevertheless show generalization to unseen conditions and downstream tasks.

D.2 Dataset Limitations

  • Because VAP-Data is generated using commercial-model templates and community LoRAs, VAP may inherit their stylistic biases, artifacts, and conceptual limitations.For example, the authors note that poor source-model hand generation may limit VAP's ability to learn hand generation.Source: §D.2 · p. 21
  • Building a large real-world semantic-controlled video dataset could address this issue, but is beyond this paper's scope and left for future work.Source: §D.2 · p. 23
  • Zero-shot experiments report generalization to unseen semantic conditions, including crumble, dissolve, levitate, and melt.Source: §D.2 · p. 23
  • Semantic transfer quality is lower when reference and target captions misalign or their subjects differ substantially in structure.The figure contrasts aligned descriptions and similar subjects with a mislabeled description and a markedly different subject.Source: Figure 14 · p. 24
  • VAP also generalizes across downstream tasks, including prompting one reference image with different reference videos or different images with the same reference video.Source: §D.2 · p. 24

E Limitation Analysis

VAP's semantic transfer depends on caption and subject alignment, while multiple references can blend unwanted visual cues. The plug-and-play design also adds inference cost.

E.1 Influence of Reference Video and Caption

  • Transfer is more reliable when reference and target captions describe aligned semantics and their subjects have similar structure.Mislabeled descriptions or large subject mismatches reduce alignment and visual quality.Source: §E.1 · p. 24

E.2 Influence of Multiple Reference Videos

  • With three structurally divergent references, the model can mix unwanted appearance and shape cues into the output, although results are otherwise similar to single-reference prompting.The authors attribute the leakage to generic captions without explicit referents; tailored multi-reference control or instruction-style captions may mitigate it.Source: §E.2 · p. 24, Figure 15 · p. 25

E.3 Efficiency

  • VAP avoids retraining the backbone at pre-training scale, but its added parameters increase memory use and runtime, with inference time roughly doubling on average.The overhead mainly comes from MoT-expert computation and in-context full attention; optimization is left for future work.Source: §E.3 · p. 25

F Ablation Study

The ablations examine MoT design, position embeddings, scalability, transfer to another DiT, layer allocation, and prompt noise. They report advantages for the full MoT design and temporal-only positioning, alongside trade-offs in reduced interaction and noisy prompts.

F Ablation Study

  • MoT supports plug-and-play in-context control while preserving the base DiT's generative ability; LoRA capacity is limited, and residual addition retains a mismatched pixel-to-pixel prior.Layer-wise bidirectional information exchange lets the reference representation adapt with target tokens, improving semantic alignment.Source: F Ablation Study · p. 25
  • Using identical RoPE for reference and target imposes an unrealistic pixel-wise alignment prior, while adding a width bias makes spatial referencing harder; both variants degrade performance.Source: F Ablation Study · p. 25
  • VAP improves across all metrics as training data grows, showing strong scalability.The authors relate this to its unified reference-prompt design and MoT framework.Source: F Ablation Study · p. 25
  • On Wan2.1-I2V-14B, VAP improves dynamic degree and aesthetic score but has slightly worse reference alignment than the CogVideoX version.The authors attribute the alignment difference to having only one-quarter as many in-context interactions.Source: F Ablation Study · pp. 25–26
  • Balanced MoT layer distributions improve generation quality, while reducing layers improves efficiency at the cost of some performance.The first-and-last-layer setup outperforms the first-half setup, and all-odd layers outperform odd layers limited to the first half.Source: F Ablation Study · p. 26
  • Adding noise to the video prompt often causes severe artifacts because it corrupts contextual information when reference and target videos differ in appearance and layout.Source: F Ablation Study · p. 26
Loading 2510.20888v1…