Source-linked AI summary
Scaling Zero-Shot Reference-to-Video Generation
Zijian Zhou, Shikun Liu, Haozhe Liu, Haonan Qiu, Zhaochong An, Weiming Ren, Zhiheng Liu, Xiaoke Huang, Kam Woh Ng, Tian Xie, Xiao Han, Yuren Cong, Hang Li, Chuyan Zhu, Aditya Patel, Tao Xiang, Sen He
TL;DR
R2V methods rely on costly, hard-to-scale reference-image, video, and text triplets. Saber instead trains on video-text pairs using masked training, tailored attention, and mask augmentation. It consistently outperforms explicit-R2V-trained methods on OpenS2V-Eval, including a 1.68% higher total score than Kling1.6.
Problem
Existing R2V methods rely on costly, difficult-to-scale reference-image, video, and prompt triplets, whose limited reference diversity also restricts generalization to unseen subject categories.
Source: §1 · p. 1Method
Saber is a zero-shot R2V framework trained on video-text pairs, using masked training, tailored attention, and spatial mask augmentation.
Source: §1 · p. 1Results
Saber consistently outperforms explicitly R2V-trained methods on OpenS2V-Eval and achieves a 1.68% higher total score than Kling1.6.
Source: §1 · p. 2, §5.2 · p. 7Takeaways & Limitations
The results show that effective R2V models can be trained without dedicated datasets.
Source: §6 · p. 11R2V may collapse with many references, such as 12, and fine-grained motion control and temporal consistency under complex prompts remain challenging.
Source: §6 · p. 11
Abstract
from arXiv · showhide
Reference-to-video (R2V) generation aims to synthesize videos that align with a text prompt while preserving the subject identity from reference images. However, current R2V methods are hindered by the reliance on explicit reference image-video-text triplets, whose construction is highly expensive and difficult to scale. We bypass this bottleneck by introducing Saber, a scalable zero-shot framework that requires no explicit R2V data. Trained exclusively on video-text pairs, Saber employs a masked training strategy and a tailored attention-based model design to learn identity-consistent and reference-aware representations. Mask augmentation techniques are further integrated to mitigate copy-paste artifacts common in reference-to-video generation. Moreover, Saber demonstrates remarkable generalization capabilities across a varying number of references and achieves superior performance on the OpenS2V-Eval benchmark compared to methods trained with R2V data.
1 Introduction
Saber addresses the costly, hard-to-scale construction of explicit R2V triplets with a zero-shot framework trained on video-text pairs. It combines masked training and tailored attention with mask augmentation, and outperforms explicitly R2V-trained methods on OpenS2V-Eval.
1 Introduction
- Explicit R2V triplets of reference images, videos, and prompts require costly collection, annotation, clustering, and filtering, and limited reference diversity restricts generalization to unseen subject categories.Source: §1 · p. 1
- Saber eliminates bespoke R2V data construction by training solely on large-scale video-text pairs.This follows the data paradigm used for text-to-video and image-to-video models.Source: §1 · pp. 1–2
- Randomly sampled, partially masked video frames serve as training references, exposing the model to diverse conditions and encouraging identity- and appearance-consistent representations.The strategy simulates R2V training without explicit R2V data.Source: §1 · p. 1
- Tailored attention masks guide the model toward reference-aware features, while spatial mask augmentations mitigate copy-paste artifacts and improve generated-video visual quality.Source: §1 · pp. 1–2
- Saber supports varying numbers of references without extra data preparation or training-pipeline changes, including multiple views of the same subject.The framework supports richer multi-subject customization.Source: §1 · p. 2, Figure 1 · p. 2
- On OpenS2V-Eval, Saber consistently outperforms methods explicitly trained on R2V data and adapts to foreground or background references by changing the masking ratio.The introduction also reports strong generalization and scalability.Source: §1 · p. 2
2 Related Work
Video-generation models have advanced, but R2V still requires fine-grained subject-driven control and faces costly data construction. Prior R2V work expanded the task and developed varied methods, while relying on explicit triplet datasets; Saber instead trains on video-text pairs.
2.1 Video Generation
- Large-scale video-text models advanced video generation, but fine-grained subject-driven control remains challenging and costly R2V dataset construction impedes large-scale training.Source: §2.1 · p. 3
2.2 Reference-to-Video Generation
- R2V expanded from human identity preservation to reference images of objects and backgrounds; subject-consistent and subject-to-video are also names for the task.Source: §2.2 · p. 3
- Prior R2V methods use varied alignment, adapter, fusion, masking, and identity-injection designs, but depend on costly triplet datasets with complex construction pipelines.The cited datasets require candidate extraction, filtering, clustering, cross-pair matching, and expensive API calls.Source: §2.2 · p. 3
- Saber contrasts with these approaches by training solely on video-text pairs and achieving strong performance on public benchmarks.Source: §2.2 · p. 3
3 Preliminary
The preliminary section describes Wan2.1’s latent-video flow-matching setup and defines the R2V task. R2V generation must follow a text prompt while preserving the identities and appearances shown in reference images.
3.1 Video Generation Models
- Wan2.1 encodes videos into compressed latents, linearly interpolates data and Gaussian noise over time, and trains its diffusion model to predict target velocity.The VAE decodes latents back to pixel space, reducing token count and computation.Source: §3.1 · pp. 3–4
3.2 Task Definition and Notations
- R2V generates a video whose subjects preserve the identities and appearances of reference images while following the provided text prompt.The task notation distinguishes each reference image’s dimensions from the generated video’s frame count and spatial dimensions.Source: §3.2 · p. 4
4 Method
Saber replaces explicit R2V training references with randomly masked video frames, using an attention-based model to condition video generation on references and text. Its zero-shot inference accepts foreground-subject or background-scene references, and it achieves the highest overall score on OpenS2V-Eval in a zero-shot setting.
4 Method
- Saber simulates R2V training by replacing collected reference images with randomly masked video frames, using video-text pairs rather than explicit R2V triplets.Mask augmentations mitigate copy-paste artifacts, while tailored attention guides the model toward relevant reference features.Source: §4 · p. 4
4.1 Masked Frames as Reference
- Randomly sampled, partially masked video frames serve as diverse training references, and spatial augmentations disrupt correspondence with their source frames.The mask generator samples from predefined shape categories and controls foreground area ratios through a scale parameter and bisection search.Source: §4.1 · pp. 4–5
4.2 Model Design
- Reference latents are concatenated after video latents; masked self-attention restricts reference-token access to valid regions, while text cross-attention guides video and reference semantics.The transformer then refines the results with an FFN and outputs a predicted latent.Source: §4.2 · pp. 5–6
4.3 Zero-Shot Inference
- At inference, a pretrained segmenter extracts foreground masks for subject references; background-scene references instead use the full image and an all-ones mask.References and masks are resized and padded to the target video dimensions before prediction.Source: §4.3 · p. 6
- On OpenS2V-Eval, Saber achieves the highest overall score in a zero-shot setting.It also attains the best NexusScore for subject consistency and competitive performance on GmeScore and NaturalScore.Source: Table 1 · p. 7
5 Experiments
Experiments evaluate Saber on OpenS2V-Eval, compare its videos with existing R2V methods, ablate its components, and probe emergent capabilities.
5.1 Datasets, Metrics and Implementation Details
- Training uses ShutterStock video-text pairs, OpenS2V-Eval’s 180 prompts across seven categories, and Wan2.1-14B finetuning with probabilistic foreground-area masking.Evaluation reports visual quality, temporal coherence, motion, identity preservation, subject consistency, naturalness, and text-video alignment.Source: §5.1 · p. 7
5.2 Quantitative Results
- Saber achieves the best overall OpenS2V-Eval performance in zero-shot R2V, scoring 1.68% higher total score than Kling1.6.On NexusScore, Saber exceeds Phantom by 9.79%, VACE by 3.14%, and BindWeave by 0.36%.Source: §5.2 · pp. 7–8
5.3 Qualitative Results
- Across single and multiple human and object references, Saber preserves identity and integrates references coherently, while comparison methods miss subjects, duplicate identities, omit objects, or show discontinuity.The comparisons include Kling1.6, Phantom, and VACE.Source: Figure 4 · p. 8, §5.3 · pp. 8–9
5.4 Ablation Study
- Table 2: masked training raises total score by 1.67%; single-mask variants and fixed r=0.3 reduce it.Using only ellipse, Fourier, or polygon masks reduces total score by 3.35%, 1.58%, and 1.42%, respectively; fixing the foreground area ratio reduces it by 6.18%.Source: Table 2 · p. 9, §5.4 · p. 9
- Mask augmentation avoids copy-paste compositions, while attention masking removes gray artifacts and improves subject separation and blending.The attention mask constrains reference-video token interaction.Source: Figure 5 · p. 9, §5.4 · p. 9, Figure 6 · p. 10
5.5 Emergent Abilities
- Saber combines front, side, and back views of one robot into a coherent subject and adjusts generated details to swapped text descriptions.The reported prompt changes include clothing color and subject positions.Source: §5.5 · p. 10, Figure 7 · p. 10, Figure 8 · p. 10
6 Conclusion
Saber is a zero-shot R2V framework trained on video-text pairs, and experiments on OpenS2V-Eval show it consistently outperforms methods trained on explicit R2V data. Its limitations include fragmentation with many references and challenges in fine-grained motion control and temporal consistency under complex prompts.
6 Conclusion
- Saber consistently outperforms methods trained on explicit R2V data on the OpenS2V-Eval benchmark.The authors conclude that effective R2V models can be trained without dedicated datasets.Source: §6 · p. 11
- Trained solely on large-scale video-text pairs, Saber combines masked training, tailored attention, and mask augmentation for identity-consistent, natural, coherent video generation.Source: §6 · p. 11
- Saber supports multi-identity and multi-view inputs without additional data preparation or changes to the training pipeline.Source: §6 · p. 11
- With significantly more references, such as 12, R2V generation may collapse into fragmented compositions; fine-grained motion control and temporal consistency under complex prompts remain challenging.Source: §6 · p. 11