PaperlayerView exact revision on arXiv

Source-linked AI summary

DomainShuttle: Freeform Open Domain Subject-driven Text-to-video Generation

Nan Chen, Yiyang Cai, Rongchang Xie, Junwen Pan, Cheng Chen, Weinan Jia, Zhuowei Chen, Wen Zhou, Zhenbang Sun, Wenhan Luo

arXiv:2606.26058v1cs.CV

TL;DR

Open-domain subject-driven video generation must preserve intrinsic subject features while allowing irrelevant properties to adapt across in-domain and cross-domain scenarios. DomainShuttle addresses this with decoupled domain-aware modeling, separate spatial encoding, and consistent feature extraction, outperforming existing methods with an 18.7% higher Cross-Domain Score.

  • Problem

    Entanglement between intrinsic subject features and domain-specific attributes makes it difficult to preserve subjects while enabling flexible cross-domain transitions.

  • Method

    DomainShuttle decouples video and reference features, separates their RoPE spaces, and uses consistent cross-reference training to extract intrinsic subject representations.

  • Results

    18.7% improvement in Cross-Domain Score over state-of-the-art methods accompanies comprehensive gains in subject consistency and text controllability across complex scenarios.

  • Takeaways & Limitations

    DomainShuttle provides competitive performance across in-domain and cross-domain video personalization while jointly optimizing subject consistency and generative flexibility.

Abstract

from arXiv · show

Open domain subject-driven text-to-video (S2V) generation has drawn significant interest in academia and industry. Open domain S2V mainly involves two scenarios: in-domain, which requires retaining the reference subject features as much as possible, and cross-domain, which preserves the intrinsic features of the subject while allowing subject-irrelevant properties to vary flexibly according to the text prompt. Existing methods primarily focus on maximizing subject fidelity in in-domain scenarios, which limits their editability and adaptability in cross-domain scenarios, such as novel styles, semantic combinations, or domain attributes. In this study, we propose that an ideal S2V method should flexibly shuttle between different domains, achieving strong performance in both in-domain and cross-domain scenarios. To this end, we propose DomainShuttle, which could achieve high fidelity and generative flexibility for open domain video personalization. Specifically, we introduce Domain-MoT, which decouples videos and reference features and introduces the domain-aware AdaLN for domain-specific modeling of reference images. We then introduce the Video-Reference DualRoPE scheme, which places reference image tokens and video tokens in separate RoPE spaces to enable precise subject-level spatial modeling, and Cross-Pair Consistent Loss, which aims to extract intrinsic subject features unaffected by irrelevant features. Extensive experiments demonstrate that DomainShuttle achieves significant performance improvements over existing methods, exhibiting high subject fidelity and generative flexibility across diverse open domain application scenarios.

1 Introduction

DomainShuttle targets open-domain subject-driven video generation by jointly preserving intrinsic subject features and enabling flexible generation across in-domain and cross-domain scenarios. It achieves this through decoupled video/reference modeling, domain-aware reference processing, specialized spatial modeling, and consistent subject-feature extraction.

  • Introduction: Open-domain S2V must preserve distinctive subject features while generating videos flexibly across in-domain and cross-domain scenarios.Applications include advertising, creative design, and AI filmmaking, with subjects such as humans, objects, fantasy IPs, and backgrounds.
  • Introduction: Existing methods mainly optimize in-domain subject fidelity, neglecting creative cross-domain scenarios and thereby sacrificing editability and flexibility.Cross-domain examples include real-world subjects in fantasy domains, fantasy subjects mapped to real-world objects, and complex mixed interactions.
  • Introduction: DomainShuttle jointly optimizes subject consistency and generation flexibility by independently processing video and reference branches with domain-aware reference modeling.Its DomainMoT module decouples videos and reference features, while domain-aware AdaLN supports domain-specific reference-image modeling.
  • Introduction: Video-Reference DualRoPE separates reference-image and video token spaces for subject-level spatial modeling, while Cross-Pair Consistent Loss extracts intrinsic subject features.These mechanisms are designed to distinguish subject-level spatial distance and reduce the influence of irrelevant features.
  • Introduction: 18.7% improvement in Cross-Domain Score over SOTA methods demonstrates DomainShuttle’s stronger subject consistency and text controllability across complex scenarios.The paper reports comprehensive improvements over existing methods across various complex scenarios.

2 Related Work

Related work spans the evolution of video diffusion architectures and the expansion of subject-driven text-to-video generation from single-identity human scenarios toward more general personalization settings. DomainShuttle addresses this context through decoupled domain-aware modeling, separated RoPE spaces, and consistent feature alignment.

  • Video Diffusion Models: Video diffusion models evolved from UNet-based approaches with temporal modules toward DiT architectures that scale visual generative capacity and capability.Early methods augmented image diffusion models with temporal modules, while newer video diffusion models increasingly adopt DiT.
  • Subject-Driven Text-to-Video: Subject-driven text-to-video generation preserves specified subject features from reference images under textual guidance, expanding from single-identity humans to multi-subject personalization.The targeted features include identity, domain semantics, style, and attributes; recent work covers humans, objects, and backgrounds.
  • DomainShuttle: DomainShuttle decouples reference and video branches, separates their RoPE spaces, and aligns reference-image features to learn intrinsic subject characteristics.Its overview highlights domain-aware AdaLN guidance, precise subject-level spatial relationships, and Cross-Pair Consistent Loss for reducing redundant features.

3 Methodology

DomainShuttle combines Domain-MoT, Video-Reference DualRoPE, and Cross-Pair Consistent Loss to disentangle subject features from domain attributes and improve flexible, high-fidelity personalization. Its design separately processes video and reference features, injects domain-aware modulation, models reference subjects spatially, and learns shared features across reference pairs.

  • Domain-MoT: Domain-MoT separates video latents and reference image features into independent paths, enabling domain-aware modulation and more precise subject-feature extraction.The branches use separate patch embeddings, independent QKV projections, and independent RoPE while preserving the video branch as the base model.
  • Domain-MoT: Domain-aware AdaLN structurally decouples noise and reference modulation so domain attributes can vary without disturbing content and temporal structure.Reference AdaLN is conditioned on time and domain attributes, supporting in-domain prior knowledge and cross-domain attribute replacement.
  • Video-Reference DualRoPE: Video-Reference DualRoPE assigns reference images and video tokens to separate positional spaces, addressing the lack of temporal continuity among reference images and their subject-level relationships.Reference temporal indices are set to 0 while video indices start from 1; offsets distinguish different subjects and preserve proximity for images depicting the same subject.
  • Cross-Pair Consistent Loss: Cross-Pair Consistent Loss samples two reference sets at the same timestep to extract shared subject features while suppressing redundant single-frame details.The shared features include shape, texture style, and subject identity.
  • Training Data: The model is trained on image and video personalization data, including a 200K image dataset and the Phantom-Data, OpenS2V, and Ditto-1M video datasets.Low-quality videos and references are filtered using aesthetics, motion, and visual-semantic alignment criteria.

4 Experiments

DomainShuttle is evaluated on balanced in-domain and cross-domain test sets using video quality, text controllability, and subject-consistency metrics. It achieves strong quantitative, qualitative, ablation, and human-preference results, especially for cross-domain subject preservation and editability.

  • Test Dataset: The test set contains 110 in-domain and 110 cross-domain samples spanning multi-subject interactions, background preservation, and real-to-fantasy or fantasy-to-real transformations.Among the in-domain samples, 90 come from OpenS2V-Eval and the remainder are self-constructed.
  • Evaluation Metrics: Evaluation covers normalized Aesthetic Score and Motion Smoothness for video quality, GMEScore for text controllability, and DINO-I and CLIP-I for in-domain subject consistency.Cross-domain evaluation additionally uses NANO-CLIP, Qwen-CLIP, CD-Score, and Qwen-Score.
  • Quantitative Results: 18.7% improvement in cross-domain CD score accompanies the best baseline performance in motion smoothness and text controllability, with strong gains on most subject-consistency metrics.The results demonstrate high-quality videos with competitive text controllability and subject consistency.
  • Qualitative Results: DomainShuttle preserves inherent subject features while following watercolor and 3D-animation style instructions, whereas existing methods lose consistency or fail to follow the guidance.The qualitative comparisons also cover more challenging cross-domain scenarios.
  • Ablation Studies: Combining VR-DualRoPE and CCL yields the best ablation result, while naive RoPE causes incorrect subject interactions and CCL improves CD-Score by 5.9%.CCL improves fidelity by 0.3% on CLIP and 1.5% on DINO, supporting its stronger effect on cross-domain controllability than fidelity.
  • Human Preference Evaluation: 40 volunteers rated DomainShuttle superior across video quality, text controllability, and open-domain subject consistency in a no-ties ranking evaluation.Each volunteer ranked 20 randomly selected videos using scores from 5 to 1.

5 Conclusion · Appendix

DomainShuttle is designed for high-fidelity and flexible open-domain video personalization through three specialized components. The appendix documents training-set construction, additional experiments, and demonstrations of the model’s capabilities.

  • 5 Conclusion: DomainShuttle targets high fidelity and generative flexibility for open-domain video personalization.The paper presents it as a novel architecture for this goal.
  • 5 Conclusion: Domain-MoT decouples video and reference features for domain-aware reference modeling.This component models reference features according to domain-specific requirements.
  • 5 Conclusion: Video-Reference DualRoPE separates reference-image RoPE space for fine-grained subject-level spatial modeling.Reference images and video features are placed in separate RoPE spaces.
  • 5 Conclusion: Cross-Pair Consistent Loss extracts intrinsic subject representations accurately.The loss is introduced to focus representation learning on intrinsic subject features.
  • Appendix: The supplementary materials describe training-set construction in section A.Section A provides the construction details for the training set.
  • Appendix: Section B presents additional experimental setup and results.The supplementary materials extend the paper’s experimental documentation in this section.
  • Appendix: Static HTML supplementary files demonstrate the model’s capabilities and improvements over previous methods.The authors recommend viewing these files for a direct and clear demonstration.

A Construction of the Training Dataset

The training data comprises open-source image and video personalization datasets, with image data spanning single- and multi-subject collections. Dataset filtering primarily considers aesthetic and personalization quality.

  • Dataset Composition: The training datasets are drawn from open-source image and video personalization datasets.The image personalization portion includes UNO, Echo-4o, MUSAR, and Nano-Consistent-150K.
  • Dataset Composition: UNO and Nano-Consistent-150K are single-subject datasets, whereas Echo-4o and MUSAR are multi-subject datasets.The image datasets are divided into four parts according to the passage.
  • Dataset Filtering: The datasets are primarily filtered according to aesthetic quality and personalization quality.The passage states that filtering is applied to the open-source datasets, but the supplied text truncates the remaining criterion description.

B More Experiments Results · B.1 Implementation Details

DomainShuttle uses default inference settings for Wan2.1 and Wan2.2, with model-specific sampling steps and classifier-free guidance scales. All flow shift parameters are set to 5.

  • B.1 Implementation Details: Wan2.1 uses 50 sampling steps with a classifier-free guidance scale of 3.
  • B.1 Implementation Details: Wan2.2 uses 40 sampling steps, with guidance scales of 4 at high noise and 3 at low noise.
  • B.1 Implementation Details: All flow shift parameters are set to 5 for inference.

B.2 Evaluation Details

The evaluation uses dedicated in-domain and cross-domain datasets, frame-averaged subject-similarity metrics, and specialized Nano-CLIP and Cross-Domain Score procedures. These protocols measure similarity to edited references and intrinsic subject consistency under domain shifts.

  • Evaluation Dataset: The in-domain test set contains 110 cases, including 90 OpenS2V-Eval cases and 20 human–object interaction cases.The 90 OpenS2V-Eval cases cover multi-human, multi-object, and Hard_dev categories; Hard_dev references are limited to the first four subjects when necessary.
  • Evaluation Metrics: For subject-similarity metrics, 16 video frames are uniformly sampled, scored individually, and averaged into the final result.
  • Evaluation Metrics: Nano-CLIP compares generated-video frames with reference images edited by Nano Banana Pro according to the domain-transformation prompt.The method generates edited references first, then computes frame-level CLIP cosine similarities and averages them.
  • Evaluation Metrics: The Cross-Domain Score uses GPT-5.2 to evaluate intrinsic subject consistency across domains, then averages and normalizes the scores.Detailed evaluation instructions are provided in Table 4.

B.3 Evaluation Criteria for Human Preference Evaluation · B.4 More Qualitative Comparisons.

Human preference evaluation ranks DomainShuttle against four strong baselines across 100 open-domain videos using multiple metrics. Qualitative comparisons further show superior performance across three cross-domain scenarios, with flexible text controllability and preservation of intrinsic subject features.

  • B.3 Evaluation Criteria for Human Preference Evaluation: Each volunteer evaluates 20 randomly selected open-domain videos for DomainShuttle and four strong baselines.This produces 100 videos per evaluation group.
  • B.3 Evaluation Criteria for Human Preference Evaluation: Participants rank all videos under every metric using scores from 5 as best to 1 as worst.The evaluation uses a five-point ranking scale.
  • B.3 Evaluation Criteria for Human Preference Evaluation: Ties are prohibited for every ranking and metric.This enforces strict preference ordering among the evaluated videos.
  • B.3 Evaluation Criteria for Human Preference Evaluation: The participant instructions for the human preference evaluation are provided in Tab. 5.The paper directs readers to Tab. 5 for the detailed evaluation protocol.
  • B.4 More Qualitative Comparisons.: DomainShuttle outperforms existing methods across mapping real-world subjects to fantasy domains, fantasy subjects to the real world, and interactions between both.These comparisons appear in the supplementary materials as Fig. 8 and Fig. 9.
  • B.4 More Qualitative Comparisons.: The qualitative comparisons demonstrate flexible text controllability and precise preservation of intrinsic subject features.The passage presents these capabilities as outcomes of comparisons with the baselines.

B.5 More Ablation Study

The ablation study shows that DomainShuttle remains effective without Ditto-1M, retaining cross-domain state-of-the-art performance. Qualitative comparisons further demonstrate flexible subject conversion and interaction across fantasy and real-world domains.

  • Ditto-1M Ablation: Without Ditto-1M, DomainShuttle improves the CD-Score by 13.5% over baselines, reaching 0.725 compared with Kling 1.6.The method remains effective and achieves cross-domain SOTA performance without Ditto-1M.
  • Qualitative Comparison: DomainShuttle converts fantasy-domain subjects into real-world figurines and real-world subjects into papercut-style fantasy subjects, outperforming existing methods.Existing methods fail to achieve the corresponding cross-domain conversions in the qualitative comparisons.
  • Qualitative Comparison: DomainShuttle generates interactions between real subjects and fantasy paintings while preserving the painting’s static nature, unlike Kling-1.6’s blinking character.It also maps real-world subjects to a low-poly 3D domain in the real-to-fantasy scenario.
Loading 2606.26058v1…