PaperlayerView exact revision on arXiv

Source-linked AI summary

Phantom: Subject-consistent video generation via cross-modal alignment

Lijie Liu, Tianxiang Ma, Bingchuan Li, Zhuowei Chen, Jiawei Liu, Gen Li, Siyu Zhou, Qian He, Xinglong Wu

arXiv:2502.11079v2cs.CVcs.AI

TL;DR

Subject-consistent video generation requires aligning textual instructions with reference-image content. Phantom learns this alignment through text-image-video triplets and a redesigned joint text-image injection mechanism. It outperforms commercial solutions in quantitative evaluations spanning single-subject generation, multi-subject generation, and facial ID preservation.

  • Problem

    Subject-consistent video generation requires simultaneous alignment with textual instructions and reference-image content.

    Source: §1 · p. 2

  • Method

    Phantom learns cross-modal alignment from text-image-video triplets using a redesigned joint text-image injection mechanism.

    Source: §5 · p. 9

  • Results

    Phantom outperforms commercial solutions in quantitative evaluations spanning single-subject generation, multi-subject generation, and facial ID preservation.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Phantom-like approaches have potential applications in film and advertising production.

    Source: §1 · p. 3

    Complex multi-subject interactions often lead to partial subject confusion or inconsistent relative subject sizes.

    Source: §8 · p. 14

Abstract

from arXiv · show

The continuous development of foundational models for video generation is evolving into various applications, with subject-consistent video generation still in the exploratory stage. We refer to this as Subject-to-Video, which extracts subject elements from reference images and generates subject-consistent videos following textual instructions. We believe that the essence of subject-to-video lies in balancing the dual-modal prompts of text and image, thereby deeply and simultaneously aligning both text and visual content. To this end, we propose Phantom, a unified video generation framework for both single- and multi-subject references. Building on existing text-to-video and image-to-video architectures, we redesign the joint text-image injection model and drive it to learn cross-modal alignment via text-image-video triplet data. The proposed method achieves high-fidelity subject-consistent video generation while addressing issues of image content leakage and multi-subject confusion. Evaluation results indicate that our method outperforms other state-of-the-art closed-source commercial solutions. In particular, we emphasize subject consistency in human generation, covering existing ID-preserving video generation while offering enhanced advantages.

1. Introduction

Phantom frames subject-to-video generation as simultaneous alignment of text instructions and reference-image content. Its cross-modal training data and dynamic reference injection support single- and multi-subject generation while addressing image copying and subject confusion.

Motivation

  • Combining subject-to-image with image-to-video can lose identity information and forgo the multi-view training advantages motivating a specialized, unified video model.S2V training naturally supplies dynamic, multi-view subject variations. In a back-to-front motion, an initial frame lacking identity cues can prevent the downstream I2V model from preserving identity.Source: §1 · p. 2

Technology

  • Cross-video multi-subject pairs train text-image-video alignment while countering reference-image copying that can weaken responsiveness to text.The pairs preserve subject matching while introducing non-rigid deformations and color variations, addressing limitations of same-video keyframe conditioning and image transformations.Source: §1 · pp. 2–3
  • Distinct appearance descriptions and an image-aware prompt rephraser address ambiguity when similar subjects receive identical textual descriptions.Descriptions must distinguish subjects and precisely match sampled reference images; the rephraser adds detailed image-content descriptions to user prompts.Source: §1 · pp. 2–3
  • Phantom unifies single- and multi-subject generation through dynamic reference injection, routing VAE [10] image details and CLIP [62] semantics into MMDiT [11].Built on pre-trained T2V and I2V models [30], it feeds VAE latents into the video branch and CLIP encodings into the text branch. Window attention reduces computation following [54], while dynamic injection preserves window size and position encoding.Source: §1 · p. 3

Concepts

  • Phantom explicitly defines subject-to-video (S2V) [4] [23] as flexible, subject-consistent generation requiring simultaneous alignment of text instructions and reference images.T2V [35] offers creative diversity but unpredictable results, whereas I2V [2] provides greater control but restricts content through first-frame copying [4] [40].Source: §1 · pp. 1–3

Significance

  • Phantom reports superior generation quality and advantages in ID generation within a unified subject-consistent video framework.The authors construct evaluation datasets and metrics for portrait IDs, single subjects, and multiple subjects, focusing comparisons on commercial methods [40].Source: §1 · p. 3

2. Related Work

Related work traces video generation from latent diffusion models to dual-stream transformers, while subject-consistency methods span optimization, adapters, and unified image-text alignment. Video approaches face computational and alignment limitations, motivating comparison with commercial S2V systems.

2.1. Video foundation model

  • Video foundation models evolved from U-Net-based latent diffusion [42] [43] with added temporal modules toward DiT [38] and dual-stream MMDiT [11] architectures.Temporal extensions include Make-A-Video [47], SVD [2], and Animatediff [13]. Stable Diffusion 3 introduced MMDiT [11], subsequently adopted by CogvideoX [59], HunyuanVideo [27], and SeedVR [54].Source: §2.1 · p. 3

2.2. Subject-consistency image generation

  • Image subject-consistency methods span identifier-binding optimization [12] [19] [20] [44] [45], frozen-backbone adapters such as IP-Adapter [60], and unified generation/editing models [5] [15] [57].Adapter approaches, also used for facial-ID consistency [14] [55], often extract semantics using CLIP [7] or DINO [37], trading detail reconstruction against flexible text response. Unified models deeply learn image-text alignment and address degradation from multiple adapters.Source: §2.2 · p. 4

2.3. Subject-consistency video generation

  • Video consistency approaches face costly per-person optimization in Kling and limited facial-text alignment in adapter-based ID-Animator [16] and ConsisID [61].Kling requires multiple uploaded videos of the same person for optimization. ID-Animator and ConsisID were validated on small datasets of around 10k.Source: §2.3 · p. 4
  • Commercial S2V systems [40] remain state-of-the-art, making closed-source comparisons crucial amid the reported absence of open-source S2V methods.ConceptMaster [23], MovieWeaver [29], and VideoAlchemist [4] demonstrate consistent multi-subject video generation in general scenarios.Source: §2.3 · p. 4

3. Phantom

Phantom combines cross-video text-image-video training triplets with VAE–CLIP reference encoding and dynamic attention injection for single- and multi-subject video generation. Rectified-flow training and subject-aware prompt rephrasing complete its training and inference pipeline.

3.1. Data Pipeline

  • Captions describing appearance, behavior, and scene guide subject detection, aligning textual subject descriptions with reference-image elements.Long videos from Panda70M [3] and in-house sources are segmented with AutoShot [64] and PySceneDetect, filtered, and captioned with Gemini [49]. An LLM extracts subject appearance descriptions that prompt a VLM [1] to detect subject boxes.Source: §3.1 · pp. 4–5
  • Cross-video subject pairing targets copy-paste behavior by rejecting overly similar or dissimilar reference pairs.An improved-CLIP image embedder [46] scores subjects across videos, while scenario-based grouping identifies missing data categories. The pipeline yields approximately one million cross-pairs, predominantly human subjects, supplemented with paired images from Subject200k [5] and OmniGen [57].Source: §3.1 · p. 5

3.2. Framework

  • Reference-image VAE features join video latents, while CLIP features join text features, combining low-level visual information with high-level semantics.The untrained input head inherits a 3D VAE [59] and LLM [58] from video foundation models [30] [54]; its vision encoder combines VAE [10] and CLIP [41] [62]. The concatenated streams enter MMDiT’s visual and text branches and separate only for attention.Source: §3.2 · p. 5
  • Dynamic reference-feature injection supports single or multiple subjects while preserving MMDiT block input/output dimensions.Video Q/K/V features are divided into windows of size 9; reference VAE features are appended dynamically, shifting existing features into the next window. Reference CLIP features join text features before windowed self-attention, after which injected reference and text outputs are extracted and averaged.Source: §3.2 · pp. 5–6

3.3. Training and inference

  • Rectified flow [31] [32] with adjusted noise sampling [11] trains velocity prediction, excluding appended feature dimensions from the loss.50k iterations at 256p/480p precede another 20k incorporating 720p data. A 0.7 VAE dropout rate balances VAE and CLIP features, preventing the former from overshadowing the latter.Source: §3.3 · p. 6
  • 1 to 4 reference images are supported at inference, with prompt rephrasing to distinguish subjects; adding more subjects can destabilize results.The rephraser describes each subject’s appearance and behavior to align prompts with training data. Sampling uses 50 Euler steps and classifier-free guidance [17] with separate image and text conditions.Source: §3.3 · p. 6

4. Experiments

Phantom leads overall subject-consistency, identity-consistency, and prompt-following evaluations, while its multi-subject performance is comparable to commercial solutions. Ablations favor combining VAE and CLIP features over either encoder alone.

4.1. Evaluation materials

  • 450 videos form the single-reference evaluation, using 50 reference images, three prompts per image, and three random seeds per image-prompt pair.References cover humans, animals, products, environments, and clothing; additional tests comprise 50 multi-reference groups with rewritten prompts and 50 portrait references spanning celebrities and ordinary individuals. The authors constructed this benchmark because no established subject-to-video benchmark existed, and excluded T2V and I2V pre-training from evaluation.Source: §4.1 · p. 7

4.2. Quantitative results

  • Phantom leads overall subject-consistency, identity-consistency, and prompt-following metrics.The video-quality evaluation reports slightly worse performance while noting strengths in other metrics; the supplied tables contain no numerical scores.Source: Table 2 · p. 8, Table 1 · p. 8, §4.2 · p. 8
  • Evaluation measures video quality, text-video alignment, subject consistency, and facial identity using complementary automated metrics.Video quality combines six VBench [24] metrics and four internal scores; text alignment uses ViCLIP [56] cosine similarity. Subject evaluation samples 10 frames for CLIP [7] and DINO [37] Direction Scores, adds grounded-sam-segmented subject scores excluding scene graphs, and measures identity with three facial-recognition models [8] [22].Source: §4.2 · pp. 7–8
  • 20 users rated multi-subject generation, finding Phantom comparable to commercial solutions, with some advantages in subject consistency.Ratings used a three-point scale: unusable, usable, or satisfactory. Human evaluation replaced automated evaluation because subject detection and matching had high error rates.Source: §4.2 · p. 8

4.3. Qualitative results

  • Qualitative comparisons show Phantom balancing subject consistency, visual quality, and text response, with a particular advantage in facial identity consistency.Vidu also balances these dimensions, whereas Pika has poor subject consistency and Kling sometimes reproduces the reference as its first frame, hindering virtual try-on; compared methods also deform rigid objects. ConsisID [61] exhibits motion blur and weak text response, while Hailuo offers strong aesthetics but loses some facial similarity.Source: §4.3 · p. 8

4.4. Ablation study

  • Combining VAE and CLIP features improves ablation results by pairing detailed visual information with higher-level semantics.CLIP-only faces are smoother but less similar, and CLIP struggles with text and patterns; VAE-only faces are sharper but can amplify unwanted details and artifacts. CLIP therefore supplements VAE with high-level information.Source: §4.4 · p. 8, Table 3 · p. 8, Figure 8 · p. 8

5. Conclusion

Phantom achieves subject-consistent video generation through text-image-video triplet learning, redesigned joint text-image injection, and dynamic feature integration. It demonstrates competitive single/multi-subject generation and facial ID preservation, outperforming commercial solutions in quantitative evaluations.

5. Conclusion

  • Phantom outperforms commercial solutions in quantitative evaluations while demonstrating competitive performance in unified single/multi-subject generation and facial ID preservation.Source: §5 · p. 9
  • Phantom learns cross-modal alignment through text-image-video triplet learning for subject-consistent video generation.Source: §5 · p. 9
  • Phantom redesigns the joint text-image injection mechanism and leverages dynamic feature integration.Source: §5 · p. 9

Supplementary Materials · 1. S2I+I2V vs S2V

Combining subject-to-image and image-to-video can achieve effects similar to subject-to-video, but image artifacts and failures to preserve identity during motion remain limitations.

1. S2I+I2V vs S2V

  • Back views or occlusions in the initial frame can cause I2V to invent a false identity as the occlusion is removed.This breaks subject consistency even when generation starts from a reference portrait.Source: §1 · p. 12
  • Existing subject-consistent or ID-consistent image generation methods [14] [21] still produce noticeable artificial artifacts.Source: §1 · p. 12
  • I2V cannot ensure that the subject remains consistent during motion.Source: §1 · p. 12
  • Subject consistency in existing subject-consistent or ID-consistent image generation methods [14] [21] still has significant room for improvement.Source: §1 · p. 12
  • Combining S2I and I2V can achieve effects similar to direct S2V, although the combined approach has difficult limitations.Source: §1 · p. 12, Figure 9 · p. 12
  • Given a reference portrait, S2I first generates an image that serves as the initial frame for I2V.Source: §1 · p. 12

2. Copy-paste problem

Copy-paste occurs when reference-image content leaks into generated videos, weakening text control and content diversity. The proposed remedy uses cross-video multi-subject pairings that preserve subject content while allowing visual variation.

2. Copy-paste problem

  • Reconstructing videos using sampled keyframes as image conditions enables shortcut learning, simplifying content understanding and allowing image content to leak into generated videos.Source: §2 · p. 12
  • Cross-video multi-subject pairings counter copy-paste by matching subject content while permitting non-rigid deformations and changes in color distribution.Source: §2 · p. 12
  • Copy-paste examples show unchanged expressions despite text prompts, stiff reference-matching movements, and I2V-like videos with weaker scene-text control and reduced content diversity.Source: §2 · p. 12, Figure 10 · p. 12

3. Ablation study supplement

Phantom addresses multi-subject appearance confusion through distinct subject descriptions during training and prompt rephrasing at inference. This text-image alignment significantly improves subject-consistent generation success.

Multi-subject confusion issue

  • Training aligns text descriptions with video subjects, ensuring each subject has a distinct description to address appearance confusion across multiple references.Appearance confusion can occur when multiple reference subjects are supplied simultaneously.Source: §3 · p. 12
  • An inference-time rephraser adjusts prompts to the training format, using explicit appearance descriptions to resolve subject confusion.Replacing a generic family prompt with descriptions of a woman in black, a girl in white, and an elderly man in a suit resolved confusion.Source: §3 · p. 12
  • Text-image alignment significantly increases the success rate of subject-consistent multi-subject generation without additional complex data structures or model designs.The supplied Table 4 caption identifies the comparison with and without alignment, but provides no numerical values.Source: Table 4 · p. 12, §3 · p. 13

4. Data pipeline for face ID · 5. Data distribution · 6. Model architecture

Phantom adds face-specific detection and matching to its data pipeline and characterizes its dataset by object counts and types. Its architecture combines image and text encoders with MMDiT blocks to balance detailed reconstruction and high-level alignment under unified single- and multi-subject training.

4. Data pipeline for face ID

  • The facial pipeline calibrates internally detected faces against caption-derived VLM [1] results using intersection over union.To enhance facial ID consistency, it reuses scene segmentation, video filtering, and annotation from the general subject pipeline.Source: §4 · p. 13
  • Face matching calculates facial similarity using Arcface [8] features and adds a deduplication operator to further calibrate recognition results.Source: §4 · p. 13

5. Data distribution

  • Human, clothes, and product predominate in the object-type distribution obtained by classifying caption nouns with LLM.The seven categories are human, animal, clothes, product, landmark, IP character, and others.Source: §5 · p. 13
  • Three frames at [0.05, 0.5, 0.95] of the video timeline undergo object detection before filtering small, highly overlapping, or VLM [1]-judged incomplete objects.Small objects include those occupying a small proportion of the frame; the retained objects determine the per-video object-count distribution.Source: §5 · p. 13

6. Model architecture

  • Phantom combines VAE- and CLIP-encoded reference images with text-encoded captions and added noise, then processes these features through multiple MMDiT blocks.Source: §6 · p. 13
  • The architecture balances detailed reconstruction with high-level information alignment while providing a unified training paradigm for single- and multiple-subject inputs.Source: §6 · pp. 13–14

7. Qualitative analysis · 8. Limitations and future work

Qualitative comparisons show Phantom avoiding the subject confusion seen in competing methods in a three-person example. Remaining limitations involve uncommon subjects, complex interactions, and intricate instructions, motivating improvements to conditioning, training, and encoders.

7. Qualitative analysis

  • Greater realism from two competitors in the final multi-subject example suggests stronger text responsiveness than subject consistency.The passage attributes this pattern to Vidu and Pika.Source: §7 · p. 14

8. Limitations and future work

  • Uncommon subjects remain difficult because training coverage lacks diversity, especially beyond human-centric scenarios.Proposed bias-aware training combines adversarial debiasing with synthetic data augmentation for underrepresented subjects.Source: §8 · p. 14
  • Complex multi-subject interactions can cause partial confusion and inconsistent relative sizes, while rigid reference injection struggles to disentangle subject features.Proposed remedies combine spatial-aware attention with physics-inspired motion priors to improve interactions and maintain consistent relative scales.Source: §8 · p. 14
  • Current cross-modal alignment limits adherence to intricate spatial layouts and temporal dynamics, motivating adaptive conditioning and additional control signals.Adaptive injection would dynamically prioritize text or images to reduce content leakage and improve text responsiveness. Depth maps and segmentation masks could supplement prompts for finer control.Source: §8 · p. 14
  • Inherited encoder biases include CLIP’s semantic oversimplification and VAE’s over-referenced details, motivating domain-specific encoder fine-tuning.Proposed adaptation targets domains such as medical imaging and animation while preserving generalization. The broader improvements could support virtual try-ons, interactive storytelling, and educational content creation.Source: §8 · p. 14

7. Qualitative analysis

  • Single-subject comparisons expose inconsistent fidelity in Vidu, cartoonish subjects in Pika, and reference-copying initial frames in Kling.Vidu performs well on consistency and text following in the first two cases but fails the shoe case with two seeds. Kling’s reference-copying behavior limits textual control.Source: §7 · p. 14, Figure 15 · p. 15
  • Phantom alone avoids subject confusion in the three-person comparison, while competing methods mix identities, introduce an unrelated person, or omit people.Another observed problem is an I2V-like generation pattern with unnatural transitions in the opening frames.Source: §7 · p. 14, Figure 16 · p. 16

Related papers

Loading 2502.11079v2…