PaperlayerView exact revision on arXiv

Source-linked AI summary

Phantom: Subject-consistent video generation via cross-modal alignment

Lijie Liu, Tianxiang Ma, Bingchuan Li, Zhuowei Chen, Jiawei Liu, Gen Li, Siyu Zhou, Qian He, Xinglong Wu

arXiv:2502.11079v2cs.CVcs.AI

TL;DR

Subject-consistent video generation remains exploratory because systems must align textual instructions with visual references while preserving subjects through motion. Phantom redesigns joint text-image injection and trains on text-image-video triplets in a unified single- and multi-subject framework. The paper reports competitive or superior performance against commercial solutions, while uncommon subjects, complex interactions, and intricate text adherence remain challenging.

  • Problem

    Subject-to-video generation requires simultaneous alignment of text instructions and image content, while subject consistency in video remains less developed than in image generation.

  • Method

    Phantom redesigns the joint text-image injection mechanism and uses text-image-video triplet learning with dynamic feature integration for unified single- and multi-subject video generation.

  • Results

    Phantom reports competitive performance in unified single- and multi-subject generation and facial identity preservation, outperforming commercial solutions in quantitative evaluations.

  • Takeaways & Limitations

    The unified consistency-generation paradigm covers identity-preserving video generation while extending to broader single- and multi-subject references.

  • Takeaways & Limitations

    Phantom remains limited on uncommon subjects, complex multi-subject interactions, and videos requiring intricate textual adherence.

Abstract

from arXiv · show

The continuous development of foundational models for video generation is evolving into various applications, with subject-consistent video generation still in the exploratory stage. We refer to this as Subject-to-Video, which extracts subject elements from reference images and generates subject-consistent videos following textual instructions. We believe that the essence of subject-to-video lies in balancing the dual-modal prompts of text and image, thereby deeply and simultaneously aligning both text and visual content. To this end, we propose Phantom, a unified video generation framework for both single- and multi-subject references. Building on existing text-to-video and image-to-video architectures, we redesign the joint text-image injection model and drive it to learn cross-modal alignment via text-image-video triplet data. The proposed method achieves high-fidelity subject-consistent video generation while addressing issues of image content leakage and multi-subject confusion. Evaluation results indicate that our method outperforms other state-of-the-art closed-source commercial solutions. In particular, we emphasize subject consistency in human generation, covering existing ID-preserving video generation while offering enhanced advantages.

1. Introduction

Subject-to-video generation aims to align textual instructions with visual subject references while preserving identity across generated video. Phantom addresses this through cross-modal triplet learning and a unified single- and multi-subject framework.

  • Task definition: Subject-to-video generation simultaneously aligns text instructions and image content to produce subject-consistent videos.The task extends cross-modal video generation beyond conventional text-to-video and image-to-video settings.
  • Motivation: Subject consistency in video remains behind image-generation capabilities, while combining subject-to-image with image-to-video can lose identity information during motion.Back views or occlusions can cause image-to-video systems to infer a false identity when reconstructing motion.
  • Method: Phantom redesigns joint text-image injection on pretrained text-to-video and image-to-video architectures for unified subject-consistent generation.The model is based on MMDiT and uses dynamic integration of different conditions.
  • Method: A cross-modal data pipeline constructs text-image-video triplets to reduce image-content leakage and confusion among multiple subjects.The pipeline includes filtering, captioning, subject detection, and matching, while distinct appearance descriptions improve subject disambiguation.
  • Results: Phantom reports superior generation quality and advantages over commercial solutions across unified single- and multi-subject generation and identity-preserving video tasks.The claimed scope includes facial identity preservation and broader subject-consistent generation.

2. Related Work

Video generation research has progressed from latent diffusion and temporal modules toward vision foundation models, while subject-consistent video remains less developed than corresponding image methods. Existing work includes optimization, adapter, and commercial approaches, but open-source S2V methods are not yet available.

  • Video foundation models: Video foundation models evolved from U-Net latent diffusion systems with temporal modules toward DiT-based architectures such as MMDiT.Examples of earlier video systems include Make-A-Video, SVD, and AnimateDiff.
  • Subject-consistent image generation: Subject-consistent image generation includes optimization-based methods, adapter-based training such as IP-Adapter, and facial identity consistency applications.These approaches commonly use image encoders including CLIP or DINO.
  • Subject-consistent video generation: Video subject consistency has progressed more slowly than image consistency, with optimization and adapter approaches explored for facial identity preservation.Kling requires multiple videos of the same person for optimization, incurring significant computational costs.
  • Subject-to-video gap: Existing systems demonstrate consistent multi-subject video generation in general scenarios, but no open-source method currently provides the S2V task described here.Commercial software remains state of the art, motivating comparison against closed-source solutions.

3. Phantom

Phantom builds subject-to-video generation around cross-modal text-image-video alignment, combining cross-paired training data with dynamic reference-feature injection for single and multiple subjects. Its evaluation covers video quality, consistency, and comparisons across subject-generation settings.

  • 3.1. Data Pipeline: Cross-paired text-image-video data reduces copy-paste risk by pairing reference subjects across different video clips, while filtering excessively similar or dissimilar pairs.In-pair references may cause models to ignore text prompts and reproduce image content; cross-video pairing addresses this issue.
  • 3.1. Data Pipeline: The data pipeline segments examples by subjects and interactions, including people, animals, objects, multi-person interactions, human-pet interactions, and human-object interactions.Scenario-based segmentation supports quantitative supplementation of missing data types.
  • 3.2. Framework: Phantom concatenates VAE-based visual reference features with video latents and CLIP-based reference features with text features before MMDiT processing.The concatenated visual and text features enter the corresponding MMDiT branches, with injected features separated during attention calculation.
  • 3.2. Framework: Dynamic windowed attention injects reference features into each video window and supports adaptive single- or multi-subject inputs.Reference visual features are appended to windows, while text and reference CLIP features are dynamically concatenated before within-window self-attention.
  • 3.3. Training and inference: Phantom evaluates video quality and multi-subject consistency using VBench and additional model scores, with a user study for multi-subject generation.The user study surveyed 20 users because automated subject detection and matching had high error rates.
  • 3.3. Training and inference: Rectified-flow training uses a two-phase schedule, while inference accepts 1 to 4 reference images and rephrases prompts to describe each subject.More reference subjects may produce unstable results, and the training schedule includes 50k iterations followed by 20k iterations with mixed 720p data.

4. Experiments

Phantom is evaluated on newly constructed subject-to-video and portrait ID-consistency tests against commercial and open-source methods. It leads subject-consistency and prompt-following metrics, while remaining comparable to commercial systems for multi-subject generation.

  • Evaluation setup: The evaluation covers portrait ID consistency, single subjects, multiple subjects, and video quality, text-video consistency, and subject-video consistency.The authors constructed evaluation datasets and corresponding metrics because subject-to-video lacks an established benchmark.
  • Evaluation setup: 450 videos were generated from 50 reference images, three text prompts per image, and three random seeds per text-image pair.The references span humans, animals, products, environments, and clothing.
  • Baselines: Phantom is compared with Vidu, Pika, and Kling for subject-to-video, and with Hailuo and ConsisID for ID-preserving video generation.The comparison emphasizes commercial methods because available open-source systems have not matched their performance.
  • Quantitative results: Phantom leads overall subject consistency and prompt-following metrics, while its video quality is slightly worse on the reported evaluation.For multi-subject generation, a 20-user study found performance comparable to commercial solutions, with some subject-consistency advantages.
  • Qualitative results: Qualitative results show Phantom balancing subject consistency, visual effects, and text response, with particular advantage in facial ID consistency.Compared methods show issues including weak subject consistency, motion blur, reduced facial similarity, or low text response.
  • Ablation study: Combining VAE and CLIP features is more advantageous than using either encoder alone.CLIP supplies semantic information, while VAE preserves detailed information; their individual outputs show complementary artifacts and similarity limitations.

5. Conclusion

Phantom uses text-image-video triplet learning and a redesigned joint text-image injection mechanism for subject-consistent video generation. It reports competitive unified single- and multi-subject generation and facial ID preservation, outperforming commercial solutions in quantitative evaluations.

  • 5. Conclusion: Phantom achieves cross-modal alignment through text-image-video triplet learning.The method redesigns joint text-image injection and uses dynamic feature integration.
  • 5. Conclusion: Phantom demonstrates competitive performance in unified single- and multi-subject generation and facial ID preservation tasks.The conclusion reports quantitative outperformance of commercial solutions.

1. S2I+I2V vs S2V

Combining subject-to-image with image-to-video can approximate subject-to-video, but the two-stage approach has limitations in subject consistency and motion handling. These limitations motivate a specialized unified video model.

  • S2I+I2V vs S2V: S2I+I2V can achieve effects similar to S2V but existing S2I or ID-consistent image methods still show artificial artifacts and limited subject consistency.The limitation concerns both subject-consistent image generation and ID-consistent image generation.
  • S2I+I2V vs S2V: I2V cannot ensure subject consistency during motion, limiting the reliability of the combined S2I+I2V approach.The paper therefore argues for specialized video modeling with unified processing.

2. Copy-paste problem

The copy-paste problem occurs when video models leak reference-image content into generated videos, weakening text responsiveness and content diversity. Cross-video subject pairings are introduced to preserve subject matching while allowing visual changes.

  • 2. Copy-paste problem: Keyframe-conditioned reconstruction can copy-paste image content into the generated video, causing the model to disregard text prompts.The issue is especially prominent when sampled initial, middle, and final frames are used as image conditions.
  • 2. Copy-paste problem: Examples include unchanged human poses, stiff repeated cartoon movements, and videos overly similar to I2V outputs.These behaviors reduce scene-text effectiveness and content diversity.
  • 2. Copy-paste problem: Cross-video multi-subject pairings match subjects while permitting non-rigid deformations and changes in color distribution.This pairing strategy is presented as an approach to address copy-paste behavior.

3. Ablation study supplement

The ablation supplement examines text-image alignment for multi-subject generation, showing that rephrased prompts improve reference matching and reduce subject confusion.

  • Multi-subject confusion: A rephrased prompt aligns subject descriptions with the reference images and successfully matches them.For the family-of-three example, the original prompt caused confusion, whereas the rephrased prompt succeeded.
  • Data distribution: Figure 11 reports the distribution of object frequencies and classes.
  • Qualitative examples: Figure 12 presents confused and successfully improved multi-subject cases alongside their reference images.
  • Multi-subject confusion: Text-image alignment increases the success rate of multi-subject generation.Table 4 compares generation with and without text-image alignment.
  • Multi-subject confusion: The method improves multi-subject confusion without requiring additional complex data structures or model designs.

4. Data pipeline for face ID

The face-ID data pipeline extends the general subject pipeline with facial detection and alignment, while Phantom combines visual and textual features through MMDiT for unified subject training.

  • Pipeline diagrams: Figure 13 depicts the facial data processing pipeline, while Figure 14 supplements the Phantom framework diagram.
  • Face ID data pipeline: The facial data pipeline reuses scene segmentation, video filtering, and annotation from the general subject pipeline.It additionally detects faces in reference frames and calibrates detections against VLM captions using IOU.
  • Face ID data pipeline: Facial detection identifies each face and calibrates it with VLM caption results using IOU.
  • General data pipeline: Object filtering removes small, heavily overlapping, or incomplete objects after detection on three sampled video frames.The sampled frames occur at 0.05, 0.5, and 0.95 of the video timeline.
  • Phantom framework: Phantom encodes reference images with VAE and CLIP, encodes captions with a text encoder, and processes their noisy combined features through MMDiT blocks.The architecture is designed to balance detailed reconstruction with high-level information alignment.
  • Phantom framework: The architecture supports a unified training paradigm for single- and multiple-subject inputs.

7. Qualitative analysis

Qualitative comparisons show Phantom maintaining subject consistency across single- and multi-reference generation, while competing systems exhibit replication, confusion, omissions, or weaker consistency.

  • Single-reference generation: In single-subject generation, Vidu generally balances image consistency and text following but fails on one shoe case across two seeds.
  • Single-reference generation: Pika often sacrifices subject consistency for a cartoonish appearance, while Kling frequently replicates the reference as the initial frame.Kling then generates motion from text, limiting the effectiveness of textual descriptions.
  • Multi-reference generation: In multi-subject generation, all compared methods except Phantom show confusion in the three-person example.
  • Multi-reference generation: Vidu mixes the first man’s clothes with the second man’s face and adds an unrelated person, while Pika and Kling omit a person.
  • Multi-reference generation: Vidu and Pika appear more realistic in the final case, indicating stronger text responsiveness than subject consistency.
  • Scope and implications: Phantom demonstrates strong subject-consistent video generation, with potential applications including virtual try-ons, interactive storytelling, and educational content creation.
  • Scope and implications: Phantom’s limitations include uncommon subjects, complex interactions, inconsistent relative sizes, and intricate text responses.The authors attribute these challenges to dataset diversity gaps, reference-image injection rigidity, and biases in pretrained components.
Loading 2502.11079v2…