PaperlayerView exact revision on arXiv

Source-linked AI summary

LumosX: Relate Any Identities with Their Attributes for Personalized Video Generation

Jiazheng Xing, Fei Du, Hangjie Yuan, Pengwei Liu, Hongbin Xu, Hai Ci, Ruigang Niu, Weihua Chen, Fan Wang, Yong Liu

arXiv:2603.20192v1cs.CVcs.AI

TL;DR

Personalized multi-subject video generation still struggles to maintain precise face–attribute alignment because existing methods lack explicit dependency modeling and suitable annotated data. LumosX addresses this gap with a dependency-aware data pipeline and relational attention modules, achieving state-of-the-art results on fine-grained, identity-consistent, and semantically aligned generation. Its performance remains constrained by model size and the diversity and scale of training data.

  • Problem

    Personalized multi-subject video generation requires explicit face–attribute relationships because implicit captions can confuse associations and existing datasets lack annotated dependency structures.

  • Method

    LumosX builds open-set data with MLLM-assigned subject dependencies and uses Relational Self-Attention and Relational Cross-Attention to encode face–attribute bindings.

  • Results

    LumosX achieves state-of-the-art performance in fine-grained, identity-consistent, and semantically aligned personalized multi-subject video generation.

  • Takeaways & Limitations

    The benchmark supports systematic evaluation of identity-consistent and subject-consistent generation while the framework explicitly models relationships among subjects and attributes.

  • Takeaways & Limitations

    Performance is constrained by model size and by the diversity and scale of the training data.

Abstract

from arXiv · show

Recent advances in diffusion models have significantly improved text-to-video generation, enabling personalized content creation with fine-grained control over both foreground and background elements. However, precise face-attribute alignment across subjects remains challenging, as existing methods lack explicit mechanisms to ensure intra-group consistency. Addressing this gap requires both explicit modeling strategies and face-attribute-aware data resources. We therefore propose LumosX, a framework that advances both data and model design. On the data side, a tailored collection pipeline orchestrates captions and visual cues from independent videos, while multimodal large language models (MLLMs) infer and assign subject-specific dependencies. These extracted relational priors impose a finer-grained structure that amplifies the expressive control of personalized video generation and enables the construction of a comprehensive benchmark. On the modeling side, Relational Self-Attention and Relational Cross-Attention intertwine position-aware embeddings with refined attention dynamics to inscribe explicit subject-attribute dependencies, enforcing disciplined intra-group cohesion and amplifying the separation between distinct subject clusters. Comprehensive evaluations on our benchmark demonstrate that LumosX achieves state-of-the-art performance in fine-grained, identity-consistent, and semantically aligned personalized multi-subject video generation. Code and models are available at https://jiazheng-xing.github.io/lumosx-home/.

1 Introduction

LumosX addresses face–attribute misalignment in personalized multi-subject video generation through explicit data dependencies and relational attention modules. Experiments report state-of-the-art fine-grained, identity-consistent, and semantically aligned generation.

  • Open-set personalized multi-subject video generation must integrate diverse conditioning signals while preserving temporal coherence and identity fidelity across subjects.
  • Prior methods either narrowly customize foreground faces or support versatile multi-subject inputs while neglecting dependency structures.
  • Similar subject descriptions can cause attribute entanglement and face–attribute misalignment when captions implicitly encode multi-subject relationships.
  • LumosX constructs open-set data from independent videos, using captions, visual conditions, and MLLM-inferred subject-specific dependencies to build relational priors and benchmarks.The pipeline produces explicit face–attribute correspondences for customized single- and multi-subject data.
  • Relational Self-Attention and Relational Cross-Attention explicitly encode face–attribute bindings to reinforce intra-group coherence and mitigate cross-group interference.
  • LumosX achieves state-of-the-art performance in fine-grained, identity-consistent, and semantically aligned personalized multi-subject video generation, outperforming Phantom and SkyReels-A2.

2 Related Works

Related work covers rapid progress in video generation and growing efforts toward identity-consistent and arbitrary-subject video customization.

  • Video Generation: Video generation progressed from GAN-based synthesis to latent diffusion models using denoising in compressed latent spaces.
  • Multi-Subject Video Customization: Subject-driven video generation includes methods for identity-consistent facial generation from reference images.
  • Multi-Subject Video Customization: Arbitrary-subject customization methods extend personalization beyond facial identity using visual conditioning.

3 Methods

LumosX constructs relational visual conditions from raw videos and integrates them into Wan2.1 through relational attention mechanisms that explicitly preserve face–attribute bindings during personalized multi-subject video generation.

  • Dataset Construction: The dataset pipeline generates captions, detects subjects, retrieves entity words, matches attributes, and segments subjects, objects, and clean backgrounds.Visual conditions and paired word tags are produced for subjects, objects, and backgrounds.
  • Dataset Construction: MLLM-assisted entity retrieval distinguishes repeated subject nouns and assigns attributes to the corresponding detected faces.Visual information from human detection helps resolve ambiguous repeated nouns such as multiple women.
  • Framework Overview: Condition images are encoded into image tokens, concatenated with denoising video tokens, and processed by DiT blocks built on Wan2.1.The framework uses a VAE encoder before feeding the combined tokens into the denoising transformer.
  • Relational Self-Attention: Relational Rotary Position Embedding assigns shared positional structure to face and attribute tokens within the same subject group.Face and corresponding attribute tokens share the same i-index and extend along the j- and k-indices, preserving their dependency.
  • Relational Self-Attention: Causal Self-Attention confines calculations to conditional branches and lets video denoising tokens attend unidirectionally to condition tokens.A face and its attributes are treated as a unified subject condition branch.
  • Relational Cross-Attention: The proposed cross-attention mechanism strengthens relational dependency consistency and semantic visual-condition representations through a derived similarity matrix.The method approximates the customized-mask computation to balance accuracy and efficiency.

4 Experiments

Experiments evaluate LumosX on identity-consistent and subject-consistent personalized video generation, using matched inputs and dedicated video- and subject-level metrics. LumosX achieves state-of-the-art performance in identity preservation, overall subject-consistent generation, and face–attribute association accuracy, while ablations examine its relational components.

  • Experimental Setup: The benchmark contains 500 videos spanning single-, two-, and three-subject cases, with separate identity-consistent and subject-consistent evaluation tasks.The dataset includes 220 single-subject, 230 two-subject, and 50 three-subject videos.
  • Experimental Setup: Identity-consistent evaluation measures facial similarity with ArcSim and CurSim, alongside semantic similarity between generated videos and text prompts.The evaluation compares face references with generated videos and uses ViCLIP-T for text-video semantic similarity.
  • Experimental Setup: All methods receive identical face, attribute, object, background, and prompt inputs in subject-consistent comparisons.Identity-consistent comparisons use face images and a shared global prompt; subject-consistent comparisons additionally include attribute, object, and background references.
  • Main Results: LumosX achieves state-of-the-art identity similarity across both single-face and full-benchmark identity-consistent settings.The comparisons include face-specific methods on 220 videos and general multi-subject methods on all 500 videos.
  • Main Results: LumosX achieves state-of-the-art performance in overall subject-consistent video quality and face–attribute association accuracy.The evaluation combines whole-video semantic and motion metrics with subject-level semantic, visual, identity, and facial-similarity measures.
  • Ablation Study: R2PE improves ArcSim, CSAM improves CLIP-T, and MCAM improves both CLIP-T and ArcSim in the ablation study.ArcSim peaks at r = 0.5, while CLIP-T peaks at r = 1.0; LumosX selects r = 0.5 because ArcSim better reflects face–attribute matching.

5 Conclusion

LumosX combines a dependency-aware data pipeline with relational attention to explicitly model face–attribute relationships in personalized multi-subject video generation. The paper reports state-of-the-art performance across diverse benchmarks, supported by a 1.57M-sample training dataset and structured annotation procedures.

  • Conclusion: LumosX explicitly models face–attribute dependencies for personalized multi-subject video generation.Its design binds face–attribute pairs into coherent subject groups while optimizing intra-group and inter-group correlations.
  • Conclusion: LumosX introduces Relational Self-Attention and Relational Cross-Attention to bind face–attribute pairs and optimize intra-group and inter-group correlations.The model is built upon Wan2.1’s T2V model and incorporates position embeddings with attention mechanisms.
  • Additional Evaluations: The appendix reports additional evaluations covering diverse architectures, four-or-more-subject control, temporal coherence, public benchmarks, computational overhead, and human studies.It also includes extended ablations, quantitative comparisons, visualizations, and analyses of text-based control and data-collection components.
  • Conclusion: The framework addresses missing annotated data by constructing an open-set collection pipeline with subject-specific dependency annotations.The pipeline retrieves entity words, matches subjects and attributes using visual information, and obtains condition images for subjects, objects, and backgrounds.
  • Data: The cleaned training dataset contains 1.57M videos: 1.31M single-subject, 0.23M two-subject, and 0.03M three-subject samples.Construction includes filtering for visual quality, semantic alignment, motion range, subject count, and duplicate removal.

A.4 Robustness Analysis of External Modules in the Data Collection Pipeline

LumosX emphasizes quality control in its data collection pipeline through high-reliability modules and strict filtering, while training uses Flow Matching with an MSE velocity objective.

  • Robustness and filtering: The pipeline uses Qwen-VL-32B for face–attribute extraction and FLUX for background inpainting, prioritizing visual accuracy and realism.Qwen-VL-32B is preferred over Qwen-2.5-32B, while FLUX is selected over Stable Diffusion 2.0.
  • Robustness and filtering: Strict filtering accepts entity predictions only when they appear in captions and retains inpainted samples whose foreground occupies less than 50% of the image.Grounding DINO and SAM use box_threshold=0.5 and text_threshold=0.45 to remove unreliable region–text matches.
  • Robustness and filtering: 95.2% accuracy was achieved by Qwen-VL-32B versus 78.4% by Qwen-2.5-32B for word-tag extraction.Only 1.57 million of 70 million Panda70M samples were retained, a 2.2% selection ratio.
  • Training objective: LumosX adopts Flow Matching on Wan2.1, interpolating video latents with random noise to create training inputs and learning continuous-time dynamics.The formulation follows Rectified Flows and uses a timestep sampled from a logit-normal distribution.
  • Training objective: The model minimizes mean squared error between predicted and ground-truth velocity fields, conditioning on text and reference visual embeddings.u(zt, ctext, zc, t; θ) denotes the predicted velocity, while ctext and zc denote text and visual embeddings.

B.3 Generalization of LumosX to Diverse T2V Architectures

LumosX’s relational modules are designed to transfer across DiT-style architectures and support more subjects at inference, but scalability is bounded by positional extrapolation and base-model capacity.

  • Architecture generalization: R2PE, CSAM, and MCAM are architecture-agnostic modules compatible with alternative DiT-style backbones, including HunyuanVideo and MAGI-1.R2PE reorders relative position indices, while CSAM and MCAM act as self- and cross-attention masks.
  • Identity consistency: In multi-face identity preservation, LumosX outperforms baselines, whereas Phantom performs better in the single-face setting on ArcSim and CurSim.The comparison is reported in Tab. 5 across different numbers of faces.
  • Diverse samples: On additional online-source samples, LumosX outperforms other approaches on most subject-specific metrics, including CLIP-T, CLIP-I, DINO-I, ArcSim, and CurSim.The evaluation contains 50 cases spanning one-, two-, and three-subject scenarios, without ground-truth videos for ViCLIP-V measurement.
  • Subject scalability: LumosX consistently outperforms baselines in four-subject evaluation without retraining, with no significant performance drop from the three-subject setting.The model is trained on videos containing up to three subjects and evaluated quantitatively on 50 four-subject videos.
  • Subject scalability: Beyond four subjects, positional extrapolation instability and the Wan2.1-1.3B-T2V backbone constrain reliable scaling.In a 10-subject setting, the base model generates only nine people and exhibits severely degraded facial quality.
  • Temporal coherence: LumosX outperforms other methods across Subject Consistency, Background Consistency, Motion Smoothness, and Face Consistency.These temporal-coherence metrics are evaluated on 500 subject-consistent videos.

C.6 Evaluation on PublicPersonalizationBenchmark

On MSRVTT-personalization, LumosX remains competitive despite a benchmark mismatch that omits decoupled face–attribute references; additional analyses examine component effects and text-only control.

  • Public benchmark: MSRVTT-personalization evaluates LumosX in subject-mode using one subject reference and one background reference, comparing it with SkyReels-A2 and Phantom.The benchmark is single-subject and defines subjects as either human or object categories.
  • Public benchmark: The benchmark does not test LumosX’s explicit face–attribute relational modeling because human subjects are provided as holistic entities without decoupled references.SkyReels-A2 also uses a substantially stronger Wan2.1-14B-T2V backbone, while LumosX and Phantom use Wan2.1-1.3B-T2V.
  • Public benchmark: Despite these disadvantages, LumosX achieves the second-best overall performance, indicating robustness outside its primary multi-subject setting.This result is reported for the MSRVTT-personalization comparison.
  • Component analysis: Without the proposed modules, qualitative results show identity confusion, including incorrect age and skin-color attributes across subjects.R2PE binds face–attribute dependencies, while MCAM strengthens intra-group and inter-group correlations among subject groups.
  • Component analysis: Increasing MCAM’s control parameter r from 0.1 to 0.5 improves identity consistency and video quality, while r = 0.1 leaves facial similarity insufficient.The qualitative comparison reports further improvement when r reaches 1.0.
  • Text-only control: Under text-only attribute control, CLIP-T scores are higher because attributes are directly specified in prompts, and CLIP-I shows similar trends.The evaluation uses the same metrics as subject-consistent video generation.
  • Alternative pipeline: LumosX outperforms the UNO + Wan2.1-I2V-14B pipeline, particularly on ViCLIP-V, when comparing multi-subject customized image-to-video generation.The comparison uses UNO to create a customized image before video synthesis with Wan2.1-14B-I2V.

C.11 Discussion of the Importance of the Inpainting Model in Data Collection Pipeline

FLUX was selected for background inpainting after evaluation against Stable-Diffusion-2, because it performed better on both realism metrics and human judgments. Qualitative comparisons further showed that inpainting artifacts propagate into generated videos and degrade their quality.

  • The human assessment asked GPT-4o to compare inpainting results using pixel-level consistency, scene continuity, and overall plausibility.The model output only the choice of the better result.
  • FLUX achieves better background-inpainting performance than Stable-Diffusion-2 under both reported evaluation metrics.The evaluation used 2,130 test cases and a 100-case GPT-4o human comparison.
  • Background inpainting artifacts propagate into downstream generated videos, degrading overall video quality.The qualitative comparison used the two inpainting outputs as inputs to video generation.
  • The user study evaluated LumosX, SkyReels-A2, and Phantom on face–attribute alignment, face similarity, video naturalness, and prompt adherence.It covered 24 video cases and involved 30 participants.
  • LumosX achieves superior performance across all four user-study evaluation dimensions.Participants ranked the three methods, with weighted scores computed from the rankings.

C.13 Analysis of Computational Overhead and Latency in LumosX

LumosX’s relational attention mechanisms introduce minimal additional inference overhead. The reported design choices preserve efficiency while supporting custom masking and relational computation.

  • R2PE incurs no extra compute or memory overhead because it reorders relative position indices rather than adding parameters or operations.This comparison is reported between rows 1 and 2.
  • CSAM introduces only modest inference overhead after replacing FlashAttention 2.0 with MagiAttention to support custom masks.The comparison is reported between rows 4 and 5.
  • MCAM does not noticeably affect Cross-Attention latency because its keys come from relatively short 512-token T5 sequences.The additional cost of computing the dynamic scaling matrix remains relatively small.
  • LumosX achieves high computational efficiency with minimal additional inference overhead.The summary compares row 6 with rows 1 and 4 under the same video-customization scenario on an H20 GPU.

D.2 Additional Results of Subject-Consistent Video Generation

LumosX supports flexible multi-subject foreground–background customization while preserving subject identity and matching faces with their corresponding attributes. Compared with Phantom, it avoids quality degradation and cartoon-like artifacts as reference conditions increase.

  • LumosX achieves superior subject consistency by matching human faces with their corresponding attributes and maintaining reference identity.
  • LumosX generates natural and realistic videos for multi-subject foreground–background customization.
  • Unlike Phantom, LumosX avoids quality degradation, visual artifacts, and unintended cartoon-like styles as reference condition images increase.

E Limitations and Future Work

LumosX remains limited by model size and the diversity and scale of its training data. Future work targets larger models and datasets, along with motion-aware constraints for dynamic multi-subject interactions.

  • Limitations: LumosX remains constrained by model size and by the diversity and scale of its training data.The authors state that its performance has not yet reached its full potential.
  • Future Work: Future work plans to deploy LumosX on Wan2.1-14B-T2V and train it on a larger-scale, higher-quality, and more diverse dataset.
  • Future Work: The authors propose motion descriptions and motion cues in MCAM to improve alignment for dynamic behaviors and multi-subject interactions.Examples include walking, running, hugging, handshaking, and passing objects.
Loading 2603.20192v1…