PaperlayerView exact revision on arXiv

Source-linked AI summary

HOMIE: Human-object Centric Video Personalization via Multimodal Intelligent Enhancement

Yiyang Cai, Nan Chen, Rongchang Xie, Junwen Pan, Chunyang Jiang, Cheng Chen, Wen Zhou, Zhenbang Sun, Wei Xue, Wenhan Luo, Yike Guo

arXiv:2607.18217v2cs.CV

TL;DR

Existing HOCVP methods struggle to preserve subject fidelity while modeling human-object interactions and understanding intra-subject references such as logos, OCR maps, and multi-view images. HOMIE unifies inter- and intra-subject personalization with MLLM-guided global multimodal attention and modality-reference embeddings, achieving state-of-the-art performance across diverse HOCVP tasks.

  • Problem

    Existing HOCVP methods struggle to balance subject fidelity with precise interactions and to infer correspondence among intra-subject references, including abstract references such as logos.

  • Method

    HOMIE unifies inter- and intra-subject HOCVP using Global Multimodal Guidance and Modality-Reference Embedding to inject MLLM knowledge and associate reference tokens.

  • Results

    HOMIE achieves state-of-the-art performance across diverse HOCVP tasks, including a 21.8% improvement in OCR accuracy relative to SkyReels-V3.

  • Takeaways & Limitations

    HOMIE supports robust personalization across inter-subject, logo, OCR-map, and multi-view settings, with particularly strong results in challenging logo and multi-view generation scenarios.

  • Takeaways & Limitations

    Because HOMIE is built on Wan-T2V-14B, it currently shares the foundation model’s approximately 5-second short-clip duration limit.

Abstract

from arXiv · show

Human-object centric video personalization (HOCVP) is a core task within subject-driven video generation. However, existing methods suffer from two key limitations. First, most approaches focusing on inter-subject personalization still struggle to strike a balance between high subject fidelity and accurate interaction patterns between humans and diverse objects, especially when objects represent abstract concepts such as logos. Second, while intra-subject references (e.g., OCR maps, multi-view inputs) are expected to enhance subject fidelity, most existing works lack mechanisms to understand such latent correspondence. To address both challenges, we propose HOMIE, an HOCVP framework that tackles both inter- and intra-subject input settings in a unified manner. Compared to previous approaches, HOMIE proposes a better MLLM integration strategy to extract knowledge of reference-level relationships without compromising the controllability of text encoders or incurring costly re-alignment. Specifically, we introduce global multimodal guidance within self-attention to better align MLLM-derived semantic features with VAE tokens. Furthermore, we propose modality-reference embedding to differentiate tokens from MLLM features and VAE tokens and associate intra-subject reference image tokens. Extensive experiments validate that our method achieves state-of-the-art performance across various HOCVP tasks. Project Page: https://yiyangcai.github.io/homie-page.github.io/

1 Introduction

HOMIE addresses inter- and intra-subject HOCVP in a unified framework by using MLLMs to reason about implicit reference relationships. Its Global Multimodal Guidance and Modality-Reference Embedding modules improve multimodal interaction modeling and reference association without changing the stated task scope.

  • Motivation: HOCVP synthesizes videos from human and object reference images, enabling diverse human-object interactions and broad practical applications.
  • Challenges: Existing methods distinguish inter-subject references from intra-subject references, but both settings require better reasoning about reference relationships.Intra-subject references include multi-view images or OCR maps for improving multi-view consistency or text fidelity.
  • HOMIE: HOMIE unifies inter- and intra-subject HOCVP by integrating MLLM knowledge from unified multimodal inputs.The framework is designed to leverage relationships among references for video personalization.
  • HOMIE: Global Multimodal Guidance injects MLLM-derived global representations into video-token query-key computation, enriching self-attention with temporal interaction information.This strategy injects multimodal knowledge into the self-attention process.
  • HOMIE: Modality-Reference Embedding distinguishes tokens by input modality while associating multiple intra-subject references in a unified representation.It captures intra-subject associations and inter-subject distinctions to improve identity fidelity in intra-subject HOCVP.

2 Related Work

Related work spans video diffusion models and video personalization. Video diffusion research has progressed across T2V and I2V paradigms, while personalization has expanded from single identities toward challenging multi-subject generation.

  • Video Diffusion Models: Video diffusion models extend text-to-image diffusion techniques into video generation through architectural advances and improved data curation.These models generally operate within text-to-video (T2V) or image-to-video (I2V) paradigms.
  • Video Diffusion Models: Early U-Net-based video diffusion frameworks are increasingly superseded by state-of-the-art diffusion transformers.
  • Video Personalization: Video personalization generates subject-consistent videos from reference images while retaining fine-grained target features, progressing from single-identity to multi-subject generation.Recent multi-subject personalization methods use decoupled cross-attention to mitigate attribute mixing.

3 Method

HOMIE extends a DiT-based text-to-video model with multimodal conditioning from videos, reference images, and MLLM features for controllable, high-fidelity HOCVP. Global Multimodal Guidance aligns semantic MLLM information with video tokens, while Multimodal-Reference Embedding distinguishes modalities and binds reference tokens by subject identity.

  • Backbone: HOMIE uses a DiT-based text-to-video backbone with a 3D VAE, text-encoder cross-attention, and flow matching for video generation.The 3D VAE compresses raw videos into latent space, while textual prompts are injected through cross-attention.
  • Multimodal input: The conditioning set c = {c_txt, c_img} incorporates reference images and MLLM-derived multimodal features for controllable HOCVP.The DiT predicts velocity from noisy video latents conditioned on multimodal inputs.
  • Multimodal input: An MLLM processes reference images and prompts to extract semantic features, which are concatenated with target-video and reference-image latent tokens.This input-side combination enables interactions among tokens from different modalities.
  • Global Multimodal Guidance: Global Multimodal Guidance injects pooled MLLM semantics into video-token queries and keys through affine modulation before standard self-attention.Lightweight projection networks derive scaling and shifting factors, addressing the distributional gap between MLLM and 3D-VAE features.
  • Multimodal-Reference Embedding: Multimodal-Reference Embedding disentangles modality and subject representations while assigning shared embeddings to intra-subject references and distinct embeddings to inter-subject references.Reference embeddings are selectively added to tokens according to their original reference entities.

4 Experiments

HOMIE is evaluated on inter- and intra-subject HOCVP using quantitative, qualitative, ablation, and user-study comparisons. It achieves strong video quality, text following, subject consistency, OCR accuracy, multi-view fidelity, and multimodal reasoning performance across these evaluations.

  • Evaluation: The evaluation uses 200 human-object combinations, including 140 samples with more than two reference images, spanning inter- and intra-subject settings.The dataset includes diverse human identities and object categories, with references drawn from open-source and AIGC-generated sources.
  • Quantitative Comparison: HOMIE achieves state-of-the-art video quality and text-following performance among open-source methods while maintaining competitive subject consistency across diverse reference settings.It also improves OCR accuracy by 21.8% relative to SkyReels-V3, validating its use of OCR maps.
  • Quantitative Comparison: HOMIE delivers competitive multi-view intra-subject HOCVP performance while faithfully synthesizing diverse views of the same object and preserving corresponding subject fidelity.This result is reported alongside its OCR-map performance in the intra-subject evaluation.
  • Qualitative Comparison: Qualitatively, HOMIE balances subject consistency and text following on challenging inter-subject tasks, including complex four-subject scenes and logo personalization.Other methods exhibit missing subjects, incorrect attire, body degradation, less fluent motion, or failure to link logos with semantically relevant objects.
  • Ablation Study: Ablations show that GMG links logos to semantically relevant objects, whereas MRE captures semantic links among OCR-map and multi-view reference images.Without MRE, reference inputs are treated independently and can unintentionally appear in generated videos; integrating MLLM features into UmT5 fails on the logo-personalization example.
  • User Study: In a 40-participant user study, HOMIE consistently outperforms Phantom, Kling, SkyReels-V3, and UniVideo for overall quality, text following, and subject consistency.The study compares five videos across 20 samples, with participants selecting the best video for each metric.

5 Conclusion

HOMIE integrates MLLM knowledge into HOCVP through multimodal inputs and global multimodal guidance, while learnable modality-reference embeddings address intra-subject references.

  • Conclusion: HOMIE integrates MLLM knowledge to improve HOCVP performance.The framework is designed to enhance MLLM feature interactions within the video diffusion model.
  • Conclusion: Its multimodal input paradigm is paired with global multimodal guidance to enhance MLLM feature interactions within the video diffusion model.
  • Conclusion: Learnable modality-reference embeddings help distinguish token modalities in scenarios involving intra-subject references.

A Appendix

The appendix documents experimental setup and evaluation, adds qualitative and generation results, and discusses abstract-concept personalization, ethics, and limitations. Original video samples are available on the demo webpage.

  • A.1 Experimental Setup: Section A.1 details the experimental setup and HOCVP training strategies, including training costs for HOMIE and its counterparts.It specifically reports the training costs of HOMIE and competing methods.
  • A.2 Evaluation: Section A.2 describes evaluation baselines, datasets, metrics, and user-study guidelines.The evaluation section covers both comparison criteria and user-study procedures.
  • A.3 Qualitative Comparisons: Section A.3 presents qualitative comparisons across complex inter-subject and intra-subject personalization tasks, including SkyReels-A2, BindWeave, UniVideo, and VINO.These comparisons specifically include methods integrating multimodal large language models.
  • A.4 Further Results: Section A.4 showcases additional HOMIE generation results and practical application cases.The section extends the reported results with further examples and applications.
  • Supplementary Resources: Original video samples for examples in the main submission and supplementary materials are provided on the project demo webpage.The cited webpage is https://homie-demo.github.io/.

A.1 Implementation Details · A.1.1 Model Designs

HOMIE extracts Qwen3-VL-2B-Thinking hidden states for multimodal conditioning and applies global multimodal guidance throughout Wan’s self-attention stack. The implementation fixes the MLLM prompt across training and inference and constrains extracted features to specified length and dimensionality.

  • A.1.1 Model Designs: HOMIE uses Qwen3-VL-2B-Thinking to generate MLLM features.The last hidden state is extracted in advance during training.
  • A.1.1 Model Designs: The system prompt Psys remains identical during training and inference.It follows the prompt specified in Table 4.
  • A.1.1 Model Designs: The extracted MLLM hidden states have a maximum length of 1024.
  • A.1.1 Model Designs: The MLLM hidden-state dimension is 2048.
  • A.1.1 Model Designs: Global multimodal guidance is applied to every self-attention layer.This design promotes interaction between multimodal features and video tokens.
  • A.1.1 Model Designs: Global multimodal guidance spans all 40 DiT blocks in the Wan model.

A.1.2 Training Strategies

HOMIE uses a multi-stage training strategy that progresses from single-reference identity learning to multiple-reference semantic learning and high-resolution fine-tuning. By retaining the original text encoder while integrating an MLLM, the design improves training efficiency compared with approaches that replace the encoder entirely.

  • Multi-Stage Training: HOMIE trains progressively on single-reference, multiple-reference, and high-resolution multiple-reference video datasets to accommodate increasing task difficulty.The three-stage configuration is reported in Table 5.
  • Single-Subject Training: The single-subject stage uses 300K samples to learn diverse subject representations while preserving identity consistency.A randomly selected reference embedding is assigned to the single reference, ensuring embeddings are optimized for distinct subject identities.
  • HOC Training: The HOC training stage uses multiple-reference videos to enhance semantic understanding of interactions between subjects and objects relevant to logos.Reference embeddings are selected in a fixed order, which is also followed during inference.
  • High-Resolution Training: The final stage fine-tunes on filtered 720P multiple-reference videos after two stages using uniformly formatted 480P data.This short-term fine-tuning enhances high-resolution video generation.
  • Training Efficiency: Retaining the original text encoder makes HOMIE more efficient than UniVideo and VINO, which replace it with MLLMs and require extensive control-pipeline realignment.Preserving the encoder maintains its basic textual control capabilities while avoiding the substantially greater computational resources required by full replacement.

A.2 Evaluation Details

This section provides details on the backbones and technical implementations used for comparison with previous state-of-the-art methods.

  • Table 7 details the backbones and technical implementations compared with previous SOTAs.

A.2.1 Discussion on Prior SOTA Methods … A.5.3 Limitation

The supplementary sections detail HOMIE’s comparison setup, evaluation protocol, qualitative and practical results, abstract-concept discussion, ethical concerns, and inherited short-video limitation. Together, they report broad HOCVP coverage while identifying deployment risks and future extensions.

  • A.2.1 Discussion on Prior SOTA Methods: HOMIE is compared using Wan2.1-T2V-14B and Wan2.2-T2V-14B versions, alongside UniVideo and VINO, while Kling-1.6 is the only closed-source commercial baseline.UniVideo and VINO use HunyuanVideo with MLLM-based text encoders, but their training pipelines differ from HOMIE’s setting.
  • A.2.2 Evaluation Datasets: The evaluation set contains 200 human-object-centric samples spanning inter-subject and intra-subject references, including abstract concepts, OCR maps, and multi-view inputs.At least 140 samples contain three or more reference images, while 25 inter-subject samples involve abstract concept personalization.
  • A.2.3 Evaluation Metrics: Evaluation measures video quality, text following, and subject consistency through aesthetic score, motion smoothness, GMEScore, face similarity, DINO-I similarity, object similarity, and OCR accuracy.OCR accuracy uses normalized Levenshtein similarity between reference-image text and sampled-frame OCR results.
  • A.2.4 Evaluation Guidance of User Study: The user study asks participants to rate HOMIE and comparison methods on Video Quality, Text Following, and Subject Consistency.These three criteria define the evaluation guidance presented for the user study.
  • A.3 Extended Qualitative Comparisons: Additional qualitative comparisons show HOMIE handling animate characters, diverse objects, abstract concepts such as logos, OCR maps, and multi-view references more competitively than prior methods.Other methods exhibit incorrect subject counts, limited consistency, and unnatural interactions in challenging examples.
  • A.4 More Results: HOMIE produces diverse human-object combinations across daily necessities, instruments, clothing, animals, furniture, and abstract concepts, covering both inter-subject and intra-subject scenarios.The paper also demonstrates applications including live-stream product promotion and creative video generation at stable 1280 × 720 resolution.
  • A.5 More Discussions / A.5.1 Discussion of Abstract Concept Personalization: BrandFusion similarly uses MLLMs and T2V models for logo placement, whereas HOMIE’s contribution is a unified framework integrating MLLM reasoning through model design.This discussion supports using MLLM knowledge for abstract concept personalization while distinguishing HOMIE’s integration strategy.
  • A.5.2 Ethical Concerns / A.5.3 Limitation: HOMIE acknowledges deepfake misuse risks and a current approximately 5-second duration limit inherited from Wan-T2V-14B, with future work targeting long-video generation and ethical safeguards.The paper plans strict compliance guidelines for releases and notes that evaluated baselines share the same short-clip constraint.
Loading 2607.18217v2…