Source-linked AI summary
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, Ying Shan
TL;DR
Personalized human-image generation struggles to combine efficiency, identity fidelity, and flexible text control. PhotoMaker encodes multiple ID images into a stacked ID embedding and trains with an ID-oriented dataset pipeline. The method reports high-quality, diverse, efficient, and controllable generation, while supporting identity mixing and other applications.
Problem
Existing personalized generation methods do not simultaneously provide high efficiency, strong ID fidelity, and flexible text controllability.
Method
PhotoMaker stacks multiple input-image encodings into a unified ID embedding and uses an ID-oriented data construction pipeline for training.
Results
PhotoMaker reports high-quality and diverse generation, promising editability, high inference efficiency, and strong ID fidelity compared with other methods.
Takeaways & Limitations
The framework supports applications including age or gender changes, restoring people from old photos or artworks, and identity mixing.
Takeaways & Limitations
The method focuses on one generated person, performs relatively poorly for full-length portraits, has less precise age transformation than some GANs, and inherits SDXL and dataset biases.
Abstract
from arXiv · showhide
Recent advances in text-to-image generation have made remarkable progress in synthesizing realistic human photos conditioned on given text prompts. However, existing personalized generation methods cannot simultaneously satisfy the requirements of high efficiency, promising identity (ID) fidelity, and flexible text controllability. In this work, we introduce PhotoMaker, an efficient personalized text-to-image generation method, which mainly encodes an arbitrary number of input ID images into a stack ID embedding for preserving ID information. Such an embedding, serving as a unified ID representation, can not only encapsulate the characteristics of the same input ID comprehensively, but also accommodate the characteristics of different IDs for subsequent integration. This paves the way for more intriguing and practically valuable applications. Besides, to drive the training of our PhotoMaker, we propose an ID-oriented data construction pipeline to assemble the training data. Under the nourishment of the dataset constructed through the proposed pipeline, our PhotoMaker demonstrates better ID preservation ability than test-time fine-tuning based methods, yet provides significant speed improvements, high-quality generation results, strong generalization capabilities, and a wide range of applications. Our project page is available at https://photo-maker.github.io/
1. Introduction
PhotoMaker addresses the efficiency, identity-fidelity, and controllability limitations of personalized human-image generation by using multiple ID images and a stacked ID embedding. It also introduces ID-oriented data construction and supports flexible identity and attribute editing.
- Motivation: DreamBooth-based customization achieves high ID fidelity but requires manual image collection and 10–30 minutes of tuning for each identity.These costs consume substantial computing resources, especially as generation models grow larger.
- Motivation: Single-image customization can memorize expressions and viewpoints unrelated to identity, while limited ID evidence can reduce fidelity and editability.PhotoMaker therefore trains with viewpoint, expression, accessory, and multi-image variation for each identity.
- PhotoMaker: PhotoMaker replaces the text class embedding with a stacked ID embedding formed from multiple input-image encodings, allowing cross-attention to integrate identity and context without extra network modules.Each subpart corresponds to an input ID image, producing a unified identity representation.
- Results: About 10 seconds with four ID images is reported as approximately 130× faster than DreamBooth on one NVIDIA Tesla V100.The method also accepts any number of ID images during inference while retaining tuning-free generation efficiency.
- Results: PhotoMaker reports high-quality, diverse, editable generation with strong ID fidelity, including changes in viewpoint, accessories, expressions, gender, and age.The conclusion additionally cites applications such as restoring people from old photos or artworks and identity mixing.
- Applications: Different identities can be combined at inference to create a new customized identity while retaining characteristics from the inputs.The mixing ratio can be adjusted through prompt weighting or the proportions of input ID images.
2. Related work
Personalized diffusion generation methods commonly rely on test-time optimization or pretraining to reduce tuning costs. However, even accelerated approaches still require extensive fine-tuning for each new concept, limiting efficiency and applications.
- Diffusion Models: Diffusion models have enabled substantial progress in text-conditioned image generation through larger datasets, stronger foundational models, improved encoders, and better controllability.
- Personalization: Personalized diffusion methods mainly divide into test-phase optimization approaches and methods that reduce tuned parameters or use large-scale pretraining.DreamBooth and Textual Inversion are cited as representative test-time optimization methods.
- Personalization: Despite these advances, existing methods still extensively fine-tune a pretrained model for each new concept, making personalization time-consuming and restricting applications.
3. Method
PhotoMaker combines multiple image and text embeddings into a stacked ID embedding that diffusion cross-attention integrates for personalized generation. Its method also includes an ID-oriented dataset construction pipeline with filtering, segmentation, and caption marking.
- Framework: PhotoMaker targets photo-realistic human-image generation from a few ID images while changing generated content or attributes under text control in one forward pass.It also supports mixing multiple input identities while maintaining promising ID fidelity and text editability.
- Encoders: The image encoder extracts embeddings aligned with the original text representation space using CLIP.
- Stacked ID Embedding: The pipeline fuses each image embedding with its corresponding class embedding, concatenates the fused embeddings, and feeds the resulting stacked ID embedding to diffusion cross-attention layers.This lets the model adaptively merge identity content with the generated context.
- Stacked ID Embedding: Cross-attention uses Q = WQ · ϕ(zt), K = WK · t∗, and V = WV · t∗, while LoRA residuals are trained in attention matrices to improve perception of ID information.
- Data Construction: The ID-oriented dataset pipeline addresses the lack of identity annotations and limited scene richness in existing human datasets.It constructs identity-grouped data with varied viewpoints, attributes, and scenarios, and generates captions marking class words.
- Data Construction: Dataset construction includes image downloading, face detection and filtering, identity verification, cropping and segmentation, and captioning with class-word marking.The described components use resolution and face-size thresholds, ArcFace verification, Mask2Former segmentation, and BLIP2 captioning.
4. Experiments
Experiments evaluate PhotoMaker’s generation quality, applications, and design choices. Results indicate that stacked ID embeddings improve identity fidelity and flexibility, while additional inputs may trade off text controllability.
- Experimental setup: PhotoMaker is evaluated with SDXL-based implementations, 1024 × 1024 training data, CLIP ViT-L/14 image embeddings, and four input ID images for applications.Training uses Adam on 8 NVIDIA A100 GPUs for two weeks with batch size 48.
- Applications: PhotoMaker generally achieves high-quality generation, promising editability, and strong identity fidelity across five identities and corresponding recontextualization prompts.The comparison includes DreamBooth, Textual Inversion, FastComposer, and IPAdapter.
- Applications: PhotoMaker supports applications including realistic reconstruction of people from artwork or old photos, age or gender changes, identity mixing, and stylization.Identity mixing combines characteristics from different input identities, while prompt changes support age or gender editing.
- Input-image ablation: Using more input ID images improves ID-fidelity metrics, with the clearest improvement occurring when increasing inputs from one to two.The growth rate of ID-related metrics later decelerates as more images are added.
- Input-image ablation: Increasing input images enhances ID similarity but produces a linear decline in CLIP-T, indicating a trade-off between text controllability and ID fidelity.The Dwayne Johnson example also shows reduced gender-editing capability with more input images.
- Embedding ablation: The stacking composition achieves the highest ID fidelity while preserving facial diversity and offers flexibility for arbitrary input counts and identity mixing.Ablations compare averaging, adaptive linear projection, and stacking.
- Training-data ablation: Training with multiple varied images of the same identity provides better ID fidelity than single-image embedding strategies.
5. Conclusion
The paper concludes that PhotoMaker uses stacked ID embeddings to generate realistic human photos with efficient, diverse, editable, and identity-preserving customization. It also enables applications such as age or gender changes, historical-photo reconstruction, and identity mixing.
- PhotoMaker is an efficient personalized text-to-image method for generating realistic human photos.
- Stacked ID embedding preserves identity information while supporting high-quality, diverse, editable generation with strong ID fidelity.
- PhotoMaker enables age or gender changes, bringing people from old photos or artworks into realistic scenes, and mixing multiple identities.
A. Dataset Details
The constructed training dataset is organized by identity and provides multiple contextual images for each identity, supporting PhotoMaker’s multi-image training requirements.
- The constructed dataset contains about 112K images classified by about 13,000 identity names.Each image includes an identity mask and an annotated caption.
B. User Study
A user study compares PhotoMaker with DreamBooth, FastComposer, and IPAdapter across identity fidelity, quality, facial diversity, and text fidelity. PhotoMaker receives the largest vote proportion overall and is especially favored for quality, diversity, and text fidelity.
- Study design: The study presents users with 20 text-image pairs and four randomly generated images from each method for every pair.Users evaluate identity similarity, image quality, facial diversity, and text fidelity.
- Results: PhotoMaker receives the largest vote proportion across ID fidelity, generation quality, face diversity, and text fidelity.
- Results: PhotoMaker has advantages across all four evaluation dimensions, especially generation quality, diversity, and text fidelity.DreamBooth is reported as the second-best method overall in balancing these dimensions.
C. More Ablations
PhotoMaker supports flexible identity mixing by varying the contributions of different input IDs. Prompt weighting offers a more photo-efficient way to adjust the merge ratio.
- Adjusting ratio: The merge ratio between identities can be controlled through either input-image proportions or prompt weighting.These mechanisms let users make the generated identity closer to or farther from a selected input ID.
- Adjusting ratio: Prompt weighting requires fewer photos than controlling the number of input images when adjusting identity mixtures.
- Adjusting ratio: Both identity-mixing controls demonstrate PhotoMaker’s flexibility.
D. Stylization Results
PhotoMaker supports stylized human-image generation while preserving identity attributes, demonstrating robust generalizability across styles.
- D. Stylization Results: PhotoMaker allows stylization while preserving the attributes of the input identity.The paper presents stylization results across different input IDs and style prompts.
- D. Stylization Results: The stylization capability demonstrates robust generalizability of the proposed method.
E. More Visual Results
The visual results illustrate PhotoMaker’s control over identity mixtures, stylization, difficult identities, historical depictions, age or gender changes, and text-controlled synthesis.
- Identity mixing: PhotoMaker adjusts identity mixtures by changing either the proportion of IDs in the input pool or the prompt weight assigned to an ID.Figure 10 varies image proportions, while Figure 11 progressively increases one identity’s embedding weight.
- Stylization: PhotoMaker transfers input identities across multiple artistic styles while preventing realistic-only outputs.The stylization examples use prompts such as animation, comic-book, Van Gogh, and Ghibli styles.
- Non-celebrity identities: PhotoMaker successfully generates corresponding non-celebrity identities that SDXL cannot generate from name- or position-based prompts.
- Artwork and old photos: PhotoMaker generates photo-realistic images from identities represented in statues and oil paintings.The paper presents this as bringing past celebrities or artwork subjects back to reality.
- Age or gender changes: Changing the class word enables age or gender changes while retaining characteristics of the original identity.
- Identity mixing: PhotoMaker blends characteristics from different identities into a new identity that remains controllable by text prompts.The paper presents this as a flexible identity-mixing application enabled by the stacked ID embedding.
F. Limitations
The paper identifies several limitations concerning multi-person identity control, portrait framing, age transformation precision, and inherited biases.
- F. Limitations: PhotoMaker cannot simultaneously control the identities of multiple generated people in one image.
- F. Limitations: PhotoMaker performs better on half-length portraits than on full-length portraits.
- F. Limitations: Its age-transformation ability is less precise than that of some GAN-based methods.More precise control may require modifying the captions of the training dataset.
- F. Limitations: Because PhotoMaker relies on SDXL and its constructed dataset, it inherits their biases.
G. Broader Impact
PhotoMaker targets high-quality human-image generation while preserving input identity, with efficiency, diversity, and controllability. It supports personalized applications and may provide a strong baseline for broader computer-vision datasets, while raising ethical concerns about misuse.
- PhotoMaker generates high-quality human images while maintaining similarity to the input identity, with efficient and controllable generation.
- Its data creation pipeline enables more diverse datasets with varied poses, actions, and backgrounds for robust and generalizable computer-vision models.
- The method supports practical uses including realistic characters, immersive virtual-reality experiences, and quickly customized digital portraits.
- High-fidelity human-image generation raises risks of inappropriate portrait use, malicious image tampering, and false information, motivating ethical guidelines and responsible use.