Source-linked AI summary
Latent-Identity Tuning in Text-to-Image Personalization Models
Daniel Garibi, Ronen Kamenetsky, Hadar Averbuch-Elor, Daniel Cohen-Or, Or Patashnik
TL;DR
Existing personalization methods reproduce a person’s identity but offer limited control over fine-grained facial modifications that remain consistent across generations. This paper explores latent identity representations to identify localized editing directions, demonstrating subtle, semantically meaningful edits while preserving consistency across diverse generated images.
Problem
Existing personalization methods faithfully reproduce identity but provide limited control over fine-grained modifications that remain consistent across generated images.
Method
The framework analyzes structured identity tokens in a frozen personalization encoder, selecting region-relevant tokens and latent directions for localized, continuous identity manipulation without additional training.
Results
The framework produces fine-grained, semantically meaningful edits, including changes to eye openness, freckles, and facial-hair density, while maintaining consistency across generated images.
Takeaways & Limitations
The identity token space is interpretable and controllable at fine facial granularity, enabling users to refine delicate aspects of visual identity.
Takeaways & Limitations
Fine-grained control over specific facial parts, such as interpolating only the eyes, remains challenging because changes in layout or pose may appear across interpolation endpoints.
Abstract
from arXiv · showhide
Generating and editing a person's face demands high precision, as even minor modifications can significantly alter a subject's perceived identity. Current personalization and editing methods built on general-purpose text-to-image models, however, often lack the precision required for fine-grained facial edits. We present a method for fine-grained identity tuning in text-to-image personalization models. Unlike standard image editing, which operates on a given image, identity tuning modifies the latent representation of a specific identity, enabling the generation of diverse images that consistently depict the same edited identity. To enable fine-grained latent identity tuning, we explore the latent space of a pre-trained, frozen encoder for text-to-image personalization. Our approach requires no additional training. Instead, it leverages the existing architecture of a frozen encoder to uncover latent semantic directions. This space consists of a set of latent tokens that play distinct roles in capturing different aspects of an identity and often correspond to specific spatial or semantic facial regions. We show that meaningful directions can be identified within this space and within subspaces defined by selected tokens, enabling localized, fine-grained, and semantically coherent edits. We validate our approach through qualitative and quantitative experiments that demonstrate diverse localized facial edits while preserving cross-image identity consistency. Project page at: https://garibida.github.io/IdentityTuning/
1. Introduction
The paper introduces identity tuning, which edits a person’s latent identity representation so modified attributes persist across generated images. It analyzes latent tokens to identify localized semantic directions for fine-grained facial edits while preserving identity consistency.
- Motivation: Identity tuning modifies a pretrained personalization model’s latent identity representation rather than individual images, making edited attributes consistent across generated samples.This creates a persistent altered identity that can be deployed across varied prompts and settings.
- Latent-space analysis: Existing personalization methods capture identity with embeddings or latent tokens, but this internal representation has remained largely unexplored.The work shows that meaningful editing directions can be explicitly identified within the identity latent space.
- Latent-space analysis: Identity latent tokens represent distinct facial aspects, with some corresponding to specific spatial or semantic regions.The method investigates manipulating individual or jointly all identity tokens and identifying tokens relevant to a given location.
- Experiments: The framework enables fine-grained, semantically meaningful edits such as changing eye size or openness, adding a specific beard, and transferring eyebrows between identities.These edits are presented as difficult to achieve with alternative text-guided methods.
- Experiments: The approach is evaluated across identity-tuning scenarios and different personalization models, producing consistent identity representations across generated images.The supplied passage states that consistency is demonstrated alongside diverse localized facial edits.
2. Related Work
Related work spans text-to-image personalization, face personalization, image editing, and identity tuning. Prior methods enable personalized generation and semantic manipulation, but fine-grained identity control remains challenging, while existing identity-editing approaches may require per-image optimization.
- Text-to-Image Personalization: Text-to-image personalization generates new images of a concept from one or a few input images, guided by prompts specifying style, composition, and interactions.Early methods used per-subject optimization, while later methods trained encoders to map input images to concept embeddings.
- Face Personalization: Face personalization must capture fine-grained identity details because people are highly sensitive to facial differences and may reject generations that do not match their perceived appearance.The passage identifies personalized human-face generation as a central focus and highlights its unique challenges.
- Image Editing: Generative image-editing methods provide semantic manipulation and natural-language interfaces, but fine-grained, localized, continuous control remains challenging.GAN latent spaces enabled controlled manipulation of semantic attributes, while later methods pursued intuitive textual editing.
- Image Editing: Controlling identity-related attributes through textual interfaces remains a specific unresolved challenge in image editing.The related-work discussion isolates identity control as particularly difficult despite advances in natural-language editing and powerful manipulations.
- Identity Tuning: Identity editing has received limited attention; Weights2Weights uses LoRA weight space for inversion, identity sampling, and editing, but requires per-image optimization.The paper contrasts this requirement with its encoder-based approach, which operates without optimization.
3. Preliminaries: Q-Formers
Q-Formers are intermediate Transformer adapters that use learnable query tokens and cross-attention to convert source features into tokens for downstream generation. In face personalization, these outputs serve as identity tokens whose learned queries can correspond to distinct facial regions, suggesting a structured and partially disentangled latent space.
- Q-Former architecture: Q-Formers use a small set of learnable query tokens that attend via cross-attention to a source token sequence and produce downstream-adapted representations.They are used as intermediate Transformer modules between a source encoder and a downstream model.
- Identity representations: Text-to-image personalization extracts identity representation Z ∈ R^N×C, with N tokens of hidden dimension C, to condition diffusion-model generation.The representation is intended to capture a subject’s identity from one or more subject images.
- Identity representations: IP-Adapter maps pretrained image-encoder features F ∈ R^HW×d into tokens Z and injects them into the diffusion model through decoupled cross-attention layers.Q-Former-based adapters extend this framework for face personalization using a human-identity backbone encoder.
- Identity tokens: In human personalization, Q-Former outputs are called identity tokens, and different learned queries can attend to facial regions including the eyes, mouth, ears, and eyeglasses.These observations suggest a structured and partially disentangled latent space representing human facial semantics.
4. Identity Tuning
The section introduces identity tuning by manipulating individual, selected, or all identity tokens to discover and apply localized or global semantic edits. Token-selection methods and latent-space directions support region transfer, attribute control, and varying degrees of locality and precision.
- Token Selection: Identity representations can be edited globally, per token, or within selected token subsets, with dedicated selection methods identifying tokens relevant to target facial attributes.The representation is Z ∈ R^(N×C), and selected subsets are ZS ∈ R^(|S|×C).
- Token Selection: Localized attributes are selected by comparing identity representations of paired images with pasted facial regions, while global attributes use per-token linear SVMs and a validation-accuracy threshold τ.The selected token sets are computed once per facial region or attribute and reused across subjects and downstream edits.
- Region Transfer: Region transfer replaces target tokens for a selected facial part with blended source tokens, preserving the target identity elsewhere while gradually adopting the source trait within that region.The method is demonstrated for periocular features, including eyebrows and eyelids.
- Linear Directions: SVM-based latent directions provide finer edits such as rosy cheeks, whereas mean-difference directions better capture pronounced changes such as baldness or beards.SVM directions may rely on shortcut dimensions, while mean-difference directions are broader but less precise for subtle attributes.
- PCA Directions: Global PCA yields coherent but entangled edits, per-token PCA reveals localized interpretable effects but often lacks strength, and selected-token PCA enables smooth localized control of eye openness and lip color.The selected-token approach uses subsets identified for the relevant facial regions.
5. Experiments
Experiments show that identity tuning enables consistent, localized facial edits across diverse prompts, with token granularity tailored to the target attribute. Compared with baselines, it achieves stronger identity consistency and user preference while maintaining edit and prompt adherence.
- Implementation: The framework uses Omni-ID and PuLID encoders with Flux.dev, built on IP-Adapter, and derives identity-token directions from FFHQ and CelebA.FFHQ provides 70,000 face images for PCA, while CelebA provides 202,599 images with 40 binary attribute labels.
- Qualitative Results: Qualitative results show consistent identity across diverse prompts, including edits difficult to express through text, such as changing nose shape.The approach operates directly on the identity representation and supports iterative identity tuning.
- Ablation on Token Granularity: Single-token manipulation yields localized but minimal effects, global directions produce pronounced but less precise changes, and selected subsets provide fine-grained control.The experiments identify token-granularity selection as essential for precise, localized edits.
- Qualitative Comparisons: Compared with Precise Control and Weights2Weights, the method more reliably preserves identity across prompts, while W2W frequently struggles to follow input text.Against Flux Kontext, the method provides more precise local control and stronger overall consistency.
- Quantitative Comparisons: Quantitative comparisons report higher identity consistency, edit adherence, and prompt adherence than Precise Control and Weights2Weights; Flux Kontext Direct has comparable edit adherence but lower identity consistency.Flux Kontext Sequential approaches the method’s identity consistency but underperforms in edit adherence, particularly for advanced edits.
- User Study: Across 250 pairwise judgments, participants preferred the method over all baselines on Identity Preservation, Identity Consistency, Edit Adherence, Prompt Adherence, and Overall Preference.The study compared edited results generated with different prompts.
6. Conclusions · Appendix
The framework reveals a structured, spatially meaningful identity-token space that supports fine-grained, localized, continuous facial identity tuning. Exploring this space enables subtle edits while preserving consistency across generated images.
- 6. Conclusions: The method explores a Q-Former–based personalization encoder’s latent identity space for fine-grained, localized, continuous identity tuning.It requires identifying region-relevant tokens and editing directions within their subspaces.
- 6. Conclusions: Identity tokens form a structured, spatially meaningful space that captures distinct facial attributes.The structure supports region-aware manipulation through selected token subspaces.
- 6. Conclusions: The framework enables subtle, continuous adjustments to eye openness, freckle texture, and facial-hair density.These edits extend beyond prior coarse identity-preservation approaches.
- 6. Conclusions: Careful exploration of the identity space enables nuanced edits while maintaining consistency across generated images.The reported consistency applies across diverse generated images depicting the tuned identity.
- 6. Conclusions: The identity-token space is interpretable and controllable at fine facial granularity.This allows users to refine delicate aspects of visual identity.
- 6. Conclusions: The conclusions suggest future work on perceptually aligned metrics for subtle edits and interactive workflows for precise user-guided adjustments.These directions target evaluation and interaction for delicate identity refinements.
A. Additional Implementation Details
The implementation selects identity tokens for localized and global facial attributes, then derives unsupervised or supervised attribute directions from token embeddings. PCA is fit on FFHQ identity-token vectors, while linear SVM directions are trained on balanced CelebA labels.
- PCA over identity tokens: IncrementalPCA is fit on centered concatenations of selected Omni-ID token embeddings from the entire FFHQ dataset to identify salient components and roles.On the reported hardware, fitting takes about 10 minutes on a CPU.
- Supervised attribute directions via linear SVM: For each CelebA attribute and token set, balanced positive and negative examples train a linear hinge-loss classifier using scikit-learn SGDClassifier.Training takes about 10 minutes on a CPU in the reported runs and can be accelerated with a GPU.
B. Additional Experiments … B.4. Patch Editing
The additional experiments show that identity tuning supports localized, prompt-consistent facial edits, smooth control over edit strength, and coherent patch-based identity modifications. Attention maps further indicate that learned tokens focus on distinct semantic facial regions, supporting localized identity manipulation.
- B.1. Identity Tuning Resutls: Identity tuning produces consistent generations across diverse prompts and enables edits that are difficult to express textually, including changing nose shape.The method operates directly on the identity representation rather than editing a single image.
- B.1. Identity Tuning Resutls: Supervised directional editing remains consistent across varied text prompts while staying strictly localized to the target facial attribute.The examples demonstrate localized edits across different source identities and prompts.
- B.2. Continuous Tuning: Vector addition in identity space enables smooth, monotonic control of edit strength through a scale parameter across PCA, mean-difference, and SVM directions.Examples include eyebrow changes via PCA, bangs via attribute-aligned mean-difference, and freckles via SVM-based directions.
- B.2. Continuous Tuning: Interpolating between two identity embeddings changes identity gradually without abrupt jumps across varied scenes, demonstrating continuity in the learned identity representation.Intermediate embeddings are applied to all tokens to generate the gradual transitions.
- B.3. Q-Former Attention Maps: PuLID attention maps show that learned tokens consistently focus on specific semantic facial regions, including the cheeks, forehead, eyebrows, nose, and eyes.The observed patterns are similar to those reported for OmniID.
- B.4. Patch Editing: Patch editing encodes a pixel-manipulated image as a new identity representation, smoothly integrating the pasted patch while changing only the modified region.This operation provides local identity edits and fine-grained control over the edited appearance.
- B.4. Patch Editing: Localized patch modifications support consistent generation of the tuned identity across diverse contexts while preserving pasted attributes such as beard style.Figure 16 demonstrates edits involving eyes, lips, and a Van Dyke beard under multiple prompts.
B.5. Comparisons
Qualitative and user evaluations show that the proposed identity-tuning method applies facial edits consistently across diverse prompts while better preserving identity than competing approaches. Comparisons include varied identities, prompts, and basic and advanced edits.
- Qualitative Comparisons: Competing methods often perform the requested edit but fail to preserve identity across prompts, with Flux Kontext Direct showing drift between input and edited output.The drift is especially evident for large edits such as adding a beard; Flux Kontext Sequential improves identity coherence relative to Direct.
- Benchmark: The benchmark combines 20 FFHQ identity images, five basic edits, seven advanced edits, and 13 LLM-generated prompts.Applying all edits to each identity yields 100 basic and 140 advanced identity edits, with 1,300 basic and 1,820 advanced images rendered per method.
- Qualitative Comparisons: Our method consistently applies target attributes across diverse prompts while keeping the edited identity coherent.Figure 17 evaluates bangs and beard edits using four prompts per method for each edited identity.
- User Evaluation: Our method was preferred over all baselines across every evaluation category.Users judged it better at preserving the original identity, maintaining consistency across prompts, and adhering to the specified edit and target prompt.
B.6. Flux Kontext implementation details
The evaluation defines two Flux Kontext baselines for qualitative and quantitative analyses: a single-step direct edit and a two-step sequential edit. Both use the official checkpoint with default inference parameters and only the reference portrait as image input.
- Baselines: Two Flux Kontext baselines are defined for qualitative and quantitative analyses.The baselines are Kontext Direct and Kontext Sequential.
- Kontext Direct: Kontext Direct applies the local edit and renders the target scene in one pass from a reference image and scene prompt.Its template is “{EDIT} this person and generate an image of {BASE PROMPT}.”
- Kontext Sequential: Kontext Sequential separates editing and scene rendering into two steps, first editing the portrait in isolation and then conditioning on it to generate the scene.The template is “{EDIT} this person → Generate an image of this person {BASE PROMPT}.”
- Settings: Both baselines use the official checkpoint with default inference parameters, and the reference portrait is the only image input.These settings apply to the Flux Kontext implementation.
C. Limitations · D. Broader Impacts
The method is limited by Q-Former token capacity: too few queries entangle facial aspects and restrict isolated token-based edits, motivating adequate token counts in future models. Its fine-grained identity control also offers creative benefits while creating risks of misleading content and non-consensual likeness manipulation.
- C. Limitations: Localized editing performs best with enough Q-Former queries to enable natural disentanglement of facial regions.The authors identify token capacity as a limitation of the approach.
- C. Limitations: The quantitative evaluation and user study use the complete set of source prompts and edit instructions documented in Table 3.The table is identified as the source of prompts and instructions for both evaluation components.
- C. Limitations: Low token counts compress multiple facial aspects into single tokens, preventing isolated editing through single- or group-token manipulation.Semantic directions can still navigate the compressed space to edit specific attributes without affecting others, but direct token manipulation is constrained.
- C. Limitations: The findings motivate training future personalization models with adequate Q-Former token counts to preserve natural disentanglement and improve downstream editing controllability.This recommendation follows the observed relationship between token capacity and localized editing.
- D. Broader Impacts: The work advances understanding of identity representation in generative models and enables more precise, controllable editing for creative applications and personalized media.These benefits are presented alongside potential societal risks.
- D. Broader Impacts: Fine-grained facial-attribute control could be misused to generate misleading content or alter a person’s likeness without consent.The authors characterize these as societal risks similar to those of other identity-manipulation techniques.