Source-linked AI summary
HairCLIP: Design Your Hair by Text and Reference Image
Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Zhentao Tan, Lu Yuan, Weiming Zhang, Nenghai Yu
TL;DR
Sketch- and mask-based hair editing can demand tedious user interaction, and some hairstyles or colors are difficult to describe in text. HairCLIP unifies text and reference-image conditions in a CLIP-based framework with tailored structures and losses. The authors report superiority in manipulation accuracy, fidelity, realism, and irrelevant-attribute preservation.
Problem
Method
HairCLIP maps text and reference-image conditions through shared CLIP embeddings into one hair-editing framework with tailored structures and losses for disentangled manipulation.
Source: §1 · p. 2Results
Takeaways & Limitations
The framework lets users provide text and reference images individually or jointly, increasing interaction flexibility and reducing user interaction cost.
Source: §6 · p. 8Edits may fail for rare hairstyles or reference images outside StyleGAN's domain, and CLIP image embeddings can lose fine-grained hairstyle structure.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Hair editing is an interesting and challenging problem in computer vision and graphics. Many existing methods require well-drawn sketches or masks as conditional inputs for editing, however these interactions are neither straightforward nor efficient. In order to free users from the tedious interaction process, this paper proposes a new hair editing interaction mode, which enables manipulating hair attributes individually or jointly based on the texts or reference images provided by users. For this purpose, we encode the image and text conditions in a shared embedding space and propose a unified hair editing framework by leveraging the powerful image text representation capability of the Contrastive Language-Image Pre-Training (CLIP) model. With the carefully designed network structures and loss functions, our framework can perform high-quality hair editing in a disentangled manner. Extensive experiments demonstrate the superiority of our approach in terms of manipulation accuracy, visual realism of editing results, and irrelevant attribute preservation. Project repo is https://github.com/wty-ustc/HairCLIP.
1. Introduction
The paper addresses costly sketch- and mask-based hair editing with a framework that accepts text or reference images, individually or jointly. It uses CLIP-based conditions, tailored network structures, and losses for disentangled editing, with experiments reporting improvements in editing quality and attribute preservation.
1. Introduction
- Earlier methods often rely on sketches or masks, while StyleCLIP needs a separate mapper per description, has limited disentanglement, and does not support reference-image hair editing.Some hairstyles or colors are also difficult to describe in text, making reference images useful in those cases.Source: §1 · pp. 1–2
- Drawing a good hairstyle sketch can take several minutes, motivating a more intuitive interaction mode based on language.The paper frames text as a natural alternative to the interaction burden of sketch-based editing.Source: §1 · p. 1
- The framework supports text and reference-image hairstyle or color conditions within one model, individually or jointly.It maps conditions to latent-code changes using a pretrained StyleGAN generator and a mapper network.Source: §1 · p. 2
- Shared CLIP embeddings, separate hairstyle and hair-color sub-mappers, and conditional modulation are designed to support disentangled editing and direct condition control of latent codes.The framework also uses text, image-manipulation, and attribute-preservation losses to guide condition matching and preserve irrelevant attributes such as identity and background.Source: §1 · p. 2
- Qualitative and quantitative comparisons and a user study report superiority in manipulation accuracy, manipulation fidelity, and irrelevant-attribute preservation.The authors also report ablation analysis supporting their network and loss designs.Source: §1 · p. 2
2. Related Work
Related work includes controllable image manipulation with StyleGAN, hair editing through masks or sketches, and text-guided manipulation. The paper positions its framework as unifying text and image conditions while addressing limitations in existing hair-editing approaches.
2. Related Work
- StyleGAN synthesizes high-fidelity faces but is difficult to control; prior hair editing uses masks [26] [50] or sketches [40] [49], which remain costly interactions.MichiGAN [40] can fail on arbitrary shape changes, while LOHO [34] is time-consuming and may introduce background-blending artifacts.Source: §2 · p. 2
- StyleCLIP requires a mapper per hair description, while TediGAN-A [48] is less decoupled and TediGAN-B [48] is unstable and time-consuming.The paper states that no existing method is specifically tailored for hair editing.Source: §2 · p. 3
- The paper presents a unified framework for text and image conditions in one model and reports better hair-manipulation quality than existing works.The framework is intended to provide a more intuitive and convenient interaction mode with diverse conditions.Source: §2 · p. 3
3. Proposed Method
HairCLIP combines StyleGAN’s synthesis and latent spaces with CLIP’s shared image-text embeddings to edit hair from text or reference images. Its mapper routes hairstyle and color conditions to different latent levels, while tailored losses guide edits and preserve unrelated attributes.
3.1. Overview
- StyleGAN provides the pretrained generator and editable W+ latent code, while CLIP encodes text and images into a shared 512-dimensional embedding space.The pipeline inverts the input image to W+, predicts a latent change from the code and editing conditions, then sends the modified code to StyleGAN.Source: §3.2 · p. 3, §3.1 · p. 3
3.2. HairCLIP
- Text prompts and reference images supply hairstyle or color conditions, which HairCLIP encodes with CLIP and can flexibly switch within one mapper.With a missing hairstyle or color condition set to zero, the corresponding mapper modules act as identity functions, allowing either attribute or both to be edited.Source: §3.2 · pp. 3–4
- Hairstyle conditions drive the coarse and medium mappers, while hair-color conditions drive the fine mapper to match their respective StyleGAN semantic levels.The three sub-mappers predict latent changes for coarse, medium, and fine parts of W+; the figure illustrates hairstyle text with a color reference image.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
- A modulation module uses each condition embedding to modulate an intermediate fully connected output rather than concatenating the condition with the input latent code.When a condition is absent, its modulation modules become identity functions.Source: §3.2 · p. 4
3.3. Loss Functions
- Three loss types guide text- and reference-image-based manipulation while preserving identity, background, and other irrelevant attributes.Text manipulation uses CLIP image-text cosine distance; image manipulation compares hairstyle embeddings in CLIP space and uses hair-region color difference for reference-based color edits. Attribute preservation includes identity, hair-color preservation during hairstyle edits, background, and latent-step terms; no hairstyle-preservation loss is added for color-only edits because hairstyle was empirically found to remain well preserved.Source: §3.3 · pp. 4–5
4. Experiments
Experiments compare text- and reference-driven hair editing with prior methods, evaluate attribute preservation and user rankings, and ablate the proposed losses and network design.
4.1. Quantitative and Qualitative Comparison
- For text edits, TediGAN fails hairstyle tasks; StyleCLIP is stronger on hairstyle-only edits but overmanipulation harms realism, while HairCLIP improves joint edits and irrelevant-attribute retention.The comparison includes hairstyle, hair-color, and combined edits on inverted real images.Source: §4.1 · p. 6, Figure 3 · p. 6
- HairCLIP achieves the best irrelevant-attribute preservation across ten text descriptions, consistent with the visual comparison.The quantitative comparison reports averages for the ten descriptions.Source: Table 1 · p. 6, §4.1 · p. 6
- For reference transfer, LOHO and MichiGAN show boundary artifacts and struggle with pose-mismatched references; CLIP hair-region embeddings support plausible unaligned transfer.Figure 4 compares hairstyle and hair-color reference images with the input image.Source: §4.1 · p. 7, Figure 4 · p. 7
- HairCLIP outperforms competing methods in average user rankings for manipulation accuracy and visual realism across text-driven editing and hair transfer.The study used 20 participants; lower average rankings indicate better results.Source: Table 2 · p. 7, §4.1 · p. 7
4.2. Ablation Analysis
- Lbg, Lid, and Lnorm contribute to preserving unrelated attributes, while Lsmc keeps hair color unchanged during hairstyle-only editing.Network ablations also show that only the full model completes both hairstyle and hair-color manipulation; variants support the modulation module and disentangled semantic-matching injection.Source: §4.2 · p. 7, Table 3 · p. 7, Figure 6 · p. 8
- Shared condition embedding supports reasonable edits for unseen text descriptions, and the framework accepts text or image conditions individually or jointly.The paper reports extrapolation after training on a limited set of hair-editing descriptions.Source: §4.2 · p. 8
5. Limitations and Negative Impact
Editing is constrained by the coverage of pretrained StyleGAN and can lose fine hairstyle structure; the paper also reports interpolation and plausible results for unseen descriptions.
5. Limitations and Negative Impact
- Some rare hairstyle descriptions or reference images cannot be edited when they fall outside the pretrained StyleGAN domain.The authors say adding corresponding images to StyleGAN pretraining can address this boundary.Source: §5 · p. 8
- Increasing the blending parameter λ from 0 to 1 supports fine-grained hair edits, including color and hairstyle changes.Examples include yellow-to-pink hair and ringlets-to-shingle-bob hairstyle changes.Source: Figure 7 · p. 8
- The method produces plausible manipulations for unseen descriptions including curly short and mushroom hairstyles, and violet and silver hair.These descriptions were not used during training.Source: Figure 8 · p. 8
- CLIP hairstyle-reference embeddings can lose fine-grained structure, preventing perfect transfer of hairstyle details; edited images may also spread malicious information.The paper notes existing state-of-the-art GAN-image detectors as a possible way to evade that misuse.Source: §5 · p. 8
6. Conclusions
The paper presents a unified hair-editing interaction using text or reference images, individually or jointly. It reports improved manipulation capability, preservation of irrelevant attributes, and image realism compared with competing methods.
6. Conclusions
- The framework unifies text and image conditions, allowing users to edit hairstyle and hair color individually or jointly.The authors say this multimodal interaction increases flexibility and reduces user interaction cost.Source: §6 · p. 8
- CLIP, tailored network structures, and loss functions support high-quality hair editing in a decoupled manner.The conclusion describes this as the framework's approach to hair editing.Source: §6 · p. 8
- The method is reported to outperform competing methods in manipulation capability, irrelevant-attribute preservation, and image realism.The conclusion cites extensive qualitative and quantitative comparisons and a user study.Source: §6 · pp. 8–9
A. Quantitative Results
The authors caution that FID does not adequately evaluate hair-manipulation ability: TediGAN has the best FID results but performs poorly on hair editing in qualitative comparisons.
A. Quantitative Results
- TediGAN [48] performs best in FID [14], yet hardly performs any hair-editing task well in the qualitative results.The authors therefore argue that FID may not be suitable for evaluating manipulation ability.Source: A. Quantitative Results · p. 11
- The authors argue that FID [14] may not be a suitable metric for evaluating manipulation ability.Their stated reason is the mismatch between TediGAN's FID performance and its qualitative hair-editing results.Source: A. Quantitative Results · p. 11
- The authors note that e4e [43] drew a similar conclusion about FID's suitability for evaluating manipulation ability.The passage does not specify further details of that conclusion.Source: A. Quantitative Results · p. 11
B. More Qualitative Results
The qualitative comparisons show hair edits against StyleCLIP, TediGAN, LOHO, and MichiGAN, alongside examples of cross-modal inputs. The paper reports improved photorealism and irrelevant-attribute preservation against StyleCLIP and TediGAN, and plausible transfers in extreme cases against LOHO and MichiGAN.
B. More Qualitative Results
- Against StyleCLIP [31] and TediGAN [48], the method shows better photorealism and irrelevant-attribute preservation while performing the specified edits.The compared input images are inversions of real images.Source: Figure 9 · p. 12
- The method produces plausible hairstyle transfers against LOHO [34] and MichiGAN [40], including an extreme example.The passage identifies the third row as the extreme example.Source: Figure 10 · p. 13
- The figures include hairstyle and hair-color edits individually or jointly, with conditions supplied by text or reference images.Figure 11 identifies “Style” in text descriptions as an abbreviation for hairstyle.Source: B. More Qualitative Results · p. 11, Figure 11 · p. 14