PaperlayerView exact revision on arXiv

Source-linked AI summary

Talk-to-Edit: Fine-Grained Facial Editing via Dialog

Yuming Jiang, Ziqi Huang, Xingang Pan, Chen Change Loy, Ziwei Liu

arXiv:2109.04425v1cs.CV

TL;DR

Facial-editing systems have struggled to provide continuous, fine-grained manipulation with natural user interaction. Talk-to-Edit addresses this gap with dialog-driven editing over a location-specific semantic field and introduces CelebA-Dialog; experiments report smoother editing, stronger preservation, photorealism, and dialog quality, with about 80% user preference.

  • Problem

    Existing facial-editing methods do not provide continuous, fine-grained editing together with natural user interaction.

  • Method

    Talk-to-Edit uses language requests to guide curved, location-specific trajectories in a GAN semantic field and generates feedback for iterative dialog.

  • Results

    The framework achieves smoother changes, better identity and attribute preservation, visual photorealism, and dialog fluency, with about 80% user preference reported.

  • Takeaways & Limitations

    Talk-to-Edit provides a dialog-based framework and CelebA-Dialog dataset for fine-grained, language-driven facial editing.

  • Takeaways & Limitations

    The authors identify robust GAN inversion, stronger identity regularization, and more complex text requests as future improvements.

Abstract

from arXiv · show

Facial editing is an important task in vision and graphics with numerous applications. However, existing works are incapable to deliver a continuous and fine-grained editing mode (e.g., editing a slightly smiling face to a big laughing one) with natural interactions with users. In this work, we propose Talk-to-Edit, an interactive facial editing framework that performs fine-grained attribute manipulation through dialog between the user and the system. Our key insight is to model a continual "semantic field" in the GAN latent space. 1) Unlike previous works that regard the editing as traversing straight lines in the latent space, here the fine-grained editing is formulated as finding a curving trajectory that respects fine-grained attribute landscape on the semantic field. 2) The curvature at each step is location-specific and determined by the input image as well as the users' language requests. 3) To engage the users in a meaningful dialog, our system generates language feedback by considering both the user request and the current state of the semantic field. We also contribute CelebA-Dialog, a visual-language facial editing dataset to facilitate large-scale study. Specifically, each image has manually annotated fine-grained attribute annotations as well as template-based textual descriptions in natural language. Extensive quantitative and qualitative experiments demonstrate the superiority of our framework in terms of 1) the smoothness of fine-grained editing, 2) the identity/attribute preservation, and 3) the visual photorealism and dialog fluency. Notably, user study validates that our overall system is consistently favored by around 80% of the participants. Our project page is https://www.mmlab-ntu.com/project/talkedit/.

1. Introduction

Talk-to-Edit addresses the lack of natural, continuous, fine-grained facial editing by combining language-driven dialog with a location-specific semantic field in GAN latent space. It also introduces CelebA-Dialog to support this setting.

  • Motivation: Existing facial-editing methods often lack user interaction or constrain users to fixed controls and discrete states.The system feedback in prior methods may be limited to the edited image itself.
  • Talk-to-Edit: Talk-to-Edit performs facial editing round by round through user requests and system feedback in natural language.The interaction continues until the user is satisfied with the edited result.
  • Semantic Field: The semantic field models location-specific directions and magnitudes, enabling curved latent-space trajectories for fine-grained attribute changes.This accounts for nonlinear attribute transitions and identity-specific editing trajectories.
  • Dataset: CelebA-Dialog provides fine-grained facial-attribute labels together with user requests and captions describing the attributes.The dataset is designed to facilitate learning semantic-field and dialog-based editing.
  • Contributions: The framework reports smoother changes, better identity preservation, and superior results than other counterparts.These contributions are summarized alongside the CelebA-Dialog dataset contribution.

2. Related Work

Prior facial-editing research includes attribute-specific, image-translation, latent-space, and language-conditioned methods, but these approaches generally do not provide round-by-round dialog with meaningful feedback.

  • Semantic Facial Editing: Attribute-specific methods edit properties such as age, hair, or smiles, while image-to-image translation commonly maps faces between discrete states.The reviewed methods use different forms of facial priors or fixed translations.
  • Latent-Space Manipulation: Latent-space methods manipulate pretrained GAN codes along semantically meaningful directions discovered with supervised or unsupervised procedures.These methods focus on finding directions whose latent-code movements produce desired image-space edits.
  • CelebA-Dialog: CelebA-Dialog illustrates fine-grained smiling labels through example images, attribute degrees, textual descriptions, and their label distribution.The figure presents the dataset representation for the smiling attribute.
  • Language-Driven Editing: Natural-language systems can condition facial edits on text, but several do not provide meaningful feedback, clarification, suggestion, or interaction.Chatpainter uses completed dialogs but does not edit images with users round by round.

3. CelebA-Dialog Dataset

CelebA-Dialog is designed for continuous, fine-grained facial editing and dialog by pairing multi-degree attribute annotations with natural-language descriptions and user requests.

  • Motivation: Binary labels are insufficient for representing the intermediate attribute degrees needed during multi-round facial editing.Fine-grained labels also support precise system feedback and suggestions about edited attributes.
  • Dataset Properties: CelebA-Dialog annotates facial images with rich fine-grained labels that divide attributes into multiple semantic degrees.Each image is also paired with captions and a user-request sample.
  • Annotated Attributes: The dataset selects Bangs, Eyeglasses, Beard, Smiling, and Young from CelebA for fine-grained annotation.These attributes are selected because their degrees cannot be exhaustively expressed by binary labels.
  • Annotation Scheme: Smiling annotations use exposed-teeth and open-mouth ratios, while other attributes use attribute-specific visual criteria or age categories.Bangs use exposed-forehead proportions; eyeglasses use frame thickness and type; beard uses thickness; age uses six rough ranges.

4. Our Approach

Talk-to-Edit combines language-guided request understanding, a location-specific semantic field for continuous latent-space manipulation, and conversational feedback. Its training and editing objectives seek fine-grained attribute control while preserving identity and visual realism.

  • System Pipeline: Talk-to-Edit iteratively updates a latent code along semantic-field vectors, checks the target attribute degree with a pretrained predictor, and returns an edited image with language feedback.Feedback can clarify whether the degree is satisfactory, suggest alternatives, or request further instructions.
  • User Request Understanding: The system encodes a user request into an editing representation that specifies the request type, target attribute, editing direction, and degree of change.The encoded request and editing history determine whether and how the semantic field performs an edit in each dialog round.
  • Semantic Field for Facial Editing: The semantic field assigns location-specific directions in GAN latent space, producing curved trajectories that model nonlinear attribute transitions and vary across identities.The field is defined from an attribute-score gradient, whose direction is the fastest local increase of the corresponding attribute.
  • Semantic Field for Facial Editing: The method discretizes the continuous field in practice and repeatedly applies latent-code shifts until the desired attribute score is reached.The semantic-field vector is obtained from the mapping network, and the step size is set to α = 1 in this work.
  • Semantic Field for Facial Editing: The semantic field is trained with predictor, identity-keeping, and discriminator losses to change the target attribute while preserving unrelated attributes, identity, and realism.Identity preservation is supervised by comparing pretrained face-recognition features between the original and edited images.
  • Experiments: The qualitative comparison reports more realistic edits that are less likely to change identity and other attributes than InterfaceGAN, Multiclass SVM, and Enjoy Your Editing.The comparison is presented as a qualitative evaluation of realism and preservation.

5. Experiments

Experiments evaluate Talk-to-Edit on latent-code facial editing, dialog interaction, photorealism, preservation, high-resolution images, and trajectory curvature. The method achieves stronger preservation and produces smooth, realistic, dialog-driven edits across the reported settings.

  • Experimental setup: The evaluation compares Talk-to-Edit with InterfaceGAN, Multiclass SVM, and Enjoy Your Editing using identity and attribute preservation metrics.The evaluation uses latent codes directly for comparisons on 128 × 128 images to avoid GAN-inversion error.
  • Quantitative evaluation: Our method achieves the best identity and attribute preservation scores among the compared methods.Lower identity scores indicate better identity preservation, while lower attribute scores indicate less change to irrelevant attributes.
  • Qualitative evaluation: The user study reports that the majority of participants prefer our results for photorealism, editing smoothness, and identity preservation.The study includes 27 participants comparing 25 groups of images.
  • Dialog fluency: Over half of participants judge the system feedback to be natural and fluent.The dialog example shows sequential beard and smile editing driven by user requests.
  • Further analysis: Cosine similarity decreases as attribute class changes, indicating that editing directions change with location and produce curved trajectories.The analysis samples 100 latent codes while continuously adding eyeglasses to 1024 × 1024 images.
  • Further analysis: The method generalizes to 1024 × 1024 images and supports real-image editing after GAN inversion.The real-image example adds bangs and then smiling to the inverted image.

6. Conclusion

Talk-to-Edit performs dialog-driven, fine-grained facial editing by modeling nonlinear GAN latent-space changes with a semantic field. The paper reports continuous editing, identity preservation, and a dataset contribution, while identifying real-image editing as an area for improvement.

  • Conclusion: Talk-to-Edit uses users’ language requests and system feedback to support dialog-based fine-grained facial editing.The system models nonlinear latent-space changes with a semantic field to produce more continuous edits.
  • Conclusion: The semantic field enables curved editing trajectories that model nonlinear attribute changes in GAN latent space.Figure 10 reports decreasing cosine similarity as the attribute class changes.
  • Conclusion: CelebA-Dialog provides fine-grained facial attribute annotations and textual descriptions for language-driven editing research.The paper presents the dataset as a resource for fine-grained and language-driven facial editing tasks.
  • Conclusion: Future work targets real facial image editing with more robust GAN inversion and stronger identity-keeping regularization.The paper also proposes handling more complex text requests with stronger pretrained language models.

Supplementary

The supplementary material explains dataset annotations, implementation, experimental details, visual results, and failure cases.

  • Supplementary: The supplementary file covers CelebA-Dialog annotation definitions, implementation details, experiments, visual results, and failure cases.Its sections include dataset annotations, comparison-method details, evaluation information, additional visual results, and failure-case discussion.

A. CelebA-Dialog Dataset Annotations

CelebA-Dialog supplies fine-grained facial attribute annotations, textual descriptions, and example images for each annotated attribute.

  • Dataset annotations: Each CelebA-Dialog image includes fine-grained attribute annotations and textual descriptions.The dataset pairs each fine-grained label with an example image and its corresponding textual description.
  • Dataset annotations: The supplementary tables define and illustrate the bangs, eyeglasses, beard, smiling, and young attributes.Tables A2–A6 provide annotation definitions and example images for these five attributes.

B. Implementation Details

Talk-to-Edit encodes language requests, tracks dialog states, and iteratively edits facial attributes through a semantic-field pipeline. Its implementation also defines evaluation metrics for identity and attribute preservation.

  • Language Encoding: Language requests are encoded as request type, target attribute, editing direction, and change degree.The encoder uses word embeddings, a two-layer LSTM, and fully connected layers to produce the editing encoding.
  • Dialog Control: A finite-state machine tracks start, edit, no-edit, and end states throughout the dialog.The system can perform an edit, wait for instructions, or terminate the conversation based on the current state and user request.
  • Semantic-Field Training: Semantic-field training uses pretrained attribute prediction, face recognition, StyleGAN, and discriminator models.The attribute predictor is trained on CelebA-Dialog, while StyleGAN models are trained on CelebA and FFHQ at 128 × 128 and 1024 × 1024 resolutions.
  • System Feedback: Feedback checks attribute degree, suggests further edits, or requests new instructions after each editing round.Suggestions generally avoid attributes already edited, while degree checks address whether the current result satisfies the user.
  • System Feedback: Feedback language is generated by sampling templates and replacing phrases with predefined synonyms.The authors report that this simple design provides meaningful feedback to some extent.
  • Evaluation Metrics: Identity preservation uses FaceNet feature distance, while attribute preservation uses cross-entropy from a separately retrained predictor.The identity metric compares edited-image features with the original image, and the attribute metric evaluates target-class predictions.

C.3. Implementation Details on Comparison Methods

The comparison methods represent continuous editing with either fixed latent directions or identity-specific directions that remain unchanged during editing of one image.

  • InterfaceGAN: InterfaceGAN edits along a straight latent-space direction obtained from binary SVM boundaries.Its direction is fixed during editing, with directions switched according to the current attribute state.
  • Enjoy Your Editing: Enjoy Your Editing learns identity-specific directions for each initial latent code.The learned direction remains the same throughout editing for a given image.

D. More Qualitative Results

The appendix presents additional qualitative editing results at high resolution, on real images, and in comparisons across several facial attributes.

  • Qualitative Results: Additional results cover high-resolution editing, real-image editing, and qualitative comparisons with baselines.The attribute comparisons include bangs, beard, eyeglasses, smiling, and young.

E. Failure Cases Discussion

The failure cases show that editing can lose identity or introduce artifacts, especially for difficult synthetic edits and real-image inversion settings. These issues are linked to dataset bias, GAN mode collapse, latent-space outliers, and inversion gaps.

  • Synthetic Image Editing: Synthetic eyeglasses editing can cause identity loss, particularly for female images.The discussion attributes this to dataset bias and pretrained-GAN mode collapse, noting that females with eyeglasses are underrepresented in CelebA.
  • Synthetic Image Editing: Some eyeglasses edits produce artifacts when repeated latent updates move codes into outlier regions.The semantic field may not perfectly model the attribute’s non-linearity, especially when changing eyeglasses from degree 0 to degree 1.
  • Real Image Editing: Real-image editing is more prone to identity changes because GAN inversion can create a gap between the inverted code and the original latent space.The paper suggests that improved GAN-inversion techniques could better preserve latent-domain membership.
  • Additional Visual Evidence: The appendix includes a failure-case figure, high-resolution examples, real-image examples, and attribute-specific comparisons.The listed figures cover synthetic and real editing examples plus comparisons for bangs, beard, eyeglasses, smiling, and young.
Loading 2109.04425v1…