A Large-scale Evaluation of Text-guided Models for Facial Editing

Rahul Nair; Saurav Pandit; Hannah Kerner

Source-linked AI summary

A Large-scale Evaluation of Text-guided Models for Facial Editing

Rahul Nair, Saurav Pandit, Hannah Kerner

arXiv:2608.28802v1Aug 28, 2026cs.CVcs.AI

TL;DR

Granular facial editing by text-guided models had not been comprehensively evaluated, despite their potential to combine varied edits with diffusion-based consistency. The paper benchmarks six models across sequential facial edits using 169 curated attributes and two celebrity-face datasets. Most models handle hair and accessories relatively well but struggle with pose, overedit systematically, and show demographic disparities, while the evaluation’s overediting tracking covers only defined facial categories and uses binary skin-tone labels.

Problem
Text-guided models had not been comprehensively evaluated for granular facial editing and demographic bias, despite their potential for varied and consistent facial edits.
Method
The study evaluates six text-guided models on sequential edits using 169 Face-Edit-Attributes across hair, accessories, and pose, with facial similarity, semantic consistency, and overediting measures.
Results
Most models perform hair and accessory edits well but struggle with pose; all models overedit, averaging about 25 overedits per 100 instructions, with demographic disparities in overediting.
Takeaways & Limitations
Text-guided models can support facial editing applications off the shelf, but their systematic overediting and demographic disparities remain important concerns.
Takeaways & Limitations
The overediting tracker covers only the 35 facial editing categories in Face-Edit-Attributes, and binary skin-tone labels limit demographic analysis granularity.

Abstract

from arXiv · show

Facial appearance editing powers popular applications like FaceApp and Photoshop. Generative Adversarial Networks (GANs) and 3D Morphable Models (3DMMs) have been widely used for facial editing. GANs can perform varied facial edits (e.g., changing hair color, hairstyle), but often produce unstable edits. 3DMMs produce stable edits, but can only alter pose and facial expression. Recently, text-guided diffusion models like Nano Banana have become popular for image editing. Text-guided models are a compelling alternative to GANs and 3DMMs since they can produce both stable and varied image edits. While text-guided models have been widely tested for whole-scene edits (e.g., ``make the woman play a guitar''), they have not been comprehensively tested for facial editing. We conducted the first large-scale evaluation ($\sim1$M images evaluated) of six popular text-guided models on a sequential facial editing task. We present Face-Edit-Attributes, the largest collection of $169$ facial editing attributes focused on hair, accessories, and pose edits. We compared model performance using two popular celebrity face datasets: CelebA and CelebSET. Our results show that most models performed hair and accessory edits well, but struggled with editing pose. All models over-edit (e.g., changing hair color when asked only to change the hairstyle). We also evaluated demographic biases in each model. Our results show surprising biases in overediting: almost all models created more overedits for dark-skinned male faces and old faces. The code and data for our results (including our repository of $\sim 1$M images) can be accessed \href{https://github.com/rahul1801/Face-Edit-Bench}{\textcolor{blue}{here}}.

1 Introduction

Text-guided models offer varied, consistent facial edits, but their granular facial-editing ability and demographic fairness had not been comprehensively evaluated. This study introduces a large-scale benchmark and finds strong hair and accessory editing alongside pose difficulties, systematic overediting, and demographic disparities.

Motivation: Text-guided models offer more varied facial edits than 3DMMs while retaining diffusion-based consistency, positioning them as an alternative to GANs and 3DMMs.GANs support varied edits but can be unstable, whereas 3DMMs provide stable but mainly pose and expression edits.
Motivation: Real-world facial editing requires granular edits, demographic fairness, and avoiding unintended changes beyond the requested edit.Examples include specific hair or hat attributes, balanced performance across demographic groups, and changing hair color without changing hairstyle.
Research gap: Previous benchmarks evaluated whole-scene editing but did not comprehensively test granular facial editing or demographic bias across groups.Existing facial-editing benchmarks also contained only 15–35 useful editing attributes, while common image-editing benchmarks did not use face datasets.
Study design: The study evaluates six text-guided models using Face-Edit-Attributes, a manually curated collection of 169 attributes spanning hair, accessories, and pose or gaze.The evaluation uses sequential editing on CelebA and CelebSET, with separate hair, accessories, pose, and multi-axis test sets.
Evaluation: The evaluation uses facial similarity, semantic consistency, and the Overediting Matrix to measure identity preservation, requested-edit execution, and unintended additional edits.The Overediting Matrix extends overediting analysis beyond a single numerical score by identifying which extra edits occurred.
Results: Most models execute edits well and preserve over 70% of facial identity, but they struggle with pose edits and all models overedit.For every 100 instructions, corresponding to 25 sequences, models produced about 25 overedits; hair edits commonly triggered unintended hair-color changes.
Results: Demographic disparities appear in both identity or edit-execution performance and overediting, with more overedits for men—especially dark-skinned men—and older people.Only Qwen-Edit and Flux-Dev showed consistent gender and skin-tone bias across test sets, while several models better preserved young faces.

2 Evaluation Setup

The evaluation uses multi-turn facial editing sessions across celebrity face datasets, with hierarchical attributes, identity and alignment metrics, and explicit overediting analysis.

Evaluation protocol: Each edit session applies four sequential instructions, transforming an initial face image through four model-generated edits.The evaluation begins with I0 and iteratively produces I1 through I4.
Face datasets: The study uses CelebA and CelebSET celebrity face datasets, filtering images with facial-similarity thresholds to obtain 549 and 517 base images, respectively.CelebA and CelebSET also provide demographic information used for later bias analysis.
Editing attributes: Face-Edit-Attributes organizes 169 facial editing attributes hierarchically across hair, accessories, and pose axes.The hierarchy progresses from broad axes to sub-axes, categories, and fine-grained attributes such as specific hairstyles.
Editing attributes: The attribute design supports coarse and fine-grained test sets while providing the basis for measuring unintended edits.The evaluation excludes non-facial accessories, ambiguous categories, identity-modifying edits, facial expressions, and makeup.
Evaluation metrics: DropFSS measures average percentage identity-similarity loss across an edit session, with lower values indicating better identity preservation.The study also uses semantic consistency, whose strict score is the minimum of editing success and overediting-degree scores.
Evaluation metrics: The Overediting Matrix records intended categories by rows and unintended edited categories by columns, quantifying which additional edits models perform.For example, [red hair][stubble] counts cases where a red-hair request also produced stubble.
Models and scale: The experiments evaluate five models on CelebA and six across both datasets, covering about 200K sessions and approximately 1M images.SeedEdit 3.0 was evaluated only on CelebSET because of cost constraints; Gemini-2.5-Flash was selected as the state tracker after the highest human-annotator alignment.

3 Results

Across CelebA and CelebSET, models generally executed hair and accessory edits better than pose edits, while identity preservation and overediting varied substantially by model and demographic group.

Identity preservation: 22–32%: facial similarity typically dropped within this range across four edits, with hair edits best preserving identity and pose edits worst.Across CelebA, average DropFSS was 28.7–32.7% for most models; across CelebSET, it was 22.3–30.8% for four of six models.
Editing alignment: Nano-Banana and SeedEdit generally executed edits best, while Flux-Dev performed worst; Nano-Banana reached average SC values of 79.6 on CelebA and 77 on CelebSET.SeedEdit also reached 77 on CelebSET.
Overediting: Models frequently overedited hair attributes, including changing mutton chops to a handlebar mustache or bob cuts to straight hair.These patterns were especially prevalent in Bagel-Edit, Kontext-Pro, and Qwen-Edit.
Overediting: Nano-Banana and SeedEdit produced the fewest overedits, with average OER values of 13.8 on CelebA and 17.49 on CelebSET, respectively.Accessory and pose edits generally had the lowest OER, while multi-axis edits had the highest.
Demographic comparisons: Demographic disparities were strongest for overediting: dark-skinned males were consistently the worst group, and old faces usually had worse OER.Identity-preservation and editing-alignment biases varied by model; Qwen-Edit struggled with female identity preservation, while Flux-Dev struggled with darker skin tones.

4 Discussion

The discussion identifies Nano-Banana as the strongest overall model and Flux-Dev as the weakest, while emphasizing that reliable performance is limited to short sequences and that overediting remains widespread.

Discussion: Nano-Banana was the best overall model, whereas Flux-Dev was the worst and often produced cartoonish edits that lowered its evaluation scores.Flux-Dev was optimized for image generation rather than image editing.
Discussion: 3–4 edits: models were considered reliable for short facial editing sequences, but performance consistently dropped with each additional edit.The discussion does not recommend using these models for longer sequences.
Discussion: About 25 overedits per 100 instructions: models commonly changed attributes that were not requested, likely reflecting spurious correlations learned during training.The discussion gives bob cuts changing to straight hair as an example and suggests more diverse training data as one mitigation.

5 Limitations

The evaluation has three important limitations: overedit tracking is restricted to 35 predefined facial categories, skin-tone analysis uses binary labels, and the state tracker sometimes misidentifies edits.

Overedit tracking covers only the 35 facial editing categories in Face-Edit-Attributes, excluding cases such as toothbrush artifacts from a “toothbrush mustache” prompt.The authors identify concept activation vectors as a future step toward tracking broader overedit types.
Skin-tone bias analysis uses only binary light-versus-dark labels because CelebSET lacks finer-grained annotations.Fitzpatrick scores could distinguish dark-skinned from very dark-skinned groups, such as scores 4−5 versus 5−6.
The Gemini-2.5-Flash state tracker reliably tracks most edits but performs poorly for some cases, including looking-left edits.

6 Conclusion

The paper presents a large benchmark for facial editing with diverse attributes, sequential instructions, and dedicated evaluation of overediting and demographic bias. Its results motivate using some models off-the-shelf while prioritizing mitigation of overediting, especially for dark-skinned and older males.

6 Conclusion: The study evaluates six text-guided models for facial appearance editing and introduces Face-Edit-Attributes, a collection of 169 hair, accessory, and pose attributes.It also releases a repository of approximately 1M edited images.
6 Conclusion: All evaluated models overedit, motivating future work to mitigate overediting, particularly for dark-skinned and older males.
6 Conclusion: The benchmark organizes attribute inventories and editing sessions across hair, accessories, pose, and multi-axis test sets.
6 Conclusion: The released instruction examples include granular changes to hairstyles, hair colors, facial hair, hats, glasses, and other accessories.
6 Conclusion: The editing pipeline generates state-aware instructions using KEEP, CHANGE, ADD, REMOVE, and MAKE verb categories.Instructions use the current image state to select an appropriate operation and preserve the exact edit candidate phrase.

E Human Experiment for State Tracking

The human state-tracking experiment compares model-generated attribute dictionaries with human annotations on 300 images. Gemini-2.5-Flash was selected because it achieved the strongest overall alignment, although some attribute categories remained difficult.

E Human Experiment for State Tracking: 300 images were annotated across 12 CelebSET base images, five SeedEdit sessions per image, and four edits per session.
E Human Experiment for State Tracking: Four human annotators labeled visible categories across eight sub-axes, using majority voting to create reference state dictionaries.
E Human Experiment for State Tracking: State-dictionary accuracy requires every category within a sub-axis to match the human reference.
E Human Experiment for State Tracking: 79.65% accuracy made Gemini-2.5-Flash the best-aligned state tracker, ahead of Gemini-2.5-Pro at 79.26%, GPT-4o at 70.31%, and CLIP at 35.47%.
E Human Experiment for State Tracking: Gemini-2.5-Flash showed over 85% human alignment in most cases but aligned on only 2 of 96 human “none” labels for hair type.The discrepancy reflected stricter human labeling versus the model’s preference to assign a hair type.
E Human Experiment for State Tracking: Figure 4 compares human labels with Gemini-2.5-Flash predictions using confusion matrices, with human labels in rows and model predictions in columns.
E Human Experiment for State Tracking: The evaluation reports editing-session counts by gender, age, and skin tone across CelebA and CelebSET.
E Human Experiment for State Tracking: WD > 3 was selected as the threshold for prominent bias because almost all values above it were statistically significant.

H Age Bias

Age-related effects differ by evaluation axis: older faces generally show poorer identity preservation and more overediting, while editing-alignment bias is limited and less consistent across datasets.

H Age Bias: Most models showed poor identity preservation on older faces, with several CelebSET cases exceeding the WD_age > 3 bias threshold.Except for Qwen-Edit, older faces were the worst-performing group in these cases; CelebA showed fewer age-bias instances.
H Age Bias: Most models showed minimal or no age bias in editing alignment across both datasets.Flux-Dev showed age bias across three CelebA test sets, but not the same trend in CelebSET.
H Age Bias: Several models produced more overedits on older faces, and age bias appeared in all models for most test sets across CelebA and CelebSET.Older faces were the higher-OER group in almost all of these cases.

I All mean values

The evaluation reports subgroup means for identity preservation, alignment, and overediting across gender, intersectional gender–skin-tone, and age groups. Across five attribute tables, hair-based edits produced the highest demographic overediting biases, especially for men, dark-skinned faces, and older faces.

Evaluation setup: Mean values with 95% confidence intervals were reported across CelebA and CelebSET demographic groups for identity preservation, alignment, and overediting.Identity preservation used gender, intersectional gender–skin-tone, and age groups; alignment and overediting used the same subgroup structure.
Bias analysis: WD was computed from OER for all 40 unique editing attributes across models and each dataset–bias-axis combination.The 40 attributes came from hair, accessories, pose, and multi-axis evaluation sets.
Demographic bias findings: Across five tables, hair-based editing operations caused the highest demographic biases in overediting for almost all models.The implicated edits include hairstyles, facial hair, and hair color.
Demographic bias findings: The overediting results were consistent with greater overediting on men, dark-skinned faces, and old faces.This pattern was reported across the five attribute tables.

K FSS: Demographic Bias Analysis on Base Images

The study measured demographic bias in base-image FSS using subgroup means, confidence intervals, and Wasserstein distance comparisons.

FSS demographic bias: Average FSS was computed for each demographic group in the base images to assess demographic bias before editing.Bias between groups was measured with the WD metric.
FSS demographic bias: CelebA comparisons covered gender and age, while CelebSET comparisons covered gender–skin-tone intersections and age.The comparisons were reported in Tables 29a–29d.
FSS demographic bias: Each demographic comparison reported mean FSS values with 95% confidence intervals and used WD to measure disparity.The same reporting structure was applied across the listed datasets and subgroup axes.

L FSS: Occlusion Sensitivity Analysis on Base Images

The occlusion analysis tested whether facial accessories affect base-image FSS. FSS remained similar for images with and without accessories, indicating no detected sensitivity to these occlusions.

Occlusion groups: CelebA was used because it annotates facial accessories, producing groups with accessories and without accessories.The accessory group contained hats, glasses, or both, with 34 images; the no-accessory group contained 515 images.
Occlusion results: Mean FSS was 96.77 ± 0.37 with accessories and 96.82 ± 0.10 without accessories.The reported group means were nearly identical.
Occlusion results: WD was 0.14 with p = 0.9, so the disparity was not statistically significant.The authors conclude that FSS is not sensitive to occlusions from glasses or hats in the base images.

M Stereotype Analysis

The stereotype analysis tested whether male-centric facial-hair instructions produced better edits on male faces. The study found no clear overall evidence of that pattern, although some model-specific subgroup disparities appeared.

Stereotype test design: The analysis selected 17 male-centric instructions from 140 evaluation instructions, all focused on facial-hair edits.Examples included adding a pencil mustache or bushy beard.
Stereotype test design: Mean alignment scores were computed across models for male and female CelebA groups and four intersectional CelebSET groups.The CelebSET groups were male-light, female-light, male-dark, and female-dark.
Stereotype findings: CelebA showed gender bias in three models, but only Kontext-Pro and Bagel-Edit had worse alignment on females.In CelebSET, only Flux-Dev had females in both of the two worst groups; other models included at least one male intersectional group.
Stereotype findings: The study found no clear trend that models perform male-centric edits better for males.This conclusion applies to the selected male-centric instruction subset.

N Face ID Progression Analysis

The analysis tracks facial similarity through four sequential edits using FSS on CelebA and CelebSET. Similarity declines after every edit, with the largest drop occurring at the first edit.

Analysis setup: FSS is plotted from edit 0, the base image, through edit 4, the final edited image, using 95% confidence intervals.The analysis uses average facial similarity across all images at each editing step.
Analysis setup: Figure 18 reports aggregated FSS for CelebA and CelebSET across hair, accessories, pose, and multi-axis test sets.CelebA results appear in the left plot and CelebSET results in the right plot.
Findings: FSS consistently drops after every edit for all models on both datasets.The largest decline occurs from edit 0 to edit 1, followed by smaller but consistent decreases in later edits.