PaperlayerView exact revision on arXiv

Source-linked AI summary

PixelSmile: Toward Fine-Grained Facial Expression Editing

Jiabin Hua, Hengyuan Xu, Aojie Li, Wei Cheng, Gang Yu, Xingjun Ma, Yu-Gang Jiang

arXiv:2603.25728v1cs.CVcs.AI

TL;DR

Fine-grained expression editing remains difficult for semantically overlapping expressions and discrete labels. PixelSmile combines continuous affective supervision with symmetric training and textual latent interpolation, and experiments report effectiveness in confusion, accuracy, controllability, and identity preservation.

  • Problem

    Fine-grained editing struggles to distinguish semantically overlapping expressions, while discrete categories impose rigid boundaries on inherently continuous expressions.

    Source: §1 · p. 2

  • Method

    The paper introduces FFE continuous affective annotations and FFE-Bench, then proposes PixelSmile, which uses symmetric joint training and textual latent interpolation to disentangle expression semantics.

    Source: §1 · p. 2

  • Results

    0.8627 editing accuracy on six basic expressions, versus 0.8431 for Nano Banana Pro and 0.8039 for GPT-Image; PixelSmile demonstrates effectiveness across structural confusion, expression accuracy, linear controllability, and identity preservation.

    Source: §5.2 · p. 9, §6 · p. 12

  • Takeaways & Limitations

    The work establishes a standardized framework for fine-grained facial expression editing and advances research toward continuous and compositional facial affect manipulation.

    Source: §6 · p. 12

    The authors identify potential misuse in identity-related scenarios and state that the work is intended for noncommercial academic research.

    Source: Ethics Statement · p. 13

Abstract

from arXiv · show

Fine-grained facial expression editing has long been limited by intrinsic semantic overlap. To address this, we construct the Flex Facial Expression (FFE) dataset with continuous affective annotations and establish FFE-Bench to evaluate structural confusion, editing accuracy, linear controllability, and the trade-off between expression editing and identity preservation. We propose PixelSmile, a diffusion framework that disentangles expression semantics via fully symmetric joint training. PixelSmile combines intensity supervision with contrastive learning to produce stronger and more distinguishable expressions, achieving precise and stable linear expression control through textual latent interpolation. Extensive experiments demonstrate that PixelSmile achieves superior disentanglement and robust identity preservation, confirming its effectiveness for continuous, controllable, and fine-grained expression editing, while naturally supporting smooth expression blending.

1. Introduction

The introduction identifies semantic overlap and discrete labels as barriers to fine-grained expression editing, then presents continuous-affect supervision and PixelSmile as an approach to disentangled, controllable editing with identity preservation.

Introduction

  • Semantically overlapping expressions entangle representations, causing cross-category confusion, unintended expression leakage, limited intensity control, and degraded identity consistency.The paper argues that structured semantic overlap—not purely classification error—is a primary cause of failures in recognition and generative editing.Source: §1 · p. 2
  • FFE replaces one-hot labels with continuous 12-dimensional affective distributions, while FFE-Bench evaluates confusion, editing accuracy, linear controllability, and identity-preservation trade-offs.FFE spans 12 expression categories across real and anime domains, with diverse expressions within identities.Source: §1 · p. 2
  • PixelSmile uses fully symmetric joint training to disentangle confusing expression pairs and flow-matching-based textual latent interpolation for linear intensity control without reference images.The framework combines continuous affective supervision and symmetric learning while preserving identity fidelity.Source: §1 · p. 2

2. Related Work

Prior expression editors and continuous-control methods leave gaps in fine-grained control, identity preservation, and evaluation; the paper positions FFE and FFE-Bench as supporting continuous, disentangled expression editing.

Facial Expression Editing

  • Earlier editors used conditional GANs, StyleGAN semantic directions [36] [37], and facial priors [16] [22] [33] [71], yet often lack fine control, identity consistency, and generalization.Diffusion models improved generation and editing quality [29] [49], while large-scale foundation models demonstrate zero-shot flexibility [46].Source: §2 · pp. 2–3

Continuously Controlled Generation

  • Continuous editors interpolate LoRA weights [20] or text embeddings and features [3] [7] [12] [21] [23] [26] [29] [35] [77] [85], but entangled latents cause ambiguity and identity drift at high strengths.FLUX.1 Kontext methods extend continuous control [81] [57] [72] [75] [79]; PixelSmile's disentangled formulation is positioned as enabling fine-grained linear control and identity preservation across strengths.Source: §2 · p. 3

Facial Expression Datasets and Benchmarks

  • Controlled datasets offer same-identity expressions but limited diversity, whereas in-the-wild datasets lack paired identities [2] [76]; FFE adds continuous affective annotations.F-Bench [44] and SEED [87], along with standard metrics such as CLIP, SSIM, and LPIPS, offer limited insight into disentanglement and continuous control; FFE-Bench targets those evaluation gaps.Source: §2 · p. 3

3. Dataset and Benchmark

FFE provides large-scale, same-identity expression variations with continuous affective annotations, while FFE-Bench evaluates expression editing across four complementary dimensions. Together they support systematic assessment of disentanglement and controllability.

3.1. The FFE Dataset

  • Its prompt library covers 12 expressions, decomposed into facial attributes and filtered for anatomical consistency and semantic conflicts.The taxonomy includes six basic emotions and six extended emotions; prompts specify attributes such as mouth shape, eyebrow movement, and eye openness.Source: §3.1 · p. 4
  • FFE contains 60,000 images—30,000 each from real and anime domains—with identity-preserving expression variations across diverse conditions.Base identities include approximately 6,000 real-world portraits and around 6,000 anime images retained from 207 productions covering 629 characters; face detection and manual verification support identity clarity and image quality.Source: §3.1 · p. 4
  • Each image receives a 12-dimensional continuous score vector, with human verification and quality checks supporting soft labels that capture semantic overlap.Gemini 3 Pro predicts expression intensities; consistency checks and manual spot verification remove ambiguous or low-confidence samples.Source: §3.1 · p. 4

3.2. The FFE-Bench Benchmark

  • FFE-Bench evaluates structural confusion, expression-editing accuracy, control linearity, and the trade-off between expression editing and identity preservation.All expression classifications and intensity scores are predicted by Gemini 3 Pro.Source: §3.2 · p. 5
  • FFE-Bench defines mSCR for confusion between predefined expression pairs and HES for jointly assessing expression strength and identity fidelity.Lower mSCR indicates less cross-category confusion; HES combines the target expression score with identity similarity averaged across ArcFace [14], AdaFace [38], and FaceNet.Source: §3.2 · p. 5
  • CLS measures Pearson correlation between inference intensity coefficients and predicted expression intensity, while Acc measures target-category matches.CLS uses uniformly spaced coefficients α ∈[0, αmax]; higher CLS indicates more linear, predictable control.Source: §3.2 · p. 5

4. Method

PixelSmile adapts a pretrained MMDiT [58] with LoRA [31], combining Flow-Matching-based textual interpolation [43] with fully symmetric joint training to support continuous expression control, reduce semantic confusion, and preserve identity.

4.1. Textual Latent Interpolation for Continuous Editing

  • Textual latent interpolation moves from neutral to target expression embeddings via α, with smooth intensity changes, α > 1 extrapolation, and inference control that needs no reference images.Score supervision pairs each training image’s continuous annotation-derived αgt with its conditioning embedding during LoRA fine-tuning, coupling interpolation strength to the visual transformation.Source: §4.1 · pp. 5–6

4.2. Fully Symmetric Joint Training for Disentanglement

  • Fully symmetric joint training generates both expressions in a confusing pair, reversing positive and hard-negative roles to reduce directional bias and separate overlapping categories.A frozen CLIP image encoder captures expression semantics; the symmetric contrastive loss pulls generations toward their targets and away from confusing expressions, primarily using InfoNCE for stable optimization.Source: §4.2 · p. 6

4.3. Identity Preservation

  • A frozen ArcFace [14] identity encoder supplies an identity-preservation loss that stabilizes biometric features during strong extrapolation or contrastive training while allowing expression variation.The loss compares generated images with their corresponding ground-truth images.Source: §4.3 · p. 6

4.4. Overall Training Objective

  • The overall objective fine-tunes frozen MMDiT LoRA parameters in symmetric dual-branch training, jointly optimizing confusing expression pairs for the same subject.Its λsc and λid weights control the trade-off between disentanglement and identity preservation while enforcing continuous intensity control, expression separation, and identity consistency.Source: §4.4 · p. 6

5. Experiment

Experiments show PixelSmile achieves strong expression accuracy and disentanglement with stable, identity-preserving control. Ablations, a human study, and expression blends further examine its components, perceived performance, and compositionality.

5.1. Experimental Setup

  • The evaluation compares general editing models on Acc-6, Acc-12, and mSCR, and linear-control models on CLS-6, CLS-12, and HES.Baselines are grouped by their primary strengths: general-purpose text-guided editing quality or continuous latent-space control; earlier ConceptSlider and AttributeControl are treated as reference baselines because they cover only two attributes and can lose information through inversion.Source: §5.1 · pp. 7, 9

5.2. Quantitative Evaluation

  • PixelSmile achieves Acc-6 0.8627, mSCR 0.0550, CLS-6 0.8078, CLS-12 0.7305, and HES 0.4723 across the reported evaluations.Its Acc-6 exceeds Nano Banana Pro (0.8431) and GPT-Image (0.8039), while its mSCR is below GPT-Image (0.1107) and Nano Banana Pro (0.1754). It is among the best on twelve extended expressions; the authors note that Gemini 3 Pro is less consistent on those categories. PixelSmile also maintains identity similarity within the reported natural 0.6–0.7 range while expression scores reach ∼0.8 across control intensities.Source: §5.2 · pp. 9–10

5.3. Qualitative Comparison

  • PixelSmile produces clear expression edits while preserving identity, and maintains monotonic control where comparison models show weak, unstable, or identity-degrading responses.This qualitative pattern holds for general editing models and for Happy, Surprised, and Disgust control examples; other methods’ linear response and identity preservation deteriorate further on Surprised than on Happy.Source: Figure 6 · p. 8, §5.3 · p. 11

5.4. Ablation Study

  • Ablations show that identity loss preserves facial identity while contrastive loss strengthens editing and disentanglement; the two losses play complementary roles.The symmetric framework converges more slowly initially than asymmetric training but yields better disentanglement; InfoNCE balances editing and identity better than Log-ratio or Hinge. Training on MEAD underperforms the full FFE-trained model, which the authors attribute to MEAD’s limited identity diversity and discrete intensity labels; FFE supplies richer identity variation and continuous soft labels.Source: Figure 7 · p. 9, Figure 8 · p. 10, §5.4 · pp. 11–12

5.5. User Study

  • In a study of 2,400 images rated by 10 trained annotators, PixelSmile scores highest for continuity (4.48) while maintaining strong identity consistency (3.80).K-Slider scores 1.36 for continuity and 4.06 for identity; SliderEdit scores 3.16 and 1.14, respectively. Human judgments align with the machine-based evaluation.Source: §5.5 · p. 12, Figure 10 · p. 12

5.6. Expression Blend

  • Of 15 zero-shot pairwise blends of six basic expressions, 9 produce plausible compound expressions, while others collapse to a dominant expression or become unstable.The reported examples include Fear+Surprise collapsing into one expression and Angry+Happy producing unstable results attributed to physiological conflicts.Source: §5.6 · p. 12

6. Conclusion … A.1. Triplet Constraint Formulations

PixelSmile addresses semantic entanglement in facial expression editing through continuous expression supervision and symmetric joint training, with experiments assessing structural confusion, expression accuracy, linear controllability, and identity preservation. The paper also describes FFE’s data provenance and responsible-use safeguards, and specifies triplet-constraint formulations for contrastive learning.

6. Conclusion

  • PixelSmile uses the FFE continuous expression manifold and symmetric joint training to enable precise, linearly controllable editing, with experiments across four dimensions: structural confusion, expression accuracy, linear controllability, and identity preservation.The paper presents FFE and FFE-Bench as a standardized framework for fine-grained editing and research toward continuous, compositional facial affect manipulation.Source: §6 · p. 12

Ethics Statement

  • FFE data comes from publicly available sources used under their licenses; its real-world subset includes Human Images Dataset and Matting Human Dataset, while its anime subset uses fictional characters from public media.The authors state that they use no private or login-restricted data and retain no personal metadata.Source: Ethics Statement · p. 13
  • The authors identify potential misuse in identity-related scenarios, limit the work to noncommercial academic expression manipulation, exclude offensive content, and encourage lawful and ethical use.They state that they do not aim to alter identity or enable deceptive applications.Source: Ethics Statement · p. 13

A.1. Triplet Constraint Formulations

  • Triplet-constraint features come from a frozen CLIP image encoder, are ℓ2-normalized, and use cosine distances and similarities between the generated, positive, and negative samples.The passage denotes the generated-positive and generated-negative distances as dG,P and dG,N, and their similarities as sG,P and sG,N.Source: §A.1 · p. 16
  • The appendix describes hinge-based margin, smooth log-ratio distance, and probabilistic InfoNCE-style triplet objectives, with a fixed margin, numerical-stability constant ϵ, and temperature τ, respectively.The supplied passages name these formulations and parameters but do not provide the equations.Source: §A.1 · p. 16

A.2. Implementation Details

The implementation specifies contrastive-loss hyperparameters and evaluates all variants under identical training schedules.

A.2. Implementation Details

  • InfoNCE uses temperature τ = 0.07, the hinge variant uses margin m = 0.2, and the log-ratio variant uses ϵ = 10−6 for numerical stability.All variants are evaluated under identical training schedules.Source: §A.2 · p. 16

B. Details of Experiment · C. Details of Dataset Ablation

PixelSmile was trained on four NVIDIA H200 GPUs with LoRA applied to major diffusion-transformer attention and MLP components. Training used AdamW with cosine learning-rate scheduling, mixed precision, and symmetric InfoNCE supervision.

B. Details of Experiment

  • Training used 4 NVIDIA H200 GPUs, with rank-64 LoRA (α = 128, dropout = 0) on major diffusion-transformer attention and MLP components.Source: B. Details of Experiment · p. 16
  • Models were optimized for 100 epochs using AdamW (β1 = 0.9, β2 = 0.999, weight decay = 0.001, ϵ = 1e−8) and a 1e−4 learning rate with cosine scheduling and 500 warmup steps.Source: B. Details of Experiment · p. 16
  • Training used bf16 mixed precision, symmetric InfoNCE with λSC = 1.0, λID = 0.1, and a per-GPU batch size of 4 with one gradient-accumulation step.Source: B. Details of Experiment · p. 16

C.1. Dataset Overview … D.1. Additional Linear Expression Editing Results

The appendix evaluates PixelSmile training with MEAD-derived triplets and adds qualitative demonstrations of linear editing and expression blending. Additional linear edits show smoothly changing intensity with consistent facial identity across real and anime domains.

C.1. Dataset Overview

  • MEAD supplies seven discrete expressions at low, medium, and high intensities; the ablation uses front-view data mapped to 0.5, 0.75, and 1.0.These continuous values match PixelSmile’s input range.Source: §C.1 · p. 16

C.2. Preprocessing and Triplet Construction

  • Uniformly sampled MEAD video frames become independent images for constructing source-and-expression triplets in the same manner as FFE.Each triplet contains a source frame and two frames of the same subject with distinct expressions.Source: §C.2 · p. 16
  • The MEAD triplets pair data from the same identities for symmetric contrastive training under the default configuration.Source: §C.2 · p. 16

D. Additional Qualitative Results

  • Additional qualitative examples cover linear editing across multiple expression categories and expression blending through interpolation in the learned expression space.Source: D. Additional Qualitative Results · p. 16

D.1. Additional Linear Expression Editing Results

  • Figure 11 shows smooth intensity changes with consistent facial identity across the remaining ten expressions in real and anime domains.The control parameter increases across the edits.Source: §D.1 · p. 16

D.2. Expression Blend Results · E. Additional Dataset Details · E.1. Annotation and Scoring Prompts

The supplementary sections show additional linear-expression and blending examples, then document FFE annotation and intensity-scoring prompts and their model bases.

D.2. Expression Blend Results

  • Figure 12 shows examples of expression blending generated by pairwise interpolation between basic expressions.The figure visualizes compositional facial expressions blended smoothly across multiple emotional categories.Source: §D.2 · p. 16, Figure 12 · p. 18
  • Figure 11 presents additional linear editing results for the remaining ten expressions in real and anime images, with intensity increasing left to right.Real images appear in the top row and anime images in the bottom row.Source: Figure 11 · p. 17

E. Additional Dataset Details

  • The supplementary dataset details cover annotation and scoring prompts alongside additional FFE dataset statistics.The supplied passage gives no specific statistics.Source: E. Additional Dataset Details · p. 18

E.1. Annotation and Scoring Prompts

  • Table 4 provides the prompt template for statistical annotation of the human subset, based on Qwen3-VL-235B-A22B.The template extracts structured semantic attributes for dataset analysis.Source: §E.1 · p. 18
  • Table 5 provides the prompt template for statistical annotation of the anime subset, also based on Qwen3-VL-235B-A22B.The template extracts structured semantic attributes for dataset analysis.Source: §E.1 · p. 18
  • Table 6 provides the prompt for assigning expression-intensity scores to images, based on Gemini 3 Pro.Source: §E.1 · p. 18

E.2. Dataset Statistics

FFE combines diverse real-world and anime data with demographic and annotation imbalances, while domain-specific appearance descriptions make expression editing require generalization across heterogeneous visual and semantic distributions.

E.2. Dataset Statistics

  • Real-world FFE is demographically imbalanced (53.5% young adults; female and light-to-medium skin-tone samples predominate), while anime has broader styles and flatter age distribution but more unknown labels.CG and 2D anime each account for about 44% of the anime subset.Source: §E.2 · p. 18, Figure 13 · p. 19
  • Real-world descriptions emphasize natural appearance cues, whereas anime descriptions are more stylized and visually distinctive, requiring generalization across domain-dependent semantic patterns.FFE’s expression-editing challenge therefore involves both visual transformation and semantic interpretation.Source: §E.2 · p. 18, Figure 14 · p. 20
  • FFE’s diversity alongside realistic demographic biases makes it a challenging, representative benchmark for fine-grained, diverse, real-world facial expression editing.Stylized anime characters also yield more unknown gender and age labels under real-world categorization, complicating consistent annotation and evaluation.Source: §E.2 · p. 18

Related papers

Loading 2603.25728v1…