PaperlayerView exact revision on arXiv

Source-linked AI summary

PixelSmile: Toward Fine-Grained Facial Expression Editing

Jiabin Hua, Hengyuan Xu, Aojie Li, Wei Cheng, Gang Yu, Xingjun Ma, Yu-Gang Jiang

arXiv:2603.25728v1cs.CVcs.AI

TL;DR

Fine-grained expression editing is difficult because semantically overlapping expressions are poorly represented by discrete labels. The paper introduces FFE, FFE-Bench, and PixelSmile’s symmetric diffusion framework, which together support disentangled, continuously controllable editing with strong benchmark performance and identity preservation.

  • Problem

    Semantically overlapping facial expressions confuse existing models, while discrete labels fail to capture continuous affective structure.

  • Method

    The paper constructs FFE and FFE-Bench, then uses PixelSmile with continuous intensity supervision, symmetric contrastive training, and textual latent interpolation.

  • Results

    PixelSmile achieves the best reported results across the benchmark dimensions, including CLS-6 0.8078, CLS-12 0.7305, and HES 0.4723.

  • Takeaways & Limitations

    The framework supports continuous, controllable, fine-grained expression editing while preserving identity fidelity.

  • Takeaways & Limitations

    The work is intended for noncommercial academic research and acknowledges dual-use risks in identity-related scenarios.

Abstract

from arXiv · show

Fine-grained facial expression editing has long been limited by intrinsic semantic overlap. To address this, we construct the Flex Facial Expression (FFE) dataset with continuous affective annotations and establish FFE-Bench to evaluate structural confusion, editing accuracy, linear controllability, and the trade-off between expression editing and identity preservation. We propose PixelSmile, a diffusion framework that disentangles expression semantics via fully symmetric joint training. PixelSmile combines intensity supervision with contrastive learning to produce stronger and more distinguishable expressions, achieving precise and stable linear expression control through textual latent interpolation. Extensive experiments demonstrate that PixelSmile achieves superior disentanglement and robust identity preservation, confirming its effectiveness for continuous, controllable, and fine-grained expression editing, while naturally supporting smooth expression blending.

1 Fudan University 2 StepFun

PixelSmile supports continuous facial-expression editing and expression blending across real-world and anime domains, with the paper noting author-contribution roles.

  • PixelSmile enables continuous, precise expression-intensity control across real-world and anime domains.
  • The framework supports editing across 12 distinct expression categories.
  • The paper identifies equal contributors, a project lead, and corresponding authors.

1. Introduction

The paper addresses the difficulty of editing semantically overlapping facial expressions by replacing discrete supervision with continuous affective annotations and introducing PixelSmile for disentangled, controllable editing.

  • Existing models struggle to distinguish highly correlated expressions such as fear versus surprise and anger versus disgust.
  • Continuous semantic overlap causes confusion among annotators, classifiers, and generative models, entangling latent representations.
  • FFE replaces one-hot labels with continuous 12-dimensional affective score distributions and supports systematic evaluation through FFE-Bench.
  • PixelSmile combines continuous supervision, fully symmetric joint training, and textual latent interpolation for precise, linearly controllable editing without reference images.
  • The paper contributes semantic-overlap analysis, the FFE dataset and FFE-Bench, and a diffusion framework for disentangled expression editing.

2. Related Work

Prior work includes general facial-expression editing, semantic-overlap analysis, and large-scale editing datasets, but the cited passages emphasize persistent limits in fine-grained, disentangled control.

  • Facial-expression editing has used conditional GANs, StyleGAN latent manipulation, and explicit facial priors such as Action Units or 3DMM parameters.
  • Semantic overlap produces systematic confusion across human annotators, recognition models, and generative models.
  • Foundation editing models provide zero-shot flexibility, while in-the-wild datasets improve generalization but lack paired same-identity expressions.
  • Video and multimodal datasets address temporal or cross-modal dynamics, whereas existing static-editing resources remain limited in continuous control and structured disentanglement.

3. Dataset and Benchmark

The paper introduces FFE and FFE-Bench to provide continuous, cross-domain, same-identity expression data and evaluate confusion, editing quality, controllability, and identity preservation.

  • Dataset: FFE provides large-scale same-identity expression variations with continuous affective annotations for analyzing disentanglement and controllability.
  • Dataset: The dataset contains 60,000 images across real and anime domains, generated through a collect–compose–generate–annotate pipeline.
  • Dataset: Expression prompts are filtered with a vision-language model to remove anatomically inconsistent or semantically conflicting descriptions.
  • Dataset: Multiple target expressions with varying intensities are synthesized for each base identity using global categories and localized facial attributes.
  • Dataset: Each image receives a continuous 12-dimensional score vector, with a human-verified subset supporting representation of semantic overlap.
  • Benchmark: FFE-Bench evaluates structural confusion, the editing–identity trade-off, control linearity, and expression editing accuracy.
  • Benchmark: mSCR averages bidirectional confusion rates over predefined confusing expression pairs; lower values indicate reduced cross-category confusion.
  • Benchmark: HES combines target expression strength with identity similarity, while CLS measures correlation between intensity coefficients and predicted expression scores.

4. Method

PixelSmile combines textual latent interpolation, score supervision, symmetric joint training, and identity preservation to enable continuous, disentangled facial-expression editing. Its design targets smooth intensity control, reduced confusion between similar expressions, and a wider editing range while preserving identity.

  • Evaluation: PixelSmile achieves a wider expression manipulation range while preserving identity fidelity, balancing ID similarity against expression score.The qualitative comparison likewise reports clearer expression changes without sacrificing facial identity.
  • PixelSmile builds on a pretrained MMDiT with LoRA adaptation and adds textual interpolation plus fully symmetric joint training.The two components target continuous intensity control, reduced cross-category confusion, and identity and background consistency.
  • Textual Latent Interpolation for Continuous Editing: Textual latent interpolation represents expression strength with α, where 0 is neutral, 1 is the target, and intermediate values produce smooth intensities.The same direction supports α > 1 extrapolation for stronger expression transfer while maintaining structural consistency.
  • Textual Latent Interpolation for Continuous Editing: Score-supervised flow matching couples the interpolation coefficient to the corresponding visual transformation, enabling continuous control without reference images.Training uses ground-truth intensity coefficients from continuous expression annotations, while inference varies α.
  • Fully Symmetric Joint Training for Disentanglement: Symmetric contrastive learning pulls generated samples toward target expressions and pushes them away from confusing expressions.The framework evaluates hinge-based, log-ratio, and InfoNCE-style objectives, primarily adopting InfoNCE for stable optimization.
  • Identity Preservation: An ArcFace-based identity preservation loss stabilizes biometric features when strong extrapolation or contrastive forces risk degrading identity consistency.The loss is designed to enforce identity consistency while allowing expression variation.

5. Experiment

PixelSmile is evaluated against general editing and linear control models using metrics for accuracy, disentanglement, controllability, and identity fidelity. Results show strong expression editing, stable linear control, identity preservation, and coherent but imperfect expression blending.

  • Experimental Setup: PixelSmile is compared with general editing models and linear control models, whose primary strengths are overall editing quality and continuous intensity control, respectively.
  • Ablation Study: Removing identity loss causes identity drift, while removing contrastive loss or symmetric training causes expression confusion and weak disentanglement.The ablations expose a trade-off between expression editing capability and identity preservation; the asymmetric variant reduces loss faster initially but converges worse.
  • Quantitative Evaluation: PixelSmile achieves the highest six-expression editing accuracy at 0.8627 and the lowest structural confusion rate at 0.0550 among general editing models.Nano Banana Pro reaches 0.8431 accuracy and 0.1754 structural confusion, while GPT-Image reaches 0.8039 and 0.1107, respectively.
  • Quantitative Evaluation: PixelSmile achieves the best linear-control results across CLS-6 0.8078, CLS-12 0.7305, and HES 0.4723.The evaluation covers linear intensity control and identity fidelity.
  • Quantitative Evaluation: Across a wide intensity range, PixelSmile reaches expression scores of ∼0.8 while maintaining identity similarity within 0.6–0.7 and producing monotonic responses.SliderEdit’s identity similarity drops to ∼0.4 near expression scores of 0.5, while KSlider fails to establish linear controllability.
  • User Study: In a study of 2,400 images, PixelSmile receives continuity and identity scores of 4.48 and 3.80, achieving the best balance among three methods.K-Slider scores 1.36 and 4.06, while SliderEdit scores 3.16 and 1.14.
  • Expression Blend: Nine of fifteen zero-shot expression interpolations form plausible compound expressions, although some combinations collapse or become unstable.Fear+Surprise may collapse into one dominant expression, while Angry+Happy can produce instability due to physiological conflicts.

6. Conclusion

The paper presents PixelSmile, FFE, and FFE-Bench to address semantic entanglement in fine-grained facial expression editing. It concludes that continuous supervision and symmetric joint training support precise, controllable, and compositional editing while preserving identity.

  • PixelSmile addresses semantic entanglement in facial expression editing through continuous expression-manifold supervision and symmetric joint training.
  • FFE and FFE-Bench provide continuous affective supervision and evaluation across structural confusion, expression accuracy, linear controllability, and identity preservation.
  • The work establishes a framework for continuous and compositional facial affect manipulation while advancing fine-grained expression editing.
  • The triplet constraint uses frozen, ℓ2-normalized CLIP features and cosine distances or similarities to represent expression semantics.The appendix describes hinge-based, log-ratio, and InfoNCE-style formulations; τ is the temperature parameter in the latter.

A.2. Implementation Details

The contrastive objective uses fixed formulation-specific hyperparameters and identical training schedules.

  • The InfoNCE-style formulation uses temperature τ = 0.07, the hinge-based variant uses margin m = 0.2, and the log-ratio variant uses ϵ = 10−6.All variants are evaluated under identical training schedules.

B. Details of Experiment

PixelSmile is trained with LoRA-based diffusion adaptation using specified optimization settings and loss weights.

  • LoRA Configuration: LoRA is applied to major diffusion-transformer attention and MLP components with rank 64, α = 128, and dropout 0.
  • Training Hyperparameters: Training runs for 100 epochs with AdamW, a 1e−4 learning rate, cosine scheduling, 500 warmup steps, and bf16 mixed precision.
  • Training Hyperparameters: The symmetric InfoNCE structural-confusion loss weight is λSC = 1.0, the identity-loss weight is λID = 0.1, and each GPU uses batch size 4.

C.1. Dataset Overview

The supplementary results evaluate PixelSmile across additional expression categories and domains, showing smooth intensity control, identity consistency, and expression blending through interpolation.

  • Dataset Overview: The MEAD ablation uses front-view video frames from seven discrete expressions, mapping low, medium, and high intensities to 0.5, 0.75, and 1.0.Frames are uniformly sampled and used to construct training triplets from the same subjects.
  • Dataset Overview: Sampled MEAD frames form triplets containing a source frame and two same-subject frames with distinct expressions for symmetric contrastive training.
  • Qualitative Results: Additional linear editing results cover ten remaining expressions across real and anime images, with intensity increasing from left to right.The top row contains real images and the bottom row anime images.
  • Qualitative Results: As the control parameter increases, expression intensity changes smoothly while facial identity remains consistent across the additional real and anime results.
  • Qualitative Results: Pairwise interpolation between basic expressions produces compositional facial-expression blends.These examples visualize smoothly blended emotional categories in PixelSmile's learned expression space.
  • Supplementary Materials: The supplementary material includes annotation prompts, expression-scoring prompts, and additional FFE statistics.Human and anime annotation templates use Qwen3-VL-235B-A22, while intensity scoring uses Gemini 3 Pro.

E.2. Dataset Statistics

FFE statistics reveal non-uniform demographic characteristics in the real-world subset and distinct stylistic and annotation patterns in the anime subset.

  • Dataset Statistics: The supplementary statistics cover categorical distributions and textual description patterns across the real-world and anime domains.Figures 13 and 14 provide the corresponding statistical visualizations.
  • Dataset Statistics: The real-world subset is diverse but imbalanced, with young adults comprising 53.5% and children, teens, and seniors forming smaller proportions.
  • Dataset Statistics: The real-world subset contains more female samples and light-to-medium skin tones, indicating non-uniform demographic characteristics.
  • Dataset Statistics: The anime subset has broader stylistic diversity, with CG and 2D anime each accounting for about 44%, alongside chibi, manga, and sketch styles.
  • Dataset Statistics: Compared with the real-world subset, anime images show a flatter age distribution but more unknown gender and age labels.The text attributes this ambiguity to the difficulty of applying real-world categorization schemes to stylized characters.
  • Dataset Statistics: Real-world descriptions emphasize clothing, hairstyles, and facial details, whereas anime descriptions are more stylized and visually distinctive.These domain-specific textual patterns require interpretation across heterogeneous distributions.
Loading 2603.25728v1…