Source-linked AI summary
Exploring Disentangled Feature Representation Beyond Face Identification
Yu Liu, Fangyin Wei, Jing Shao, Lu Sheng, Junjie Yan, Xiaogang Wang
TL;DR
The paper addresses the difficulty of learning face representations that are both identity-discriminative and broadly reusable beyond predefined supervision. It proposes D2AE, which adversarially separates identity-distilled and identity-dispelled streams while reconstructing the complete face, and reports strong verification, attribute-recognition, and editing performance.
Problem
Existing face features are often specialized to predefined identity or attribute supervision, limiting their direct generalization to new face-analysis tasks.
Method
D2AE uses identity supervision and adversarial learning to produce complementary identity-distilled and identity-dispelled features that jointly reconstruct the input face.
Results
The learned features achieve state-of-the-art performance across face verification, attribute prediction, and face editing, including 99.80% accuracy on LFW.
Takeaways & Limitations
The representation supports face verification, attribute recognition, and semantic face generation or editing without attribute supervision or separately trained control units.
Takeaways & Limitations
Without the identification loss that constrains the identity classifier, adversarial training can let the dispelling branch produce non-informative outputs rather than identity-dispelled features.
Abstract
from arXiv · showhide
This paper proposes learning disentangled but complementary face features with minimal supervision by face identification. Specifically, we construct an identity Distilling and Dispelling Autoencoder (D2AE) framework that adversarially learns the identity-distilled features for identity verification and the identity-dispelled features to fool the verification system. Thanks to the design of two-stream cues, the learned disentangled features represent not only the identity or attribute but the complete input image. Comprehensive evaluations further demonstrate that the proposed features not only maintain state-of-the-art identity verification performance on LFW, but also acquire competitive discriminative power for face attribute recognition on CelebA and LFWA. Moreover, the proposed system is ready to semantically control the face generation/editing based on various identities and attributes in an unsupervised manner.
1. Introduction
The paper introduces D2AE, a face-feature framework using only identity supervision to learn complementary identity-distilled and identity-dispelled representations. These features support verification, attribute recognition, and controllable face generation or editing.
- Motivation: Existing face-feature methods are often tied to predefined identity or attribute supervision, limiting straightforward generalization to new tasks.The paper notes that such features may fail to capture relationships among attributes or identity-relevant information.
- Proposed framework: D2AE uses minimal face-identity supervision to learn identity-distilled features and complementary identity-dispelled features.The dispelled stream captures intra-personal variation and background clutter while being adversarially discouraged from identity-informative judgments.
- Proposed framework: The two-stream framework adaptively assembles identity-distilled and identity-dispelled features in an end-to-end representation learning scheme.The authors emphasize adversarial supervision, interpretability, and reuse of the learned features across face-related tasks.
- Results: 99.80% accuracy is achieved on the LFW face-verification benchmark using a single model.The figure caption reports that this outperforms most state-of-the-art methods without losing identity-related editing ability.
- Results: The framework is also evaluated on LFWA and CelebA attribute classification and on semantic face-editing tasks.The paper reports remarkable attribute-classification performance and superior capability for various generative tasks.
2. Related Work
Prior work spans discriminative, generative, adversarial, and disentangled representations, but often depends on task-specific or explicit attribute supervision. D2AE instead learns identity-invariant non-identity information using identity supervision alone.
- Learning Feature Representations: Discriminative models support facial identity and attribute classification, whereas generative models learn representations through the joint distribution of observed and target variables.The related-work discussion contrasts supervised flexibility with generative modeling of how data are produced.
- Combining Discriminative and Generative Models: GANs combine discriminative and generative modeling and have enabled progress in image translation, editing, inpainting, and texture synthesis.This places D2AE within a broader line of hybrid representation and generation methods.
- Disentangled Representation: Earlier disentanglement methods commonly require explicit attribute supervision and encode each attribute as a separate feature-vector element.Prior examples include pose-invariant recognition and identity-preserving image editing.
- Disentangled Representation: D2AE requires no attribute supervision and learns identity-dispelled features encoding varied attributes while identity-distilled features remain invariant to them.The paper contrasts this with methods restricted to pose invariance or dependent on multi-source supervision.
3. Learning Disentangled Face Features
D2AE learns complementary face representations by combining an identity-distilled branch with an identity-dispelled branch under adversarial supervision and reconstruction. The encoder-decoder objective encourages these branches to separate identity from complementary information while preserving a complete image representation.
- Framework: D2AE uses identity supervision to learn an identity-distilled feature fT and adversarial supervision to learn an identity-dispelled feature fP.The two branches are extracted in parallel after shared encoding layers.
- Identity Dispelling: The identity-dispelling branch trains an identity classifier while forcing its feature extractor to produce a uniform identity distribution.Classifier gradients from the adversarial loss update the classifier, whereas the identity-dispelling loss updates the branch and encoder with the classifier fixed.
- Identity Dispelling: Removing the identification loss allows the identity classifier to produce non-informative outputs, providing no guarantee that fP removes identity information.The identification loss constrains the classifier so adversarial training cannot be satisfied merely by changing the classifier.
- Complete Representation: Reconstruction loss forces fP to encode identity-irrelevant information needed to recover the original image alongside identity information encoded by fT.The decoder reconstructs the input from the fused features, addressing whether the non-identity information is retained.
- Statistical Augmentation: Gaussian feature perturbations encourage larger identity margins, channel decoupling, concentrated representations, and improved interpretability.Independent channel perturbations are applied to both feature types, with the perturbation scale set by each channel’s standard deviation.
- Learning Algorithm: The final training objective combines identity extraction, adversarial identity dispelling, reconstruction, and augmented reconstruction losses with weighted coefficients.The network parameters are updated by stochastic gradient descent through the autoencoder while identity classifiers are excluded from reconstruction updates.
4. Experimental Setting
The experiments train D2AE on checked MS-Celeb-1M images and evaluate generalization on LFW, CelebA, and LFWA. Evaluation covers identity verification, attribute recognition, branch and loss comparisons, and controllable face generation or editing.
- Datasets: D2AE is trained and validated on 4M checked MS-Celeb-1M images covering 80K identities after removing overlaps with LFW.The dataset is split using a 9:1 ratio.
- Model Architecture: The model uses an Inception-ResNet encoder, two parallel subnetworks producing 256-dimensional features, and a decoder reconstructing the input image.The encoding module uses a 235 × 235 input, and the two branches are denoted BθT and BθP.
- Applications: The shared application pipeline concatenates features from the identity-distilling and identity-dispelling branches for face-related tasks.The experiments assess the resulting representation across identity verification, attribute recognition, and generation or editing settings.
- Evaluation Protocols: Identity verification is evaluated on LFW using accuracy and TPR@0.001FPR, with cosine distance measuring similarity between feature vectors.Attribute recognition is evaluated on CelebA and LFWA over 40 annotated attributes using accuracy.
- Face Editing: Replacing one identity feature with another while preserving the complementary feature enables attribute-preserving identity exchange and smooth identity variation.The editing formulation also supports modifying attributes with perturbations constrained by learned uncertainty.
- Face Generation: For β ∈ [0, 1], generated images retain the target identity while preserving the remaining semantic information and background.This interpolation supports controlled identity changes within the learned representation.
5. Ablation Study
The ablation study shows that D2AE learns complementary branches: D2AE-T preserves identity information, while D2AE-P captures identity-irrelevant attributes and complementary image content. Adversarial supervision and statistical augmentation support this disentanglement and produce a compact, convex feature space.
- 5.1. Branch Selectivity: D2AE-T clusters identities distinctly and achieves nearly the same identity verification performance as the combined D2AE features.D2AE-P has extremely low verification TPR, indicating that it expels most identity-related information.
- 5.1. Branch Selectivity: D2AE-P responds more strongly to identity-irrelevant local attributes, whereas D2AE-T produces holistic responses for identity-related attributes such as gender and race.Residual maps show local responses for attributes such as pale skin and rosy cheeks, versus image-wide responses for gender and race.
- 5.1. Branch Selectivity: Modifying D2AE-T changes race while minimally affecting smiling, whereas modifying D2AE-P changes expression but hardly changes race.These editing results support distinct selectivity for identity-related and identity-irrelevant attributes.
- 5.2. Loss Functionality: Removing the identity-confusion loss leaves identity information in fP, improving identity verification but degrading attribute recognition across 36 of 40 attributes.The identity classification loss alone does not guarantee that fP becomes independent of identity.
- 5.2. Loss Functionality: The learned channels are nearly Gaussian, while dense interpolation produces a compact convex feature space with smooth changes along identity and attribute axes.Statistical augmentation encourages distinctive, densely Gaussian-clustered channel distributions.
6. Performance Comparison
D2AE is evaluated across identity verification, attribute recognition, and semantic face editing. It preserves competitive recognition performance while enabling smooth, identity-aware attribute manipulation and more realistic GAN-based reconstructions.
- 6.1. Face Identity Preserving: D2AE achieves comparable face verification performance to the baseline on MS-Celeb-1M and outperforms it when trained on the smaller WebFace dataset.The authors attribute the WebFace advantage to disentangling identity from attributes that may be biased toward particular identities in limited-scale data.
- 6.2. Face Attribute Recognition: D2AE achieves attribute-recognition results comparable with supervised methods on LFWA and CelebA despite using no attribute supervision.The D2AE embedding can be partitioned by either attributes or identities, although attribute category boundaries overlap.
- 6.3. Face Editing: D2AE edits local and global attributes while preserving naturalness and identity-related or identity-irrelevant information as appropriate.Examples include changing smiling, narrow eyes, bangs, race, and age while retaining other facial characteristics and background details.
- 6.3. Face Editing: D2AE supports smooth attribute interpolation and identity transfer, including gradual changes in expression, eye state, identity, and hairstyle.The results illustrate a compact, convex feature space.
- 6.3. Face Editing: Replacing reconstruction loss with an adversarial loss in a GAN extension produces reconstructed faces with more realistic details and noise.The extension is used for identity-aware attribute transfer.
7. Conclusion
D2AE disentangles face representations into two orthogonal streams with adversarial supervision while preserving complete face information. The learned features support verification, attribute prediction, and face editing.
- D2AE disentangles face representations into two orthogonal streams with novel adversarial supervision.
- The two streams completely represent information in the whole face and form a distinctive, densely distributed convex latent space.
- The learned features support face verification, attribute prediction, and face editing with state-of-the-art performance.
A. Training Details
The encoder uses an Inception-ResNet backbone with two 256-dimensional fully connected branches, while a convolutional decoder reconstructs 256 × 256 images.
- Encoder Module: The encoder uses Inception-ResNet and replaces average pooling with two 256-dimensional fully connected layers, BθT and BθP.The input size is modified to 235 × 235, and the model is initialized through face-classification pretraining.
- Decoder Module: The decoder contains 20 convolutional layers, 6 upsampling layers, and 1 deconvolutional layer.Its output images are 256 × 256 and are trained with reconstruction loss.
- Decoder Module: Table 4 presents the decoder's structural details.
B. Interpretation for Learned Gaussian Space
The learned D2AE latent space has approximately Gaussian, mutually independent channels. This structure is interpreted as an ellipsoid that permits sampling points within the space.
- All variables in the first 120 channels of D2AE-P and D2AE-T follow Gaussian distributions.The channel number and adjusted R-square score are reported above each distribution.
- More than 99.3% of absolute inter-channel correlation coefficients are below 0.3, indicating channel independence.
- The 512 dimensions in D2AE-P and D2AE-T are Gaussian and independent, so the learned latent space is an ellipsoid suitable for sampling.
C. More Examples Generated by D2AE
D2AE generates identity-aware attribute edits by changing target attributes while preserving irrelevant attributes. The edits remain consistent across different identities.
- D2AE generates additional faces with varied target attributes, including changes to Laugh.
- For each identity, the mouth changes from closed to increasingly open while irrelevant attributes remain preserved.Previously unseen teeth and tongue are generated with high fidelity.
- Across identities, attribute changes follow consistent directions such as upward mouth-corner movement and wider mouth opening.
- D2AE produces significant changes for target attributes while keeping those changes consistent across identities.
D.1. Attribute Classification Results
This section presents a complete ablation study for face attribute recognition and compares face-verification results on LFW.
- Table 5 reports a complete ablation study of face attribute recognition results.
- The attribute-classification analysis covers 40 face attributes identified by IDs 1 through 40.The supplied passage lists attributes including 5 O Clock Shadow, Arched Eyebrows, Attractive, Bags Under Eyes, Bald, and Bangs.
- Table 6 compares face-verification results on the LFW dataset.