Source-linked AI summary
Semi-Adversarial Networks: Convolutional Autoencoders for Imparting Privacy to Face Images
Vahid Mirjalili, Sebastian Raschka, Anoop Namboodiri, Arun Ross
TL;DR
Face images can reveal sensitive attributes beyond identity, while de-identification can compromise biometric utility. This paper trains a convolutional autoencoder to confound gender classification while retaining face matching, and reports gender privacy with matching accuracy not unduly impacted.
Problem
Face images can reveal attributes such as gender beyond identity, motivating privacy protection that suppresses such information without unduly affecting biometric recognition.
Source: §1 · p. 1, §2.1 · p. 2Method
A convolutional autoencoder is trained with auxiliary gender-classifier and face-matcher feedback and losses for image realism, gender confounding, and matching utility.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · p. 3Results
Experiments using two independent gender classifiers and a state-of-the-art commercial face matcher unseen during training report gender privacy without unduly impacting face-matching accuracy.
Source: §3.1.1 · p. 6, §3.2 · p. 7, §4 · p. 8Takeaways & Limitations
The authors present semi-adversarial learning as a way to perturb faces for gender privacy while retaining recognition utility, with a design intended to generalize to unseen classifiers and datasets.
Source: §1 · p. 1, §1.1 · p. 2
Abstract
from arXiv · showhide
In this paper, we design and evaluate a convolutional autoencoder that perturbs an input face image to impart privacy to a subject. Specifically, the proposed autoencoder transforms an input face image such that the transformed image can be successfully used for face recognition but not for gender classification. In order to train this autoencoder, we propose a novel training scheme, referred to as semi-adversarial training in this work. The training is facilitated by attaching a semi-adversarial module consisting of a pseudo gender classifier and a pseudo face matcher to the autoencoder. The objective function utilized for training this network has three terms: one to ensure that the perturbed image is a realistic face image; another to ensure that the gender attributes of the face are confounded; and a third to ensure that biometric recognition performance due to the perturbed image is not impacted. Extensive experiments confirm the efficacy of the proposed architecture in extending gender privacy to face images.
1. Introduction
Face images can reveal sensitive attributes beyond identity, creating a privacy challenge: suppressing those attributes must not sacrifice biometric recognition. The paper proposes a semi-adversarial autoencoder and frames it as a way to confound one classifier while preserving another.
1. Introduction
- Privacy protection must balance suppressing attributes such as gender with retaining biometric utility, since de-identification can compromise recognition.Automated attribute extraction can also raise consent concerns and enable gender- or age-based profiling.Source: §1 · p. 1
- The proposed convolutional autoencoder perturbs faces to impede gender classification while preserving their use for face matching.The paper also describes the design as applicable to other attributes and problem domains where some classifiers should be confounded while others retain utility.Source: §1 · p. 1
1.1. Related work
- Unlike prior perturbations tied to a specific gender classifier, the proposed approach targets arbitrary classifiers and is designed to transfer across datasets.The authors identify an explicit matching-accuracy term and generalization to previously unseen datasets as contributions.Source: §1.1 · p. 2
- The authors report that semi-adversarial learning generalizes to classifiers not used or available during training while optimizing one classifier's performance against another's.They describe this as the first use of adversarial training to design such a generator component.Source: §1.1 · p. 2
2.1. Problem formulation
The problem is to perturb a face image so it remains visually similar and useful for biometric verification while an arbitrary gender classifier is more likely to misclassify it. The objective combines image similarity, gender-classification loss, and matching loss.
2.1. Problem formulation
- The target output should resemble the input, confound an arbitrary gender classifier, and preserve verification match scores against unperturbed images of the same subject.The formulation describes the desired properties of the perturbed image.Source: §2.1 · p. 2
- The objective combines dissimilarity, gender-prediction, and matching losses to produce realistic-looking faces, reduce gender-classifier accuracy, and avoid substantially diminishing matching accuracy.The three terms are weighted in the stated objective function.Source: §2.1 · p. 2
- The authors optimize these competing goals with a semi-adversarial convolutional autoencoder.Source: §2.1 · p. 2
2.2. Semi-adversarial network architecture
The semi-adversarial network uses a convolutional autoencoder with auxiliary gender-classification and face-matching feedback to perturb faces. Prototype images guide reconstruction, while the auxiliary classifiers are used during training to suppress gender information without unduly affecting matching.
2.2. Semi-adversarial network architecture
- The autoencoder is trained with feedback from two auxiliary subnetworks: a gender classifier whose error is increased and a face matcher whose error is not unduly affected.Both auxiliary components are detachable and used only during training.Source: §2.2 · pp. 2–3
2.2.1 Convolutional autoencoder
- The autoencoder encodes a grayscale face with a same-gender prototype, then combines decoder features with a same-, neutral-, or opposite-gender prototype to produce a perturbed image.The decoder produces a 224 × 224 × 128 feature map before prototype combination and final convolution.Source: §2.2.1 · p. 3, Figure 2 · p. 3
2.2.2 Gender prototypes
- Same-gender prototypes are concatenated with the input before encoding, while same-, neutral-, and opposite-gender prototypes are added after decoding; neutral-prototype outputs are reserved for evaluation.The male and female prototypes are averages of CelebA training-set images, and the neutral prototype is their weighted mean.Source: §2.2.2 · p. 4
2.2.3 Auxiliary CNN-based gender classifier
- The auxiliary gender classifier is independently trained on CelebA-train and reaches 96.14% accuracy on CelebA-test; it is used to train the autoencoder, not to evaluate it.Its architecture has six convolutional layers and two fully connected layers.Source: §2.2.3 · p. 4, Figure 4 · p. 4
2.2.4 Auxiliary CNN-based face matcher
- The auxiliary matcher uses a VGG-face network to produce 2622-dimensional face descriptors for training feedback, while evaluation uses commercial matching software.The VGG-face weights were used without further performance tuning, and its first-layer filters were modified for grayscale inputs.Source: §2.2.4 · p. 5
2.3. Loss function
The loss function combines image reconstruction, gender-attribute control, and face-matching similarity to train the autoencoder. Same-gender outputs retain correct gender predictions, while opposite-gender outputs use flipped labels to confound the classifier.
2.3. Loss function
- The reconstruction loss is pixelwise cross-entropy between input and perturbed images, encouraging the autoencoder to produce outputs similar to its inputs.This loss is used during pre-training before the auxiliary classifier and matcher are connected.Source: §2.3 · p. 5
- The gender loss uses ground-truth labels for same-gender outputs and flipped labels for opposite-gender outputs, preserving correct predictions in one path while confounding them in the other.The authors report that omitting this configuration overfits perturbations to the auxiliary gender classifier.Source: §2.3 · p. 5
- The matching loss compares VGG-face representations of the input and its same-gender-prototype output to preserve biometric similarity.The total loss combines the gender and matching terms.Source: §2.3 · p. 5
- The total loss gradients update the autoencoder, with the gender and matching terms weighted equally using constant coefficients of 1.The coefficients were not optimized.Source: §2.3 · p. 5
2.4. Datasets
CelebA supplied the training and test split, while MUCT, LFW, and AR-face were reserved for evaluation.
2.4. Datasets
- CelebA contains 202,599 labeled face images, split into 162,079 training images and 40,520 test images.CelebA-train was used to train both the gender classifier and convolutional autoencoder.Source: §2.4 · p. 5
- MUCT, LFW, and AR-face, alongside CelebA-test, were used to evaluate the trained model; images were resized to 224 × 224.A deformable part model preprocessing step ensured the common image dimensions.Source: §2.4 · p. 5, Table 1 · p. 6
2.5. Implementation details and software
The autoencoder and its auxiliary gender classifier and face matcher were implemented in TensorFlow, with the classifier and matcher frozen during autoencoder training.
2.5. Implementation details and software
- TensorFlow implementations used custom convolutional-layer code, and the auxiliary classifier and matcher parameters were frozen while training the autoencoder.The three components were the convolutional autoencoder, CNN-based gender classifier, and CNN-based face matcher.Source: §2.5 · p. 6
3. Experimental Results
Evaluation used independent gender classifiers and a commercial face matcher on reconstructions from same-, neutral-, and opposite-gender prototypes. Opposite-gender perturbations impaired gender prediction while matching performance remained close to that of original images.
3.1. Evaluation and verification
- The trained model was applied to CelebA-test, MUCT, LFW, and AR-face, generating same-, neutral-, and opposite-gender reconstructions for each input.The study compared outputs with the face-mixing approach and evaluated with independent software rather than the auxiliary training networks.Source: §3 · p. 6, §3.1 · p. 6
3. Experimental Results
- Figure 5 presents example inputs and reconstructions using the three gender prototypes, with CelebA-test examples in the first two rows and MUCT examples in the last two.Source: Figure 5 · p. 6
3.1.1 Perturbing gender
- Opposite-gender perturbations substantially increased gender-prediction errors, and IntraFace performance on AR-face approached random; G-COTS was more robust but also degraded.Table 2 reports error rates for original and opposite-gender-perturbed images; the face-mixing approach produced higher G-COTS error rates.Source: §3.1.1 · pp. 6–7, Table 2 · p. 7
3.2. Retaining matching accuracy
- TMR values for perturbed images from all three datasets were very close to the unperturbed originals, while opposite-gender outputs retained good matching performance.The ROC curves show same-gender reconstructions closer to the original than neutral- or opposite-gender reconstructions; the face-mixing approach heavily impacted matching.Source: Figure 7 · p. 7, §3.2 · p. 7
4. Conclusions
The proposed network confounds gender prediction while retaining biometric matching utility, and experiments using independent evaluators support its efficacy.
4. Conclusions
- The semi-adversarial network perturbs faces to confound gender prediction while retaining biometric matching utility.Training uses an auxiliary CNN gender classifier and face matcher; evaluation uses two independent gender classifiers and an unseen commercial matcher.Source: §4 · pp. 7–8
- Experiments found gender privacy could be imparted without unduly impacting face-matching accuracy.Source: §4 · p. 8