Source-linked AI summary
A Hybrid Model for Identity Obfuscation by Face Replacement
Qianru Sun, Ayush Tewari, Weipeng Xu, Mario Fritz, Christian Theobalt, Bernt Schiele
TL;DR
The paper addresses the challenge of hiding identities in increasingly searchable social-media photos without sacrificing realism. It combines parametric face synthesis for identity control with GAN-based head synthesis for detail and blending, and reports improved obfuscation and visual realism over prior methods.
Problem
Identity obfuscation must prevent recognition in widely shared photos while preserving realistic visual content.
Method
The method replaces identity-related parameters in a reconstructed parametric face while preserving expression, then uses a GAN to synthesize and blend the complete head.
Results
The best method achieves a 2.6% lower recognition rate than Blackhead+PDMDec using head features and higher image quality, with HPS 0.33 vs. 0.15.
Takeaways & Limitations
The hybrid system produces obfuscated images that confuse machine and human recognizers while retaining higher visual realism than previous state-of-the-art methods.
Takeaways & Limitations
Stage-I renderings lack high-frequency details, blend imperfectly, and require Stage-II processing to reduce contextual identity information from the full head.
Abstract
from arXiv · showhide
As more and more personal photos are shared and tagged in social media, avoiding privacy risks such as unintended recognition becomes increasingly challenging. We propose a new hybrid approach to obfuscate identities in photos by head replacement. Our approach combines state of the art parametric face synthesis with latest advances in Generative Adversarial Networks (GAN) for data-driven image synthesis. On the one hand, the parametric part of our method gives us control over the facial parameters and allows for explicit manipulation of the identity. On the other hand, the data-driven aspects allow for adding fine details and overall realism as well as seamless blending into the scene context. In our experiments, we show highly realistic output of our system that improves over the previous state of the art in obfuscation rate while preserving a higher similarity to the original image content.
1 Introduction
Reliable identity obfuscation must counter stronger recognition systems while preserving realistic visual content. The proposed hybrid method combines controllable face replacement with GAN-based head synthesis to address complementary weaknesses.
- Publicly shared photos face increasing privacy risks because deep neural networks enable more reliable indexing and recognition of personal information.
- Effective obfuscation must hide identity information while preserving the realism of the visual data.
- Existing approaches range from unpleasant occluders to imperceptible perturbations and substantial appearance changes intended to defeat machine or human recognition.
- Parametric face models control identity, pose, and expression, but naive face replacement lacks fine details, can blend unnaturally, and leaves head and hair identity cues untouched.
- The proposed two-stage method replaces identity-related face parameters while preserving expression, then uses a GAN to synthesize the complete head, inpaint missing regions, and add fine details.
- Qualitative and quantitative evaluations report lower recognition rates and higher image quality than baseline methods on publicly available datasets.
2 Related work
Prior identity-obfuscation methods trade off realism, generality, and control. The paper positions its hybrid approach against these limitations by combining explicit parametric manipulation with data-driven image synthesis.
- Blurring, mosaics, and black bars introduce unpleasant artifacts and become less effective as CNN-based recognition improves.
- Adversarial-example methods can strongly confuse target-specific recognizers while offering no guarantee against target-generic systems.
- Full-body synthesis can mismatch scene context when generated people have uniform poses, producing blending artifacts.
- A purely data-driven fake-head method achieves strong recognition-rate and image-quality results but can alter face poses and expressions, producing artifacts [1].
- The proposed second stage uses GAN-based refinement and inpainting to improve realism around rendered 3DMM faces.
3 Face replacement framework
The framework combines parametric face reconstruction and identity control with GAN-based head synthesis to produce realistic identity-obfuscated images. It uses two stages: Stage-I changes identity while preserving selected facial attributes, and Stage-II inpaints the head context and adds realism.
- The proposed face replacement approach combines a data-driven method with a parametric face model.
- Stage-II: Inpainting: Stage-II uses a GAN conditioned on the obfuscated image and rendered face to synthesize the complete head, inpaint surrounding regions, and add fine-grained detail.The generator receives the obfuscated image and Stage-I rendering, while whole-body conditioning supports natural transitions.
- Stage-I: Face replacement: Stage-I reconstructs a 3D face from an input image, then replaces identity-related shape and reflectance parameters while preserving other facial dimensions.The semantic parameter vector includes shape, reflectance, expression, pose, and illumination; shape and reflectance encode identity.
- Stage-I: Face replacement: Identity parameters are clustered into 15 representatives, and the closest, intermediate, or furthest cluster is selected to produce different obfuscation levels.Each test image receives its own Replacer1, Replacer8, and Replacer15 choices.
- Stage-I: Face replacement: When the input face is blurred or blacked out, the Stage-I network predicts reconstructions from context, while refinement optimizes only rigid pose and expression for better alignment.The resulting reconstructions preserve the face-region boundary and skin color, and ten Gauss–Newton iterations improve alignment.
- Stage-II: Inpainting: Stage-II addresses the limited expressiveness and missing high-frequency detail of the Stage-I parametric rendering, while de-identification losses from identity-verification models are not applicable to this approach.The authors report that verification loss becomes zero when the inpainted image is sufficiently close to the ground-truth image.
4 Recognizers
The paper evaluates target-generic identity obfuscation against both machine and human recognizers. Machine evaluation uses head and head+body cues, while human evaluation measures recognition of the original person and perceived identity distance.
- 4 Recognizers: Target-generic obfuscation is evaluated against both machine and human recognizers.The stated goal is to work against any recognizer, whether machine-based or human.
- 4 Recognizers: Naeil recognizes people using body and scene context, making it relatively resistant to blacking-out or blurring the head.The paper identifies naeil as a state-of-the-art recognizer for social media images.
- 4 Recognizers: Machine evaluation trains head and body feature extractors, then uses SVM identity classifiers with head or head+body features.The authors also report similar trends for AlexNet-based analogues in supplementary materials.
- 4 Recognizers: Human evaluation asks participants to recognize the original person and select the result farthest from the original identity.Participants compare inpainted outputs from method variants and other methods while focusing on identity recognition rather than image quality.
5 Experiments
The experiments evaluate the hybrid obfuscation approach on realistic social-media images using recognition rates and visual realism, across input modalities and against prior work. The results show that face replacement improves privacy while retaining image quality, with a configurable trade-off between obfuscation and realism.
- Experimental setup: The evaluation uses PIPA’s 37,107 Flickr images and 63,188 annotated person instances, with training and test identities separated.Machine recognition is measured on 1,084 test images, while human recognition uses 45 instances.
- Evaluation metrics: The study measures image realism with mask-SSIM or human perceptual scores and obfuscation with machine recognition rates and human confusion rates.Mask-SSIM and HPS are higher-is-better quality measures, whereas machine recognition is lower-is-better and human confusion is higher-is-better.
- Comparison to the state-of-the-art: v12 reduces machine recognition by 2.6% and increases human-recognizer confusion by 15.3% versus Blackhead+PDMDec, while improving HPS from 0.15 to 0.33.The comparison uses head features and reports v12 as the best obfuscation method.
- Parametric model versus GAN: The hybrid model outperforms GAN-only replacement in the quality–privacy trade-off: v14 lowers recognition to 19.7% but has mask-SSIM 0.23, whereas v9 reaches 25.7% with 0.47.Using parametric face replacement achieves 18.1% recognition while sacrificing only 0.08 HPS relative to v9 and v12.
- Analysis of different face/hair modalities: Increasing image modification generally improves obfuscation but reduces quality, although recognition rates drop faster than image quality.For example, blurring hair reduces recognition to 25.8%, and removing face and hair information reduces it to 14.2%.
- Identity obfuscation: Face replacement lowers recognition more effectively with Replacer15 than Own rendered faces, while replacing faces from farther identities further improves obfuscation.Using Replacer8 instead of Replacer1 improves human obfuscation by 25.1%.
- Trade-off between image quality and obfuscation: Scatter curves let users select the highest-quality method under a privacy threshold; at 30% recognizability, the best quality is about 0.45.Under a 20% privacy-leakage constraint, the proposed blacked-out-face and blacked-out-hair method reaches image quality 0.33 versus 0.15 for Blackhead+PDMDec.
6 Conclusion
The paper introduces a hybrid head-replacement method that combines parametric face modeling with GAN-based synthesis to control identity while producing realistic obfuscated images.
- The hybrid method combines parametric face reconstruction and rendering with GAN-based image synthesis for identity obfuscation by head replacement.The parametric component enables explicit facial-parameter control, while the GAN component supports photo-realistic synthesis.
- The synthesized images confuse both machine recognition systems and human recognizers.
- The experiments show improved obfuscation rates over prior state of the art while generating images with higher visual realism.
A Network architectures
The Head Generator G is implemented as a U-Net for stage II, producing a full 256 × 256 × 3 image whose head output is used for obfuscation.
- The stage-II Head Generator G uses a U-Net architecture corresponding to the Inpainter in the main pipeline.
- Although the network outputs a complete body-and-head image, only the cropped head region is pasted onto the obfuscated image.
- Because the pasted head is the effective output, only the head region provides feedback during back-propagation.
B Implementation details
The implementation uses separate optimization procedures for the stage-I network and the stage-II Head Generator, with fixed-size body crops prepared by resizing, cropping, and zero-padding.
- Stage I uses an AlexNet encoder trained with AdaDelta for 200k iterations, batch size 5, and learning rate 10^-3.
- The stage-II Head Generator uses Adam with λH = 1000, initial learning rates of 2 × 10^-5, batch size 6, and 10,000 iterations.The generator receives five updates per iteration and the discriminator one; learning rates halve every 5,000 iterations.
- Inputs are normalized to 256 × 256 crops by resizing the head to one-quarter of the crop height, extracting a 3-head-width × 4-head-height region, and zero-padding when necessary.
C Obfuscation performance against AlexNet
Evaluation with an AlexNet-based recognizer shows that recognition rates differ from GoogleNet, with the direction of differences depending on the input features used.
- AlexNet recognizes original images at 81.6%, below GoogleNet’s 85.6%, while recognition rates on generated images vary across modalities.
- For head features, AlexNet has higher recognition rates than GoogleNet for 12 of 21 input modalities.
- For head+body features, GoogleNet has higher recognition rates for 18 input modalities than AlexNet.
- The authors attribute these differences to feature dimensionality and to noisy body features reducing final recognition rates when concatenated.
D Visualization results
The section visualizes methods v2–v21 across original, blurred, and blacked-out face inputs, corresponding to Table 2. Results use consistently cropped images, with the best visual quality usually appearing when original hair images are used.
- Figures 9–11 visualize methods v2–v21 using original, blurred, and blacked-out face images, respectively.
- The results are consistently cropped to include small zero-padded regions.
- The second column, which uses original hair images, achieves the best visual quality in most cases.
- Table 2 compares image quality with Mask-SSIM, SSIM, and HPS, and evaluates obfuscation using machine-recognizer rates.