Source-linked AI summary
Natural and Effective Obfuscation by Head Inpainting
Qianru Sun, Liqian Ma, Seong Joon Oh, Luc Van Gool, Bernt Schiele, Mario Fritz
TL;DR
The paper addresses the need for identity obfuscation that is both shareable-looking and effective against generic person recognizers. It proposes two-stage, landmark-guided head inpainting using image context, and reports natural-looking obfuscation with reduced recognition rates across evaluated settings.
Problem
Existing blacking and blurring methods provide poor user experience and are ineffective against machine recognizers, while realistic inpainting is difficult for diverse social-media poses and backgrounds.
Method
The method generates or detects facial landmarks from context, then uses them to condition head inpainting that blends with the body and scene.
Results
The head-inpainting methods achieve below 50% and below 21% head+body recognition rates for blurhead and blackhead inputs, respectively, while improving user fooling rates over related methods.
Takeaways & Limitations
Head inpainting provides natural and effective target-generic obfuscation and can upgrade already head-obfuscated images without the original face.
Takeaways & Limitations
The reported evaluation includes a particular machine-recognition system, with similar trends verified for AlexNet-based analogues.
Abstract
from arXiv · showhide
As more and more personal photos are shared online, being able to obfuscate identities in such photos is becoming a necessity for privacy protection. People have largely resorted to blacking out or blurring head regions, but they result in poor user experience while being surprisingly ineffective against state of the art person recognizers. In this work, we propose a novel head inpainting obfuscation technique. Generating a realistic head inpainting in social media photos is challenging because subjects appear in diverse activities and head orientations. We thus split the task into two sub-tasks: (1) facial landmark generation from image context (e.g. body pose) for seamless hypothesis of sensible head pose, and (2) facial landmark conditioned head inpainting. We verify that our inpainting method generates realistic person images, while achieving superior obfuscation performance against automatic person recognizers.
1. Introduction
The paper addresses identity obfuscation in social media photos by targeting both visual naturalness and protection effectiveness. It proposes a two-stage, landmark-guided head-inpainting approach for diverse poses, backgrounds, and inputs.
- Motivation: Social-media obfuscation must balance natural-looking images with effective identity protection.Black-box obfuscation may protect identity but is unpleasant to share.
- Problem: Common blacking, eye bars, and blurring are neither visually pleasant nor effective against machine recognizers.The paper frames head inpainting as a target-generic alternative.
- Problem: Realistic head inpainting is difficult because social-media photos contain varied activities, backgrounds, and head poses, while existing face models often assume frontal or aligned faces.These conditions motivate using broader image context and explicit facial structure.
- Approach: The method first detects or generates facial landmarks, then inpaints a realistic head conditioned on those landmarks and surrounding context.Generated landmarks support already-obfuscated blackhead images when the original face is unavailable.
- Contributions: The contributions include natural and effective head-inpainting obfuscation plus landmark generation for visible-head and blackhead scenarios.The landmark generator hypothesizes plausible facial structures and poses from contextual information.
2. Related work
Prior identity-obfuscation work divides into target-specific adversarial methods and target-generic appearance changes. This paper combines context-based GAN inpainting with landmark guidance to handle diverse social-media poses and backgrounds without head-region information.
- Target-generic obfuscation: Target-generic obfuscation changes a person’s appearance for generic recognizers or humans, whereas blacking and blurring are visually unpleasant and machine-ineffective.The paper positions head inpainting within this target-generic category.
- Target-specific obfuscation: Target-specific adversarial obfuscation can provide strong protection with imperceptible changes, but only against selected machine systems.Its protection guarantee does not extend generally beyond the targeted recognizers.
- Image inpainting: Unlike prior context-based approaches assuming similarity between missing content and its surroundings, this method generates heads from body and scene context alone.The approach is designed for diverse poses and backgrounds rather than aligned faces.
- Structure-guided generation: Facial landmarks condition head generation to support realistic images with complex internal structure and natural alignment to body pose and scene context.This connects the method to structure-guided image-generation research.
3. Head inpainting framework
The framework separates head inpainting into landmark detection or generation and landmark-conditioned head synthesis, supporting both original and already-obfuscated inputs. Stage-I hypothesizes facial structure from body and scene context, while Stage-II generates a head that transitions naturally into the surrounding image.
- Framework overview: The two-stage framework accepts either an original or blackhead image, then detects or generates landmarks before producing the inpainted image.Stage-II uses the blackhead image and landmark channels as input.
- Stage-I: Landmark: Stage-I detects 68 facial keypoints from an original image or generates landmark coordinates from a blackhead image and head mask.Landmark generation uses an adversarially trained Landmark Generator and Discriminator.
- Stage-I: Landmark: The landmark generator encodes blackhead body and scene context into a latent variable and decodes it into 2 × 68 landmark coordinates.Its encoder uses an auto-encoder structure, while alternative decoders are pretrained with an auto-encoder or Point Distribution Model.
- Stage-I: Landmark: The Point Distribution Model represents landmarks with scale, orientation, translation, and non-rigid shape parameters derived from 3D landmark structure.The model uses n = 34 principal components in the experiments to balance consistency and flexibility.
- Stage-I: Landmark: Stage-I combines L2 and adversarial losses because L2-only landmark training produces noisy alignments that can be detected and remedied by a discriminator.Generated landmarks are treated as fake and detected landmarks as real during discriminator training.
- Stage-II: Inpainting: Stage-II conditions a U-Net-like head generator on the blackhead image and landmark heatmaps, using the whole body image to create a natural head-to-surroundings transition.A head discriminator compares the fused generated-head image with the original image, and optimization uses L1 plus adversarial loss.
4. Experiments
The experiments evaluate the two-stage head-inpainting pipeline on diverse social-media images, measuring landmark quality, inpainting quality, naturalness, and identity obfuscation against person recognizers. Results show that head inpainting can preserve realistic visual cues while reducing recognition rates more than blacking or blurring.
- 4.1. Dataset: The pipeline is evaluated on PIPA, using 34,383 training instances and 1,909 test instances after pruning profile and back-view heads.PIPA contains diverse events, activities, and poses; the generators use fixed-size 256×256×3 inputs.
- 4.3.1. Landmark: Landmark quality is measured by L2 distance and interocular-normalized L2 distance against detected landmarks treated as ground truth proxies.The study varies input type, loss function, and decoder type, including original, blackhead, and blurhead inputs.
- 4.3.1. Landmark: 6.32 versus 13.6 L2 distance shows blurhead inputs produce closer landmarks than blackhead inputs for a scratch-trained decoder using L2 loss.Adding adversarial loss changes the blackhead result from 13.6 to 13.0 in the stated setting.
- 4.3.2. Inpainting: 0.872 SSIM is obtained by the nearest-neighbor copy-paste baseline, lower than all reported inpainting results because it ignores blending with surroundings.The comparison evaluates whole-image SSIM and head-region mask-SSIM against original images.
- 4.3.3. Obfuscation: 17.4% recognition for the final-row head recognizer versus 5.6% shows lower rates when the recognizer relies solely on the inpainted head rather than head-plus-body cues.The comparison is near the reported 2.1% chance-level recognition rate.
- 4.3.3. Obfuscation: < 50% and < 21% head-plus-body recognition rates for blurhead- and blackhead-based inpainting outperform blacking and blurring baselines at 67.0% and 71.6%.Without obfuscation, head-plus-body recognition is 88.3%.
- 4.3.3. Obfuscation: 71.0% versus 72.2% head-region attention indicates that inpainted heads attract recognizer attention nearly as much as non-obfuscated heads.The authors connect this attention pattern with recognizer misjudgment based on realistic inpainted heads.
4.4. Qualitative results
Qualitative results examine whether generated landmarks remain plausible for the body pose while changing the apparent head identity. Generated landmarks can produce realistic, shifted head poses, and blurred inputs improve visual inpainting quality compared with blacked-out inputs.
- Detected versus generated landmarks: Generated landmarks, especially for blackhead inputs, produce different head poses while remaining plausible for the body pose and activity.Detected landmarks closely follow the original image, whereas generated landmarks can shift appearance without losing realism.
- Detected versus generated landmarks: Generated landmarks further mask identity information while maintaining reasonable realism, with consistently lower recognition rates than inpainting based on detected landmarks.The qualitative discussion links altered face shape and pose with stronger obfuscation.
- Blackhead versus blurhead: Blurred head inputs improve landmark and head-generation quality because they retain structural information about facial keypoints.The study compares blurhead and blackhead cases visually in Figure 5.
4.5. Comparing with the state-of-the-art
On PIPA, the proposed inpainting models produce more realistic head structures and higher human-perceived realism than the Context Encoder baseline, while also supporting landmark-based obfuscation.
- The human perceptual study showed that Context Encoder quality was not competitive, so it was evaluated through HPS rather than visualized.The study presented 55 real and 55 inpainted images per method to 20 users, who classified each within one second.
- Figure 5 compares head inpainting from detected versus generated landmarks and reports landmark-generation error for each example.Detected landmarks follow the original image closely, whereas generated landmarks can produce different but plausible head poses, especially for blackhead cases.
- 60% and 39% real-image judgments were achieved in blurhead cases, while blackhead inputs reached 11% with generated landmarks and improved by 8pp with detected landmarks.The 11% rate was nearly threefold higher than the Context Encoder baseline; original unaltered images were judged 93% real.
- Our methods achieved landmark detection success ratios above 95%, compared with 36% for Context Encoder.The higher LDSR indicates clearer face structures in the inpainted heads.
5. Conclusion
The paper presents a two-stage head-inpainting method for identity obfuscation in challenging social-media photos, generating natural images that confuse automatic person recognizers.
- The method generates natural and effective obfuscation patterns despite diverse head and body poses and backgrounds in social-media photos.
- The obfuscation is target-generic and is designed to work against any recognizer.The conclusion contrasts this scope with obfuscations designed for specific target machine systems.
A. Network architectures
The supplementary architectures define separate networks for landmark encoding, landmark reconstruction, and head generation, with final processing restricted to the head region.
- Figures 6–8 present architectures for the Landmark Generator encoder, landmark Auto-encoder, and Head Generator, respectively.
- The deep network outputs a 256x256x3 intact image, after which cropping and pasting using the head mask leaves only the head region generated.
B. Implementation details
Implementation uses Adam optimization with fixed loss weights and specified minibatch, iteration, update, and dataset settings for landmark and head generation.
- Both landmark and head generators use Adam with λL = 2 and λH = 50, starting at 2 × 10^-5 and halving the learning rate every 5,000 iterations.
- Landmark-generation models use minibatch size 16 for 10,000 iterations, with five generator updates and one discriminator update per iteration.Training uses 34,383 examples, corresponding to about 23.3 generator epochs and 4.7 discriminator epochs.
- Head-generation models use minibatch size 6 for 13,000 iterations, with five generator updates and one discriminator update per iteration.This corresponds to about 8.7 generator epochs and 1.7 discriminator epochs.
C. Visualization results
The PDMDec model generally produces the best visual quality and lowest landmark-generation errors, while direct copy-paste yields unpleasant outputs despite matching face poses to bodies in most cases.
- PDMDec achieves the best visual quality and lowest landmark-generation errors in most blurhead cases.The comparison includes L2, L2 + DL, and decoder variants, with PDMDec performing best visually and by landmark error.
- For blackhead inputs, PDMDec again provides the best visual quality, but smaller mean L2 landmark error does not necessarily indicate better visualization quality.Blackhead quality is judged by facial-organ consistency when predicted pose and position are reasonable; mean L2 distance is only a reference.
- Direct copy-paste matches face poses to bodies in most cases but produces unpleasant output images.Candidate heads are searched in training data using normalized L2 distance between detected landmarks.
D. Obfuscation performance against AlexNet
Against AlexNet, the proposed inpainting method remains superior to blacking or blurring, reducing recognition rates for both blurhead and blackhead inputs. AlexNet is more robust to head obfuscation partly because it relies more on non-head regions.
- AlexNet relies less on head regions, with head contributions of 66.0% on clean images versus 72.2% for GoogleNet.Head contribution also drops consistently by approximately 20%–30% on inpainted images.
- For body+head, inpainting lowers AlexNet recognition from 67.0% to 45.6% for blurheads and from 63.2% to 16.6% for blackheads.These results support the same superiority of inpainting-based obfuscation over common blacking or blurring patterns observed with GoogleNet.
- Inpainted head images direct recognizer attention toward the head region, inducing a wrong decision based on the generated head.