PaperlayerView exact revision on arXiv

Source-linked AI summary

A Study of Face Obfuscation in ImageNet

Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, Olga Russakovsky

arXiv:2103.06191v3cs.CV

TL;DR

Public ImageNet images contain incidental faces, creating privacy concerns despite the benchmark’s focus on object recognition. The paper annotates faces, obfuscates them, and evaluates recognition and transfer learning, finding minimal accuracy impact and broadly transferable features. It concludes that face obfuscation is feasible for privacy-aware visual recognition, while remaining incomplete as a privacy solution.

  • Problem

    ImageNet is publicly available and widely used, yet its images contain incidental people and faces whose privacy is not addressed by object-recognition benchmarks.

  • Method

    The authors annotate ImageNet faces with automatic detection and crowdsourced refinement, then benchmark blurred and overlaid images for classification and four downstream transfer-learning tasks.

  • Results

    Less than a 1% accuracy drop was observed for face-obfuscated classification, while features learned from obfuscated images remained transferable across four downstream tasks.

  • Takeaways & Limitations

    Face obfuscation can support privacy-aware visual recognition and is proposed as a step toward improving ImageNet and future visual datasets.

  • Takeaways & Limitations

    Face obfuscation alone is insufficient and provides no formal privacy guarantee because identity may remain inferable from other visual cues and attributes.

Abstract

from arXiv · show

Face obfuscation (blurring, mosaicing, etc.) has been shown to be effective for privacy protection; nevertheless, object recognition research typically assumes access to complete, unobfuscated images. In this paper, we explore the effects of face obfuscation on the popular ImageNet challenge visual recognition benchmark. Most categories in the ImageNet challenge are not people categories; however, many incidental people appear in the images, and their privacy is a concern. We first annotate faces in the dataset. Then we demonstrate that face obfuscation has minimal impact on the accuracy of recognition models. Concretely, we benchmark multiple deep neural networks on obfuscated images and observe that the overall recognition accuracy drops only slightly (<= 1.0%). Further, we experiment with transfer learning to 4 downstream tasks (object recognition, scene recognition, face attribute classification, and object detection) and show that features learned on obfuscated images are equally transferable. Our work demonstrates the feasibility of privacy-aware visual recognition, improves the highly-used ImageNet challenge benchmark, and suggests an important path for future visual datasets. Data and code are available at https://github.com/princetonvisualai/imagenet-face-obfuscation.

1. Introduction

ImageNet contains incidental faces that create privacy concerns despite most categories not depicting people. The paper annotates these faces and evaluates obfuscation, finding that privacy-aware recognition remains highly accurate and transferable.

  • Motivation: Only 3 of ImageNet’s 1000 ILSVRC categories are people categories, yet many images contain people alongside other objects.These incidental appearances create privacy concerns because ILSVRC is publicly available and widely used.
  • Approach: The authors construct a privacy-enhanced ILSVRC by annotating faces and creating obfuscated image versions.Faces are detected automatically with Amazon Rekognition and refined through Mechanical Turk crowdsourcing.
  • Face annotations: 17% of ILSVRC images contain at least one face, totaling 562,626 faces across 243,198 of 1,431,093 annotated images.Some non-people categories, including volleyball and military uniform, have faces in more than 90% of images.
  • Recognition accuracy: Validation accuracy drops only 0.1%–0.7% with blurring and 0.3%–1.0% with overlaying.The results support training privacy-aware visual classifiers with less than a 1% accuracy drop.
  • Transferability: Features pretrained on original or obfuscated images are evaluated across object recognition, scene recognition, object detection, and face attribute classification.The downstream tasks include both classification and spatial localization, as well as face-centric and face-agnostic recognition.
  • Implications: The paper releases face annotations and code while advocating face obfuscation for ImageNet and future dataset creation.Its stated contribution is to show that obfuscation does not significantly compromise classification or downstream-task accuracy while providing some privacy protection.

2. Related Work

The paper distinguishes privacy protection for public datasets from private-dataset security and situates face obfuscation among broader dataset-ethics and privacy approaches. It emphasizes crowdsourced verification and cautions that face obfuscation is incomplete and lacks formal privacy guarantees.

  • Privacy-preserving machine learning: Privacy-preserving machine learning primarily protects private datasets against attacks such as model inversion, membership inference, and training-data extraction.Differential privacy is presented as a general framework requiring similar model behavior whether or not an individual appears in training data.
  • Public-dataset privacy: This work instead addresses public datasets that expose private information about people who may not know they appear in the data.The distinction matters because the goal is protecting individuals represented in publicly available ImageNet, not preventing inference of the dataset itself.
  • Privacy in visual data: Prior visual datasets commonly use automatic face or license-plate obfuscation, whereas this paper combines automatic detection with crowdsourced verification.The authors argue that manual verification improves annotation accuracy and can partially mitigate racial and gender biases in automated face detection.
  • Scope of obfuscation: Face obfuscation is presented as an important first step because visual datasets contain many privacy attributes beyond faces, including names and addresses.The paper states that obfuscating faces alone is not sufficient for privacy protection.
  • Privacy guarantees: Face obfuscation provides no formal privacy guarantee because people and machines may infer identity from cues outside the face, such as height and clothing.Methods tuned to particular recognition models likewise provide no privacy guarantee.
  • Potential negative impacts: The authors warn that face obfuscation may create a false impression of privacy guarantees and may affect demographic groups unequally.They identify manual annotation as a partial mitigation for disparities introduced by automated face detection.

3. Annotating Faces in ILSVRC

The paper builds a two-stage face-annotation pipeline for all ILSVRC images, combining automatic detection with crowdsourced refinement. The resulting annotations show that faces are widespread across categories, including many non-people categories.

  • Face annotation: A two-stage pipeline combines automatic face detection with crowdsourcing to produce face annotations at ImageNet scale.Automatic detection provides coverage, while human inspection refines the results.
  • Annotation quality: Crowdsourcing reduces both false positives and false negatives relative to automatic detection in challenging categories.Across 20 deliberately difficult categories, the final annotations average 1.25 false positives and 0.95 false negatives per 50 images.
  • Annotation coverage: 1,431,093 images yield 562,626 annotated faces across 243,198 images, or 17% of all images.The annotations cover the full ILSVRC image collection.
  • Face distribution: Face prevalence varies sharply across categories: 106 categories exceed 50% images with faces, while 216 exceed 25%.The fraction ranges from 97.5% for bridegroom to 0.1% for rock beauty.
  • Face distribution: Most images containing faces have fewer than five faces, but some contain up to 100, especially sports scenes with crowds.The reported cap of 100 comes from Amazon Rekognition.
  • Category patterns: Face prevalence differs across WordNet supercategories, with clothing and musical instruments frequently co-occurring with people while birds and insects seldom do.The comparison uses supercategories whose face fractions significantly deviate from the ILSVRC average of 17%.

4. Effects of Face Obfuscation on Classification Accuracy

Face obfuscation produces only small overall classification losses, but impacts categories more when blurred faces cover the object or remove task-relevant visual cues. Some large top-1 shifts between similar categories reflect changed prediction preferences rather than broad recognition failure.

  • Blurring lowers top-5 accuracy by 0.1%–0.7%, averaging 0.4%, while overlaying averages a 0.7% drop.Both methods produce small but consistent losses across evaluated models.
  • Accuracy drops increase with the fraction of blurred area, reaching 4.04% in top-5 accuracy for categories with at least 8% blurred area.For 0%–1% blurred area, the corresponding top-5 drop is 0.30%; top-1 drops remain around 0.5% until blurred area exceeds 4%.
  • Categories with more object area covered by faces suffer larger accuracy drops, including 8.71% for mask and 8.93% for harmonica in top-5 accuracy.The corresponding face-covered fractions are 24.84% and 29.09%, respectively.
  • Grad-CAM shows that blurring can prevent models from attending to and classifying objects such as mask and harmonica.On original images, the model localizes the object of interest effectively; on blurred images, it attends to neither the correct region nor the object.
  • Large opposite top-1 changes for visually similar categories can arise from altered preferences: eskimo dog drops 12.8% while siberian husky rises 16.9%.The pairwise pattern disappears under top-5 accuracy and average precision, which are less sensitive to competing categories.

5. Effects on Feature Transferability

Features pretrained on face-obfuscated ImageNet transfer comparably to features pretrained on original images across four downstream tasks. This remains true for object and scene recognition, object detection, and face attribute classification.

  • On CIFAR-10 and SUN, models pretrained on blurred or overlaid images perform closely to models pretrained on original images.Pretraining itself provides a significant benefit on both datasets.
  • PASCAL VOC detection shows no significant difference between original, blurred, and overlaid pretraining: 79.40 ± 0.31, 79.29 ± 0.22, and 79.39 ± 0.02 mAP.The person-category AP also remains comparable: 84.40 ± 0.14, 84.80 ± 0.50, and 84.47 ± 0.05.
  • On CelebA face attribute classification, blurred and overlaid pretraining remains competitive despite the task relying heavily on faces.The authors suggest pretraining may learn low-level face-agnostic features that are later adapted during finetuning.
  • Across four downstream tasks, pretraining on obfuscated images does not hurt feature transferability.The tasks include object recognition, scene recognition, object detection, and face attribute classification.

6. Conclusion

The study finds that face obfuscation enhances privacy with minimal impact on recognition accuracy after annotating faces and benchmarking deep networks on blurred and overlaid images.

  • Face obfuscation enhances privacy with minimal impact on recognition accuracy.

A. Semi-Automatic Face Annotation

The paper combines automatic face detection with crowdsourced correction to obtain privacy-oriented annotations, while accepting detector false positives and addressing false negatives. Crowdsourcing quality control further improves annotation reliability, although NSFW areas are not distinguished from faces.

  • Annotation pipeline: Face annotation uses two stages: automatic detection followed by crowdsourced refinement.Amazon Rekognition proposes bounding boxes, which Mechanical Turk workers adjust or supplement.
  • Detector behavior: Rekognition produces false positives and false negatives, with most false positives being animal faces and false negatives occurring under poor lighting or heavy occlusion.The authors consider false positives acceptable for privacy preservation but regard false negatives as undesirable.
  • Crowdsourced refinement: Workers correct detected boxes and add missing coverage for faces and NSFW areas during Mechanical Turk refinement.Face coverage explicitly includes the mouth, nose, eyes, forehead, and cheeks.
  • Quality control: Each HIT contains 50 images and 3 gold-standard images, while a two-life game-like protocol improves annotation quality.Workers lose a life for a gold-standard mistake and restart after losing both lives.
  • Annotation scope: NSFW areas are annotated together with faces, but they are rare overall and vary substantially across ImageNet categories.In bikini, 25 of 1,350 examined images contained NSFW annotations, corresponding to 1.85%.

B. Face Blurring Method

The face-blurring method smoothly composites blurred and original image regions using an enlarged, softened face mask. This avoids sharp boundaries between blurred and unblurred areas.

  • Mask smoothing: Gaussian blurring is applied to both the image and the face mask to soften the transition between processed and unprocessed regions.The blur radius depends on the largest bounding-box diagonal.
  • Mask construction: The method enlarges face bounding boxes and represents their union as a binary mask over the image.Coordinates outside the image are truncated to valid boundaries.
  • Image compositing: The final image composites the blurred image with the original using the blurred mask: Inew = Mblurred · Iblurred + (1 − Mblurred) · I.Using Mblurred rather than the hard mask M avoids sharp boundaries.

C. Original Images for Training and Obfuscated Images for Evaluation

The study evaluates models trained on original images against images obfuscated by blurring or color overlays. All obfuscation methods cause only a small top-5 accuracy decline, with blurring performing best on average.

  • Evaluation setup: Models trained on original images are evaluated on images blurred or overlaid with average-color, red, green, or blue patches.The experiments use five face-obfuscation methods in total.
  • Accuracy results: 0.7%–1.5% average top-5 accuracy drops occur across obfuscation methods, with blurring producing the smallest decline.The decline reflects information loss and a training–evaluation distribution mismatch.

D. Obfuscated Images for Training and Original Images for Evaluation

Training on blurred images and evaluating on original images also causes only a small top-5 accuracy reduction, supporting the practical use of privacy-preserved training data for deployment without obfuscation.

  • Accuracy results: 0.25%–1.04% top-5 accuracy drops occur when models trained on blurred images are evaluated on original images.The average drop is 0.67%.

E. Details of Transfer Learning Experiments

The transfer-learning experiments finetune pretrained models on three image-classification tasks and an object-detection task. Training uses original or face-obfuscated ILSVRC images, with blurred-image training producing a small but consistent validation-accuracy drop.

  • Image classification: The classification experiments transfer pretrained models to CIFAR-10, SUN, and CelebA by replacing the output layer and finetuning for 90 epochs.CelebA face attribute classification uses binary cross-entropy independently for each label.
  • Downstream tasks: The transfer-learning evaluation covers object recognition, scene recognition, face attribute classification, and object detection across four downstream tasks.The downstream classification tasks are CIFAR-10, SUN, and CelebA; detection uses PASCAL VOC.
  • Object detection: Object detection transfers a ResNet50 backbone pretrained on original or face-obfuscated ILSVRC to Faster R-CNN on PASCAL VOC.The detector is finetuned for 10 epochs on the 2007 and 2012 trainval sets and evaluated on the 2007 test set.
  • Validation comparison: Training on blurred images leads to a small but consistent accuracy drop on validation accuracies measured on original ILSVRC images.The cited table compares models trained on original versus blurred images.
Loading 2103.06191v3…