Source-linked AI summary
Misanthrope: A Privacy-Preserving Keypoint Detector
Francesco Vultaggio, Predrag Djindjic, Markus Gerke, Sebastian Tschiatschek, Phillipp Fanta-Jende
TL;DR
Local image features enable important matching applications but are vulnerable to inversion attacks that can expose and re-identify people. Misanthrope uses privacy-directed self-distillation to avoid detecting keypoints on people, reducing privacy leakage at the source. It maintains or improves matching performance, reaching the top sparse-feature-extractor result in 7 out of 9 Phototourism scenes.
Problem
Inversion attacks can reconstruct privacy-sensitive scene content and enable person detection or re-identification from local image features.
Method
Misanthrope trains a keypoint detector through self-distillation to avoid detecting keypoints on people without a separate segmentation module at inference.
Results
Misanthrope substantially reduces person detection and re-identification from inverted images while matching or outperforming existing detectors in scenes containing people.
Takeaways & Limitations
Preventing privacy-sensitive information from entering the feature representation preserves privacy benefits while avoiding the matching penalties associated with human distractors.
Takeaways & Limitations
In portrait-like scenes, Misanthrope can assign approximately 60% of 10,000 keypoints to person-labeled regions, compared with 94% for DeDoDe.
Abstract
from arXiv · showhide
Image matching is a core component of applications such as Simultaneous Localization and Mapping (SLAM), Visual Localization, and Structure from Motion (SfM). However, the local image features central to this task are vulnerable to inversion attacks, which enable adversaries to reconstruct privacy-sensitive scene content from local features. These attacks pose a particular threat in distributed computing scenarios where the pre-computed features leave edge devices to be processed by remote servers. In this work, we introduce Misanthrope, a novel privacy-preserving keypoint detector trained through self-distillation to avoid detecting keypoints on people---a predominant source of privacy-sensitive content in most localization scenarios---thus mitigating inversion attacks at the source rather than through post-hoc obfuscation. We demonstrate how inverted images from traditional feature detection pipelines can be used to detect and re-identify people in the scene, while Misanthrope is able to mitigate these attacks. Furthermore, Misanthrope maintains image matching performance on par with the state of the art and even surpasses it in challenging settings where people act as distractors, such as phototourism and in-the-wild odometry. On the Image Matching Challenge 2021 Phototourism test set, Misanthrope is the top-performing sparse feature extractor in 7 out of 9 scenes. We make our model and its evaluation script available here: https://github.com/fratopa/misanthrope
1 Introduction
Image features support major vision applications but can expose privacy-sensitive scene content through inversion attacks. Misanthrope addresses this threat by avoiding keypoints on people during detection while preserving or improving matching performance.
- Motivation: Sparse image features can reveal scene contents and pose privacy risks when processed by inversion attacks.Such risks affect systems relying on local features or Structure from Motion representations.
- Motivation: Obfuscation defenses can be fragile, incompatible with keypoint pipelines, and harmful to localization accuracy.Newer inversion methods have succeeded against several obfuscated representations.
- Motivation: Avoiding privacy-sensitive features at detection time is preferable where private content and localizable scene geometry are separable.People can also act as visual distractors that degrade matching performance.
- Motivation: Separate segmentation-based filtering is computationally expensive and adds latency, especially on edge devices with fixed budgets.Its computational cost can reduce resources available to the feature extractor.
- Approach: Misanthrope uses privacy-directed self-distillation to train a detector that avoids detecting features on people without a separate segmentation module at inference.A DeDoDe teacher is filtered with person masks, and an identical student learns the filtered predictions.
2 Related Work
Prior work addresses image matching, semantic guidance, distillation, and privacy threats through separate lines of research. Misanthrope combines these ideas into source-level semantic avoidance rather than post-hoc representation obfuscation.
- Image Matching: Modern image matching pipelines extract keypoints and descriptors before matching correspondences across images.Deep models vary in whether detection and description use shared or separate networks.
- Semantic Guidance: Semantic guidance has been used to improve descriptors or filter dynamic-object keypoints through separate segmentation stages.Misanthrope instead targets people for privacy and embeds avoidance in detector training.
- Knowledge Distillation: Knowledge distillation in image matching has primarily served model compression, whereas Misanthrope uses identical teacher and student architectures to impose behavioral constraints.The constrained behavior is avoiding keypoints on people without changing the detector structure.
- Privacy Threats: Inversion attacks can reconstruct detailed images, 3D scenes, point clouds, and approximate contents from local features or embeddings.Reconstructed imagery can be sufficiently detailed to support person detection and re-identification.
- Privacy Threats: Existing obfuscation methods modify geometric or descriptor representations to disrupt inversion algorithms after feature extraction.Examples include permuted 3D coordinates, line representations, and ray clouds.
- Privacy Threats: Misanthrope prevents privacy-sensitive data from entering the feature representation, exploiting settings where private information is separable from localization geometry.Absent information cannot be recovered by future inversion algorithms, while residual leakage can be measured before deployment.
3 Method
Misanthrope adapts a keypoint detector through semantic-filtered teacher-student self-distillation. The student learns to suppress activations on people while retaining robust keypoints elsewhere.
- Training Design: Misanthrope trains a student detector to mimic teacher predictions while avoiding regions semantically identified as people.The detector architecture is unchanged, and supervision rather than structure is modified.
- Teacher-Student Distillation: DeDoDe supplies both teacher and student detectors, with the teacher producing a spatial keypoint activation map.The map represents the likelihood that each pixel is a keypoint.
- Semantic Filtering: Teacher probabilities are computed over spatial locations, zeroed in person regions, and re-normalized into a valid distribution.The semantic mask marks person pixels, which receive zero probability before re-normalization.
- Student Supervision: The student minimizes KL divergence between the filtered teacher distribution and its predicted keypoint probability map.This supervision teaches low activation on people while preserving robustness elsewhere.
- Training Data: COCO provides training images with person segmentation masks, yielding 64,115 training images and 2,696 validation images after filtering.The authors note that COCO polygon masks have coarse boundaries.
- Implementation: Teacher and student share pre-trained weights and architecture, while the teacher remains frozen and only the student is updated.The implementation uses DeDoDe-L as the base model.
- Implementation: Training runs for 60,000 iterations with AdamW, mixed precision, warm-up, and validation-based learning-rate reduction.The batch size is 14 and the weight decay is 10^-4.
4 Experiments
Experiments show that Misanthrope suppresses person-related keypoints and substantially reduces privacy leakage from inverted features, while preserving or improving matching in scenes with human distractors.
- Person Avoidance in Keypoint Detection: Misanthrope consistently avoids detecting keypoints on people, unlike DeDoDe, across PascalVOC2012 examples.The evaluation uses 888 images containing at least 1% person-labeled pixels.
- Person Avoidance in Keypoint Detection: Portrait-like scenes remain a failure case: Misanthrope assigns approximately 60% of keypoints to people versus 94% for DeDoDe.In these cases, it tends to avoid faces and place keypoints on clothing instead.
- Person Avoidance in Keypoint Detection: 2% to 3% of Misanthrope’s keypoints fall on people across extraction thresholds, a more than fivefold reduction versus DeDoDe.At 20,000 keypoints, this still corresponds to roughly 600 person keypoints.
- Person Detection and Re-Identification in Reconstructed Images: At LKT, Misanthrope reduces reconstructed-image person detection recall to 2.74%, compared with 63.44% for DeDoDe and 93.41% for original images.At HKT, recall remains lower for Misanthrope, but the gap narrows to 58.68% versus 72.29% for DeDoDe.
- Person Detection and Re-Identification in Reconstructed Images: At LKT with ground-truth boxes, Misanthrope reduces Rank-1 re-identification accuracy to 1.94%, versus 28.25% for DeDoDe and 61.89% for original images.With YOLO-predicted boxes, Misanthrope reaches 1.04% Rank-1 accuracy.
- Matching Performance: On Phototourism, Misanthrope outperforms other models in seven of nine scenes, but performs poorly on Lincoln Memorial and Mount Rushmore because it avoids keypoints on statues interpreted as people.Wild-SLAM evaluates matching with 2,000 keypoints per image and essential-matrix estimation from matched features.
5 Conclusion
Misanthrope is a privacy-aware keypoint detector that uses self-distillation to avoid detecting features on people. It suppresses privacy-sensitive data at the source while preserving matching performance in scenes containing people.
- Misanthrope uses self-distillation to avoid detecting features on people and suppress privacy-sensitive data at the source.
- Misanthrope substantially reduces person detection and re-identification from reconstructed images in feature inversion experiments.
- In scenes containing people, Misanthrope matches or outperforms existing detectors by avoiding spurious correspondences on human regions.
- The framework can in principle be applied to any keypoint detector and extended to suppress arbitrary semantic categories.