Source-linked AI summary
Live Face De-Identification in Video
Oran Gafni, Lior Wolf, Yaniv Taigman
TL;DR
The paper addresses video de-identification: changing perceived identity while preserving facial perception and natural temporal appearance. It uses a feed-forward encoder-decoder conditioned on face-recognition representations, with perceptual and blending mechanisms to generate de novo faces. The method is presented as suitable for live video, with recognition and human-identification results indicating effective identity change while maintaining visual quality.
Problem
Video de-identification seeks to change perceived identity while preserving pose, expression, illumination, and other scene dynamics without flicker or distortion.
Method
A feed-forward encoder-decoder concatenates face-recognition representations with the latent space and uses an attractor-repeller perceptual loss plus a learned mask.
Results
The method produces natural-looking video while state-of-the-art recognition fails to identify the person and humans cannot identify the generated face.
Takeaways & Limitations
The contribution provides a global video de-identification approach presented as suitable for live video without retraining for each identity or video.
Takeaways & Limitations
Training and application differ because training does not explicitly use videos and selects target images differently, although the method generalizes to unconstrained videos.
Abstract
from arXiv · showhide
We propose a method for face de-identification that enables fully automatic video modification at high frame rates. The goal is to maximally decorrelate the identity, while having the perception (pose, illumination and expression) fixed. We achieve this by a novel feed-forward encoder-decoder network architecture that is conditioned on the high-level representation of a person's facial image. The network is global, in the sense that it does not need to be retrained for a given video or for a given identity, and it creates natural looking image sequences with little distortion in time.
1. Introduction
The paper frames video de-identification as changing perceived identity while preserving pose, expression, illumination, and temporal visual quality. It proposes a feed-forward architecture that produces natural-looking de novo faces and supports automatic processing without identity-specific retraining.
- Video de-identification must change identity while preserving pose, expression, lip articulation, occlusion, illumination, shadow, and their dynamics.
- The method generates de novo faces for video rather than swapping in faces from a dataset.
- State-of-the-art face recognition fails to identify the person in the rendered video despite its visual similarity to the original.
- Humans also cannot identify the generated face, even without time constraints.
- The encoder-decoder concatenates face-recognition activations to its latent space, enabling feed-forward treatment of people unseen during training.
- An attractor-repeller perceptual loss preserves low- and mid-level appearance while distancing high-level identity features.
2. Previous Work
Previous approaches largely target still-image de-identification, face swapping, or face reanimation, with limitations in video, robustness, or identity generality. The paper distinguishes its method through video operation, de novo identities, and global feed-forward processing.
- The paper reports the first de-identification results on videos and uses face descriptors to distance output identity from input identity.
- Earlier de-identification methods mainly distort images, select distant faces, or average and fuse faces from existing datasets.
- Prior methods often produced low-resolution or monochrome results and lacked robustness across poses, illumination, and facial structures.
- Face swapping replaces one person’s face with another’s and commonly operates in pixel space while copying target-person expression.
- The face-swapping project requires retraining for each source-target pair, thousands of variable images, and similar facial structures, limiting its suitability for de-identification.
- Face reanimation changes the target video according to a source face and therefore differs from de-identification in the source scene.
3. Method
The method combines an adversarial autoencoder with a pretrained face-classifier representation and multi-image perceptual losses to preserve source appearance while distancing identity. A learned mask reconstructs the output, and the network generalizes from image-based training to unconstrained video despite a train–test discrepancy.
- Architecture: The architecture concatenates a pretrained face-classifier representation with the autoencoder latent space, enabling feed-forward processing of identities unseen during training.The target representation is encoded with a pretrained ResNet-50 face classifier, and the method does not require retraining for each identity or video.
- Output and masking: The decoder outputs both an image and a blending mask, whose values select the input, generated image, or a linear mixture for reconstruction.The mask is regularized to remain minimal and smooth, while losses apply to both raw and masked outputs.
- Train–test setting: Training uses distorted images and source-derived target representations, whereas deployment uses independent target images and is not explicitly trained on videos.Despite this discrepancy, the method generalizes to unconstrained videos.
- Model variants: The method uses separate 128 × 128 and 256 × 256 models, with the high-resolution model used for the reported experiments.The high-resolution decoder consists of six upscale–residual blocks.
- Perceptual loss: A multi-image perceptual loss preserves the source’s expression, pose, lighting, and low-level appearance while distancing the generated face from the target identity.Lower-level classifier activations enforce similarity to the input, a 7 × 7 layer enforces similarity to the target, and the 1 × 1 feature vector enforces dissimilarity to the target.
4. Experiments
Experiments show that the method produces natural-looking, temporally stable video while changing identity and preserving other facial properties. Automatic and human evaluations, comparisons, and controlled identity-distance tests support its effectiveness across challenging conditions.
- The method produces natural-looking output frames that match the input while modifying identity and maintaining other frame aspects.
- Per-frame processing handles videos with substantial motion, pose, expression, and illumination changes without motion- or instability-based distortions.
- The lower-resolution model can appear blurry, while the higher-resolution model provides more pleasing results when higher resolution is required.
- Human observers achieved 53.6% average success, close to random, when discriminating modified from unmodified videos.
- After de-identification, users’ identity judgments showed little correlation with the true identity, although they could identify gallery images before modification.
- The true positive rate on LFW drops from almost 0.99 to less than 0.04 after de-identification.
- Compared with prior methods, the approach better preserves expression, facial hair, and challenging illumination while introducing fewer artifacts.
- Increasing λ produces a gradual identity shift, providing control over the identity distance.
5. Conclusions
The paper presents video de-identification as a response to the need for effective methods that address the advances and abuse of face recognition. Its approach changes identity while minimizing changes to low- and mid-level visual features.
- The contribution targets video, including live video, and is presented as suitable for de-identification quality beyond prior literature methods.
- The method combines a face descriptor in the embedding space, a learned mask for blending, and a perceptual loss designed to obtain the desired identity change.
- Minimal image change supports video capability, with low- and mid-level features constrained to remain unchanged rather than pixel norms.
A.1. General Ablation Analysis
General ablations show that removing major components weakens de-identification, image similarity, or visual quality. The full method maintains image similarity while achieving identity differences comparable to or larger than most variants.
- Removing the mask produces bad face edges and poor handling of glasses occlusion.
- Applying adversarial loss only to the masked output introduces artifacts around the eye and green stripes near the mouth.
- The mask regularization ablation links omitted mask-norm regularization to occlusion distortions and excessive rendered face regions, while omitted derivative regularization yields high-frequency mask patterns.
- Without gradual λ increase, the output collapses into an unnatural blurred face.
- Adding a lower-resolution output yields weak de-identification, checkerboard patterns, and poor occlusion handling.
- The full method maintains image similarity and achieves identity differences similar to or larger than other variants except the mask-less variant.
A.2. Face-Descriptor Ablation Analysis
The face descriptor is necessary for high-quality rendered identities. Removing its concatenation from the latent representation produces more artifacts and lower-quality results.
- Removing the face descriptor from the latent embedding produces lower-quality results with noticeable artifacts in the rendered identity.
A.3. Mask Regularization Ablation Analysis
Mask regularization controls how much generated content is blended into the source and suppresses artifacts and high-frequency patterns. The ablation compares these effects across source, proposed, and variant outputs.
- Mask Regularization Ablation Analysis: α4 keeps the mask minimal, blending maximal regions from the source and reducing artifacts from excessive generated content.This is observed in the second ablation row.
- Mask Regularization Ablation Analysis: α5 keeps the mask smooth by penalizing mask derivatives, reducing high-frequency patterns in the output.This effect is observed in the third ablation row.
- Mask Regularization Ablation Analysis: Figure 12 compares the source image, the proposed result, and variants produced by the ablation study.
- Mask Regularization Ablation Analysis: Figure 13 plots mean pixel-level distance against mean ID distance, with low pixel-level distance and high ID distance preferred.
B. Additional Comparison with Previous Methods
The comparison with [44] examines multiple de-identification levels and shows a trade-off between identity recognizability and facial naturalness. The proposed method also produces stronger de-identification and higher-resolution outputs than the baseline.
- Additional Comparison with Previous Methods: Lower recognizability in [44] results is associated with less natural faces.
- Additional Comparison with Previous Methods: The proposed model provides much stronger de-identification, typically ranking in the thousands among 54,000 persons.
- Additional Comparison with Previous Methods: The baseline produces 64 × 64 outputs, whereas the proposed model produces 256 × 256 outputs.
- Additional Comparison with Previous Methods: The [44] comparison includes all generated faces across models offering different de-identification levels, with recognition rates reported in Table 6.The paper’s included sample was the first [44] output achieving below 50% recognition by an automatic face-recognition algorithm.