PaperlayerView exact revision on arXiv

Source-linked AI summary

Probabilistic Face Embeddings

Yichun Shi, Anil K. Jain

arXiv:1904.09658v4cs.CV

TL;DR

Unconstrained face images may contain ambiguous or missing features, making deterministic embeddings unreliable. PFEs represent faces as distributions with uncertainty, improving recognition and indicating image quality for risk-controlled decisions.

  • Problem

    Low-quality, unconstrained faces can lack observable identity features, causing deterministic embeddings to shift and produce false or ambiguous matches.

  • Method

    PFEs represent each face as a latent-space distribution and use estimated uncertainty to weight matching, reject inputs, and aggregate face representations.

  • Results

    PFEs improve face recognition performance across different settings and provide uncertainty estimates that indicate the discriminative quality of face images.

  • Takeaways & Limitations

    PFE uncertainty can support risk-controlled recognition by identifying low-quality inputs for rejection or human assistance.

  • Takeaways & Limitations

    Under noisy inputs, deterministic embeddings cannot realistically recover exact latent features, motivating PFEs for low-quality face recognition.

Abstract

from arXiv · show

Embedding methods have achieved success in face recognition by comparing facial features in a latent semantic space. However, in a fully unconstrained face setting, the facial features learned by the embedding model could be ambiguous or may not even be present in the input face, leading to noisy representations. We propose Probabilistic Face Embeddings (PFEs), which represent each face image as a Gaussian distribution in the latent space. The mean of the distribution estimates the most likely feature values while the variance shows the uncertainty in the feature values. Probabilistic solutions can then be naturally derived for matching and fusing PFEs using the uncertainty information. Empirical evaluation on different baseline models, training datasets and benchmarks show that the proposed method can improve the face recognition performance of deterministic embeddings by converting them into PFEs. The uncertainties estimated by PFEs also serve as good indicators of the potential matching accuracy, which are important for a risk-controlled recognition system.

1. Introduction

Probabilistic Face Embeddings (PFEs) address ambiguity in unconstrained face images by representing each face as a distribution rather than a deterministic point in latent space. Their uncertainty estimates guide matching and can help reject low-quality inputs or request human assistance to avoid false recognition.

  • Motivation: Human face descriptions include confidence, allowing uncertain or indiscernible attributes to receive less attention or be rejected.Blurred eyes can make eye size uncertain, while completely corrupted images may be unidentifiable.
  • Problem: Deterministic embeddings represent each face as a latent-space point without accounting for feature ambiguity, making low-quality inputs prone to shifted embeddings and false recognition.The problem arises when expected facial features are ambiguous or absent from the image.
  • Proposed method: During matching, PFEs penalize uncertain dimensions and emphasize more confident features; for low-quality inputs, confidence can support rejection or human assistance.These mechanisms target unconstrained scenarios with varied face-image quality and observable facial attributes.
  • Proposed method: PFEs represent each input face as a latent-space distribution whose mean estimates the most likely feature values and span represents estimation uncertainty.This distributional estimation replaces point estimation for each face image.

2. Related Work

Related work studies uncertainty in deep face representations and probabilistic modeling for face matching, but prior approaches generally use model uncertainty, multiple images, or generation-oriented methods. The paper instead positions data uncertainty for recognition from individual face images as a distinct contribution and connects its framework to quality-aware pooling.

  • Uncertainty Learning in DNNs: Deep uncertainty learning distinguishes model uncertainty, which concerns model parameters, from data uncertainty, which concerns uncertainty in outputs.Model uncertainty can be reduced by collecting additional training data.
  • Uncertainty Learning in DNNs: Prior face-recognition studies leveraged model uncertainty for analyzing and learning face representations, whereas this work claims to be the first to use data uncertainty for recognition tasks.Variational Autoencoders estimate data uncertainty but primarily serve a generation purpose, and some alternatives are unsuitable when the target space is not well-defined by labels.
  • Probabilistic Face Representation Modeling: Earlier probabilistic face-representation methods model face templates or videos as distributions, subspaces, or manifolds and compare them using between-distribution measures.These methods take a set of face images rather than a single face image; KL-divergence is one example of their comparison measure.
  • Quality-aware Pooling: Quality-aware pooling methods predict the quality of each face and use normalized quality scores to weighted-pool deep CNN features into a single compact vector.The paper states that a corresponding solution can be naturally derived under its probabilistic framework.

3. Limitations of Deterministic Embeddings

Deterministic embeddings represent each input as a single latent point, so noise and degradation can shift ambiguous or low-quality faces away from their intrinsic identity features. Empirical results show this Feature Ambiguity Dilemma causes confident false accepts of impostors and false rejects of genuine cross-quality pairs in unconstrained recognition.

  • Empirical evidence: Degradation experiments show deterministic similarities can become confidently wrong as blur, occlusion, and Gaussian noise increase.The experiment manually varied Gaussian-kernel size, occlusion ratio, and noise standard deviation on LFW images.
  • Theoretical limitation: Deterministic embeddings model p(z|x) as a Dirac delta at f(x), making exact recovery of intrinsic features unrealistic when inputs contain noise.Low-quality inputs inevitably shift away from their intrinsic latent code, regardless of training-data volume.
  • Theoretical limitation: In fully unconstrained recognition, deterministic embeddings cannot reliably preserve smaller intra-class than inter-class distances for ambiguous faces.Identical images map to the same point with distance 0 even when they do not contain a face, illustrating the limitation.
  • Empirical evidence: The resulting Feature Ambiguity Dilemma produces false accepts of impostor low-quality pairs and false rejects of genuine cross-quality pairs.IJB-A pair mining confirms the hypothesized pattern and indicates a dark space where ambiguous inputs are mapped and distances are distorted.

4. Probabilistic Face Embeddings

Probabilistic Face Embeddings represent each face as a Gaussian distribution whose mean captures likely facial features and whose uncertainty captures confidence, enabling uncertainty-aware matching and fusion. A stage-wise strategy converts existing deterministic embeddings into PFEs by fixing the embedding mean and learning an uncertainty module.

  • Representation: PFEs model each face image as a latent-space distribution rather than a point estimate, using a multivariate Gaussian with diagonal covariance.The network predicts a D-dimensional mean µ and uncertainty σ for each input image, reducing representation complexity.
  • Representation: The Gaussian mean µ encodes the most likely facial features, while uncertainty σ represents model confidence along each feature dimension.A single network predicts the distribution, and the formulation is designed to convert existing deterministic embedding networks into PFEs.
  • Matching: Mutual likelihood score compares two PFE distributions and becomes a scaled, shifted negative squared Euclidean distance when all uncertainties are equal.Unlike KL-divergence, MLS is unbounded and is not a distance metric.
  • Matching: When uncertainties differ, MLS weights less uncertain dimensions more heavily and penalizes high-uncertainty dimensions, lowering the score if either input is highly uncertain.Thus, uncertainty affects matching beyond the distance between distribution means.
  • Fusion: For template fusion, conditionally independent face observations yield a Gaussian posterior with lower uncertainty, while dimension-wise minimum uncertainty reduces bias from redundant observations.The redundancy issue is illustrated by video frames, which may contain many non-independent images.
  • Training: A stage-wise training strategy fixes µ(x) to a pretrained deterministic embedding f(x) and optimizes an added uncertainty module using mutual likelihood on genuine pairs.The uncertainty module has two fully connected layers and shares the bottleneck-layer input.

5. Experiments

Experiments evaluate PFEs across seven face-recognition benchmarks, multiple deterministic base embeddings, and qualitative analyses. PFEs improve or maintain recognition performance, with larger gains on challenging and quality-diverse datasets, while uncertainty helps explain when matching improves.

  • Evaluation setup: PFEs are evaluated on seven benchmarks, including LFW, YTF, MegaFace, CFP, and IJB-A, covering standard, unconstrained, and template-based recognition settings.CFP includes frontal-profile evaluation, while IJB-A is template-based.
  • Evaluation setup: The study tests PFE conversion across base embeddings trained with Softmax+Center, Triplet, A-Softmax, and AM-Softmax losses.The experiments use a 64-layer residual network trained on CASIA-WebFace for these loss functions.
  • Recognition results: PFE improves the original representation on nearly saturated LFW and YTF while maintaining performance on biased MegaFace, which favors deterministic embeddings.MegaFace uses high-quality probes and positive pairs, with negative pairs containing at most one low-quality image.
  • Recognition results: PFE achieves more significant improvements on challenging CFP, IJB-A, and IJB-C datasets; at IJB-C FAR=0.001%, it reduces the error rate by 64%.These datasets contain greater pose, occlusion, and facial-feature ambiguity.
  • Recognition results: On IJB-S, PFE yields larger gains in surveillance-to-surveillance matching, while PFEfuse+match significantly improves all open-set protocols.High-quality frontal mugshots reduce the gap in surveillance-to-still and surveillance-to-booking protocols.
  • Qualitative analysis: Qualitative analyses show that PFE uncertainty improves separation between difficult genuine and impostor pairs, and its uncertainty variance increases from LFW to IJB-A to IJB-S.Larger performance improvements occur on datasets with more diverse image quality.

6. Risk-controlled Face Recognition

PFE provides a natural risk-control mechanism by rejecting low-confidence face images. On LFW and IJB-A, its predicted confidence better indicates potential recognition accuracy than the comparison quality criterion.

  • Risk-controlled recognition: PFE naturally supports rejecting input images when the system lacks confidence, avoiding reliance on a separate quality-assessment tool.The evaluation uses all images from the LFW and IJB-A datasets for image-level filtering.
  • Risk-controlled recognition: Predicted confidence is a better indicator of potential recognition accuracy than the compared quality criterion.Verification performance is evaluated after filtering proportions of images on LFW and IJB-A.
  • Risk-controlled recognition: The comparison uses original deterministic embeddings and cosine similarity for both methods, with a CASIA-WebFace AM-Softmax model.This setup was chosen to avoid saturated results.

7. Conclusion

The paper proposes Probabilistic Face Embeddings (PFEs), which represent face images as latent-space distributions and derive probabilistic comparison and aggregation methods. Analyses across different settings show that converting deterministic embeddings into PFEs can improve face recognition performance while avoiding feature ambiguity in unconstrained recognition.

  • Conclusion: PFEs represent face images as distributions in the latent space.This probabilistic representation is the paper’s central proposal.
  • Conclusion: The paper derives probabilistic solutions for comparing and aggregating PFEs from face images.These solutions enable matching and combination of probabilistic face representations.
  • Conclusion: PFEs avoid the feature ambiguity dilemma affecting deterministic embeddings in unconstrained face recognition.The conclusion explicitly contrasts PFEs with deterministic embeddings on this issue.
  • Conclusion: Quantitative and qualitative analyses across different settings show that converting deterministic embeddings into PFEs can improve face recognition performance.The conclusion summarizes evidence from multiple evaluation settings.

A. Proofs … A.3. Representation Fusion

The proofs derive the mutual likelihood score by evaluating the density of a difference variable at zero, establish its simplified form under constant variances, and show that fusing conditionally independent observations preserves Gaussianity while reducing variance.

  • A. Proofs: The proof of Equation (3) replaces the original integral with the density of Δz = z_i − z_j evaluated at zero.Here z_i and z_j are sampled from the respective latent distributions for the image pair.
  • A.1. Mutual Likelihood Score: Each component of Δz is obtained by subtracting two Gaussian variables, providing the basis for the mutual likelihood score derivation.The passage identifies the componentwise Gaussian subtraction used in the proof.
  • A.1. Mutual Likelihood Score: Directly solving the integral yields the same mutual likelihood score as the alternative difference-variable derivation.The proof explicitly notes equivalence between the two solution routes.
  • A.2. Property 1: When each variance component equals a constant c > 0 across images and dimensions, the mutual likelihood score simplifies to a form involving constants c1 and c2.The constants are defined as c1 = 1/(4c) and c2 = D/2 log(4πc).
  • A.3. Representation Fusion: Under conditional independence of observations given z, the fused posterior is formed from the individual likelihoods and the prior.The proof begins from p(z|x1, x2, . . . , xn+1) and uses conditional independence.
  • A.3. Representation Fusion: The posterior remains Gaussian by induction, starting from the Gaussian single-observation case and extending from one dimension to independent multivariate feature dimensions.The derivation assumes a Gaussian posterior after n observations and combines it with the next observation and prior.
  • A.3. Representation Fusion: The resulting posterior is a new Gaussian distribution with a smaller variance, and the proof then gives the solution for fusing n samples.This is the stated consequence of the induction step.

B. Implementation Details

The models were implemented in TensorFlow r1.9, with multi-GPU training for base models and single-GPU training for uncertainty modules.

  • All models were implemented using TensorFlow r1.9.
  • Base models were trained on CASIA-WebFace using two GeForce GTX 1080 Ti GPUs and on MS-Celeb-1M using four.
  • The uncertainty modules were trained using one GPU.

B.1. Data Preprocessing · B.2. Base Models · B.3. Uncertainty Module

The paper preprocesses faces through landmark-based alignment and pixel normalization, trains deterministic base models on CASIA-WebFace and MS-Celeb-1M, and adds a two-layer uncertainty module for probabilistic embeddings. The module has minimal inference overhead, while PFE matching uses mutual likelihood scores.

  • B.1. Data Preprocessing: Faces are detected with MTCNN, aligned using five landmarks, resized to 112 × 96, and RGB pixels are normalized by subtracting 127.5 and dividing by 128.
  • B.2. Base Models: CASIA-WebFace base models are trained for 28, 000 steps with SGD momentum 0.9 and learning-rate reductions after 16, 000 and 24, 000 steps.
  • B.2. Base Models: MS-Celeb-1M base models are trained for 140, 000 steps with SGD momentum 0.9 and learning-rate reductions after 80, 000 and 120, 000 steps.
  • B.3. Uncertainty Module: All uncertainty modules use the two-layer FC-BN-ReLU-FC-BN-exp architecture, with positive σ2 outputs produced by the exponential function.
  • B.3. Uncertainty Module: The uncertainty module is trained with SGD momentum 0.9, using 3, 000 steps for CASIA-WebFace and 12, 000 steps for MS-Celeb-1M.The learning rate starts at 0.001 and decreases to 0.0001 during each training schedule; both cases use batch size 256.
  • B.3. Uncertainty Module: PFE matching uses mutual likelihood score, and IJB-A results are reported as verification rates at FAR=0.1%.The tables compare PFE with deterministic “Original” embeddings for CASIA-Net and Light-CNN models trained on CASIA-WebFace.
  • B.3. Uncertainty Module: Feature extraction takes 1.5ms per image on one GPU, while cosine-similarity and mutual-likelihood matching take 4ns and 15ns, respectively.The uncertainty module has little impact on feature-extraction time, and both matching costs are negligible relative to extraction.

C. Results on Different Architectures

The method was evaluated beyond the main paper’s 64-layer CNN on CASIA-Net and 29-layer Light-CNN architectures. The appendix adjusted preprocessing and CASIA-Net’s architecture to enable a fair evaluation of the uncertainty module.

  • Preprocessing: Inputs were zero-padded and resized to each network’s target image shape.The authors argue that these preprocessing differences should not affect fair comparison because the experiment evaluates the uncertainty module rather than the networks’ original results.
  • CASIA-Net: Because the original CASIA-Net did not converge with A-Softmax or AM-Softmax, the authors added a bottleneck layer to output the embedding representation.This architectural adjustment supported evaluation of the proposed method on CASIA-Net.
Loading 1904.09658v4…