PaperlayerView exact revision on arXiv

Source-linked AI summary

Facial Soft Biometrics for Recognition in the Wild: Recent Works, Annotation, and COTS Evaluation

Ester Gonzalez-Sosa, Julian Fierrez, Ruben Vera-Rodriguez, Fernando Alonso-Fernandez

arXiv:2210.13129v1cs.CV

TL;DR

The paper addresses limited study of soft biometrics for unconstrained person recognition by evaluating six traits manually and through two COTS on LFW, both alone and fused with deep-learning face systems. Fusion improves face-verification performance, with relative improvements up to 40% for manual estimation and 10–15% for automatic estimation.

  • Problem

    The utility of soft biometrics for unconstrained person recognition remains insufficiently studied, including their discriminative value, automatic estimation accuracy, and contribution to face verification.

    Source: §I · p. 1

  • Method

    The study assesses six manually and automatically estimated soft traits on LFW, both alone and fused with two deep-learning face verification systems.

    Source: §I · p. 1

  • Results

    Relative improvements reach up to 40% with manual soft-trait estimation and 10–15% with automatic estimation when fused with face verification systems.

    Source: §VII · p. 12

  • Takeaways & Limitations

    Soft traits can improve face verification in unconstrained scenarios, while better automatic estimation is needed to fully exploit their potential.

    Source: §VII · p. 12

    The publicly available annotated dataset was labeled by a single annotator.

    Source: §VII · p. 13

Abstract

from arXiv · show

The role of soft biometrics to enhance person recognition systems in unconstrained scenarios has not been extensively studied. Here, we explore the utility of the following modalities: gender, ethnicity, age, glasses, beard, and moustache. We consider two assumptions: 1) manual estimation of soft biometrics and 2) automatic estimation from two commercial off-the-shelf systems (COTS). All experiments are reported using the labeled faces in the wild (LFW) database. First, we study the discrimination capabilities of soft biometrics standalone. Then, experiments are carried out fusing soft biometrics with two state-of-the-art face recognition systems based on deep learning. We observe that soft biometrics is a valuable complement to the face modality in unconstrained scenarios, with relative improvements up to 40%/15% in the verification performance when using manual/automatic soft biometrics estimation. Results are reproducible as we make public our manual annotations and COTS outputs of soft biometrics over LFW, as well as the face recognition scores.

I. INTRODUCTION SOFT biometrics refer to physical and behavioral traits

The paper addresses open questions about which soft biometric traits discriminate best, how accurately they can be estimated automatically, and whether they improve unconstrained face verification. It studies six traits on LFW, assessing them alone and in fusion with deep-learning face systems.

I. INTRODUCTION SOFT biometrics refer to physical and behavioral traits

  • Soft traits may lack enough distinctiveness for highly accurate recognition on their own, yet can enhance primary biometrics under some conditions.Prior work reported benefits in surveillance settings where primary biometric traits may be degraded.Source: §I · p. 1
  • The study asks which soft biometric modalities are most discriminative, how accurately they can be estimated automatically from unconstrained faces, and how much they improve face verification.It focuses on the utility of soft traits and their fusion with state-of-the-art face verification systems.Source: §I · p. 1
  • The experiments examine six traits estimated manually and by two COTS, using LFW and two deep-learning face verification systems.LFW is selected as an unconstrained database with challenging pose, illumination, expression, and occlusion conditions.Source: §I · p. 1
  • The paper contributes a review of facial soft-biometrics recognition and an empirical assessment of gender, age, ethnicity, glasses, beard, and moustache in-the-wild.The assessment compares two COTS with publicly available manually generated ground truth.Source: §I · p. 1

II. RELATED WORKS

Prior work uses facial soft biometrics for recognition either as a set of traits or to narrow a hard-biometric search. The review notes that automatic trait errors can undermine search-space reduction.

A. Bag of Soft Biometrics

  • Prior bag-of-traits studies report LFW recognition accuracy of 85.54% with 73 attributes [22] and 85.05% with 40 attributes [33].Other reviewed work includes forensic, active-authentication, and comparative-attribute approaches; automatic extraction of comparative attributes may be difficult in practical scenarios.Source: A. Bag of Soft Biometrics · pp. 2–3

B. Search Space Reduction

  • Soft-biometric profiles can reduce watch-list comparisons by restricting the search to subjects with similar profiles [10], [41].A prior Boston Marathon example reported improved retrieval rankings as gallery size fell from 1,000,000 to fewer than 170,000 [21].Source: B. Search Space Reduction · p. 3
  • Manual gender and ethnicity pruning improved verification at FAR = 0.001 from 15.2% to 20.1%, whereas automatic pruning reduced it to 12.6% [41].The review cautions that inaccurate soft-trait estimates can leave genuine suspects outside the reduced search space.Source: B. Search Space Reduction · p. 3

C. Fusion of Soft and Hard Biometric Traits

Prior studies combine soft and hard biometrics across modalities, datasets, and estimation approaches. The review motivates further study in unconstrained settings using deep-learning face systems and automatically estimated soft traits.

C. Fusion of Soft and Hard Biometric Traits

  • Iris color fusion raised genuine acceptance rate to 99%, compared with 69% for iris and 85% for fingerprint alone [40].The study used a chimeric database and automatically extracted iris color.Source: C. Fusion of Soft and Hard Biometric… · p. 3
  • In a chimeric face-and-fingerprint study, height was more discriminative than gender or ethnicity; face improved only with height, while fingerprint improved with gender and ethnicity [18].The authors argued that face, gender, and ethnicity derive from the same facial features and are not complementary.Source: C. Fusion of Soft and Hard Biometric… · p. 3

RELATED WORK. RELATED WORKS ARE PRESENTED CHRONOLOGICALLY

  • Tome et al. [38] found that adding manually collected facial and body soft biometrics to sparse-representation face verification reduced EER from 15.96% to 7.68% in the most challenging distance scenario.A separate example used medical soft biometrics to improve a Biometrika commercial fingerprint system.Source: RELATED WORK. RELATED WORKS ARE… · pp. 4–5
  • Zhang et al. reported verification at 15.2% for the GBU Ugly partition, rising to 16.4% with gender or 15.5% with five traits, but used older methods and a dataset the authors say do not represent real unconstrained scenarios [41].The reviewed work considered both perfect knowledge and automatic estimation of five traits.Source: RELATED WORK. RELATED WORKS ARE… · p. 5
  • The paper targets gaps in evidence on soft biometrics in unconstrained scenarios, their fusion with state-of-the-art deep-learning face systems, and the impact of automatic estimation.The authors identify these as areas not extensively addressed by prior work.Source: RELATED WORK. RELATED WORKS ARE… · p. 5

III. THE LFW DATABASE AND EXPERIMENTAL PROTOCOL

The study uses LFW to evaluate soft biometrics in an unconstrained face-recognition setting. Verification follows the standard View 2 protocol and reports EER with 10-fold cross-validation.

III. THE LFW DATABASE AND EXPERIMENTAL PROTOCOL

  • LFW contains 13,233 images of 5,749 people and was designed to study unconstrained face recognition, with everyday variation in pose, lighting, expression, and other image conditions.The paper selects it as a difficult database because much prior soft-biometric research used more controlled datasets.Source: §III · p. 5
  • Verification uses the standard LFW View 2 protocol: 10 folds with 300 genuine and 300 impostor comparisons per fold, evaluated by 10-fold cross-validation using EER.Source: §III · p. 5

IV. SOFT BIOMETRICS

The section establishes six manually annotated LFW soft-biometric traits and reports their population composition and inter-trait correlations.

A. Manual Estimation

  • Manual ground truth covers gender, age, ethnicity, glasses, beard, and moustache, with annotation criteria intended to reduce single-annotator subjectivity.Age categories were assigned from physical appearance; for example, Middle Aged covered an apparent age range of 40–60.Source: A. Manual Estimation · pp. 5–6
  • LFW annotations show 81.5% Caucasian subjects, 78% male subjects, 80% with no glasses, and facial hair in 6% (beard) and 10% (moustache).The reported population also includes 67% Middle Aged, 24% Senior, and 13% Youth; the age-category percentages as supplied do not sum to 100%.Source: A. Manual Estimation · p. 6
  • Beard and moustache are highly correlated, and gender is positively correlated with both traits.Source: A. Manual Estimation · p. 6
  • Age and glasses show moderate correlation, with older people tending to wear more eyewear or sunglasses.Source: A. Manual Estimation · p. 6
  • Gender and age show a moderate negative correlation, corresponding to more older male than female subjects in the annotations.Source: A. Manual Estimation · p. 6

B. Automatic Estimation

Two commercial systems estimate overlapping but not identical soft-biometric traits. Their reported accuracies vary by trait, and age estimates also differ in consistency across multiple images of an identity.

B. Automatic Estimation

REPORTED IN TERMS OF A CC URACY FOR FACE++ AND MI CROSOFT

  • For identities with more than three images, age-estimation standard deviation is 7.10 with Face++ versus 3.70 with Microsoft.Across identities, the mean standard deviation increases as more images per identity are considered; the comparison covers roughly 901 identities.Source: REPORTED IN TERMS OF A CC URACY FOR… · p. 7, Table V · p. 7
  • Compared with [22], these COTS predict gender, moustache, and beard better, while [22] predicts age, ethnicity, and glasses better.The comparison uses the reported accuracies for gender, moustache, and beard and overall accuracies for age, ethnicity, and glasses.Source: REPORTED IN TERMS OF A CC URACY FOR… · p. 7

V. BIOME TRIC SYSTE MS

The soft-biometric verification system represents selected traits as a feature vector and combines trait-specific distances into a verification score.

A. Soft Biometric Verification

  • The feature vector contains between one and six soft-biometric modalities, with values selected from the categories defined for each trait.Gender, beard, and moustache are binary; glasses has three categories; age and ethnicity each have five.Source: A. Soft Biometric Verification · p. 7
  • Trait dissimilarities use Hamming distance for all traits except age, which uses Euclidean distance because age categories are ordered.Source: A. Soft Biometric Verification · p. 8
  • The system normalizes the sum of individual distances by the number of traits and inverts the resulting dissimilarity to compute scores.The figure caption describes the soft-biometric score as the sum of individual trait distances.Source: A. Soft Biometric Verification · p. 8, Figure 3
  • The authors characterize the approach as robust to mislabeling because age errors are most likely between adjacent ordinal groups and have a small effect after averaging.The other traits are nominal and described as straightforward to label.Source: A. Soft Biometric Verification · p. 8

B. Face Verification System

The face-verification systems use Face++ similarity scores and VGG-face features; standalone soft-biometric verification uses SFFS to select traits that minimize EER. Age ranks highest, and performance improves as more traits are included.

B. Face Verification System

  • Face++ detects faces and compares them with a similarity score, while VGG-face uses 4096-dimensional fc6 features and cosine similarity.The VGG-face model is used as a pre-trained feature extractor rather than fine-tuned.Source: B. Face Verification System · p. 8

VI. EXPERIMENTAL RESULTS

  • SFFS iteratively selects soft-biometric sets that minimize EER, and the selected set can have a specified size or the size with the best EER.The study evaluates standalone trait combinations and fusion with hard-biometric systems.Source: A. Discrimination Analysis of Soft… · p. 8, §VI · p. 8

A. Discrimination Analysis of Soft Biometrics

  • 11.8% EER was achieved with five soft-biometric traits, compared with 6.3% EER reported by prior work using 24 comparative attributes.The optimal set used all traits or all but beard; age was the most discriminative trait, followed by age and ethnicity as the best pair.Source: A. Discrimination Analysis of Soft… · pp. 8–9, Table VI · p. 9
  • Age labels can differ between images of the same person, while glasses, beard, and moustache may change over time and affect their usefulness.The figure illustrates age-reliability and within-person variation challenges for glasses and facial hair.Source: A. Discrimination Analysis of Soft… · p. 9, Figure 4 · p. 9

B. Combining Hard and Soft Biometric Systems

The paper combines soft and hard biometric evidence at score level, using manual or automatically extracted soft traits alongside two face-verification systems.

B. Combining Hard and Soft Biometric Systems

  • Soft and hard biometric scores are fused with equal weights to test whether soft information complements face-verification evidence.The two face systems are described in the preceding system section.Source: B. Combining Hard and Soft Biometric… · p. 9
  • Manual fusion evaluates SFFS-selected sets of one to six traits and compares them with individual soft-biometric configurations and both face-verification systems.The reported comparison is identified as Table VII, but the passage does not provide its cell values.Source: 1) Results With Manual Soft Biometrics · p. 9

1) Results With Manual Soft Biometrics:

Manual soft biometrics improve face verification, while qualitative examples show both rescued comparisons and cases where fusion cannot correct the face-system decision.

1) Results With Manual Soft Biometrics:

  • Up to 40% relative improvement was reported for both face-verification systems with manual soft biometrics.The best trait set was age, ethnicity, gender, moustache, and glasses; discarding sunglasses yielded additional improvements over counterpart configurations.Source: 1) Results With Manual Soft Biometrics · p. 10
  • Manual fusion can rescue genuine pairs when pose, illumination, appearance, or expression changes hinder face matching, but severe face-score errors and mismatched soft labels can still leave genuine pairs rejected.Some genuine comparisons correctly classified by face alone become incorrect after fusion, though such cases are a minority.Source: 1) Results With Manual Soft Biometrics · p. 10
  • For impostor pairs, mismatched soft profiles can correct face-system false matches, but shared soft profiles can leave impostors wrongly accepted by both systems.Resembling appearances can prompt face-system mismatches; shared soft traits may not resolve them.Source: 1) Results With Manual Soft Biometrics · p. 10
  • The next experiment uses automatically extracted soft biometrics, where trait estimates may be incorrect.The automatic system uses the COTS selected for each trait based on performance; age is represented in years rather than five manual categories.Source: 2) Results With Automated Soft… · p. 10

2) Results With Automated Soft Biometrics:

Automatically estimated soft biometrics improve face verification for the best configurations, though gains are smaller than those reported for manual estimates. The authors identify limited COTS accuracy and a restricted number of ethnicity categories as improvement opportunities.

2) Results With Automated Soft Biometrics:

  • 10.23% relative improvement for Face++ and 15.38% for VGG-face were achieved with age, gender, ethnicity, moustache, and glasses without sunglasses.The corresponding EERs changed from 12.7% to 11.4% for Face++ and from 7.8% to 6.6% for VGG-face.Source: 2) Results With Automated Soft… · p. 10, Table VIII · p. 12
  • Fusion outperformed Face++ with more than two traits and VGG-face with more than three traits.Table VIII reports the fusion results for estimated soft biometrics.Source: 2) Results With Automated Soft… · p. 10, Table VIII · p. 12
  • Limited COTS accuracy is offered as one explanation for the gap between approximately 40% manual and 15% automatic relative improvements.The passage reports accuracies of 93.01% for gender, 92.18% for glasses, 87.44% for ethnicity, 93.97% for beard, and 94.10% for moustache; the age-accuracy statement is incomplete.Source: 2) Results With Automated Soft… · p. 10

SUNGLASSES H AVE B EEN DISCARDED AS INSTANCE FROM GLASSES SOFT BIOMETRIC

  • Face++ estimates only three ethnicity instances versus five in the manual approach, and the authors suggest more ethnicity instances could further improve fusion.Microsoft estimates age with a mean standard deviation of 3.70, as stated in the passage fragment.Source: 2) Results With Automated Soft… · p. 11

VII. FINAL DISCUSSION AND CONCLUSION

On LFW, soft biometrics complemented deep face systems: manual fusion improved verification across configurations, while automatic estimates also improved performance but less. The discussion identifies estimation and annotation as open constraints.

VII. FINAL DISCUSSION AND CONCLUSION

  • The study manually labeled all LFW images for gender, age, ethnicity, glasses, beard, and moustache.Source: §VII · p. 12
  • The traits ranked from most to least discriminative were age, ethnicity, gender, moustache, glasses, and beard; the latter three were less discriminative due to intra-class variation.Source: §VII · p. 12
  • Using all six traits with Face++ and VGG-face produced the best manual-fusion results, with relative performance improvements of up to 40%.Every tested configuration using the N most discriminative traits (N from 1 to 6) improved on the standalone face systems.Source: §VII · p. 12
  • Automatic soft-biometrics fusion improved face verification, with relative performance improvements of 10–15%, but the authors note substantial room to improve estimation algorithms.The authors conclude considerable gains are possible if soft biometrics are estimated accurately.Source: §VII · p. 12
  • The authors suggest weighting traits by discrimination, permanence, or estimated accuracy, exploring alternative fusion schemes, and using more sophisticated soft-trait estimators.Source: §VII · pp. 12–13
  • The public annotated dataset has a single annotator; the authors identify diverse annotators and crowdsourcing as targets for further study.Source: §VII · p. 13
Loading 2210.13129v1…