Source-linked AI summary
Benchmarking CLIP for Zero-Shot Face and Periocular Gender Estimation
Fernando Alonso-Fernandez, Kevin Hernandez-Diaz, Jose Maria Buades, Josef Bigun
TL;DR
Gender estimation has traditionally relied on hand-crafted descriptors or supervised deep learning requiring labeled data or task-specific training; periocular estimation is also more challenging because it uses a smaller facial region. The study evaluates three pretrained CLIP backbones using male/female prompts and image-text similarity, with threshold alignment and linear SVMs as extensions. Full-face gender estimation reached 95.87% accuracy; periocular gender estimation reached 86.17% accuracy, outperforming previous Adience works by 2.8%, while threshold alignment substantially reduced zero-shot male bias.
Problem
Gender estimation has traditionally relied on hand-crafted descriptors or supervised deep learning requiring labeled data or task-specific training; periocular estimation is also more challenging because it uses a smaller facial region.
Source: §I · p. 1Method
The study evaluates three pretrained CLIP backbones using male/female prompts and image-text similarity, with threshold alignment and linear SVMs as extensions.
Source: §II · p. 1, §V · pp. 6–7Results
Full-face gender estimation reached 95.87% accuracy; periocular gender estimation reached 86.17% accuracy, outperforming previous Adience works by 2.8%, while threshold alignment substantially reduced zero-shot male bias.
Source: §V · p. 7Takeaways & Limitations
The remaining full-face/periocular performance gap suggests weaker separability of male and female periocular representations in the employed CLIP embedding space.
Source: §V · p. 7
Abstract
from arXiv · showhide
We investigate CLIP for zero-shot gender estimation from full-face and periocular images. Three CLIP backbones are evaluated on 11,299 frontal images from Adience using image-text similarity with male/female prompts, achieving 95.54% full-face accuracy without task-specific training. For periocular, zero-shot predictions are strongly biased towards males, primarily due to a misaligned decision boundary. Threshold alignment substantially reduces this bias, reaching 85.29% accuracy. Linear SVMs trained on CLIP features provide only marginal gains, with a best periocular accuracy of 86.17%, approximately 2.8% above previous Adience results in the literature. Nevertheless, the gap with full-face performance confirms the greater difficulty of periocular gender estimation
I. Introduction
Soft attributes can complement biometric recognition, and periocular images offer useful cues when the full face is unavailable, though the smaller region is more challenging. This study evaluates CLIP image-text similarity for gender estimation from full-face and periocular images, finding strong full-face performance and correcting periocular male bias through threshold alignment.
I. Introduction
- Periocular images can remain available when other facial areas are occluded, but estimating attributes from this smaller region is more challenging.Full-face images contain richer semantic information but may be unreliable under pose variation, illumination changes, or partial occlusion.Source: §I · p. 1
- Prior gender estimation used hand-crafted descriptors or supervised deep learning, while CLIP enables visual concepts to be queried through natural-language prompts.The cited passage frames foundation models as an alternative requiring little or no task-specific supervision.Source: §I · p. 1
- On Adience, across three CLIP backbones, best accuracies reach 95.87% for full-face and 86.17% for periocular images, with periocular outperforming previous Adience results by 2.8%.Source: §I · p. 1
- Earlier VLM studies mainly focused on full faces or generated text; this study instead evaluates image-text similarity for both full-face and periocular gender estimation.It evaluates three CLIP backbones on Adience using zero-shot male/female prompts.Source: §I · p. 1
II. Method
The method uses three pretrained CLIP image backbones to compare full-face or periocular image embeddings with male/female prompt embeddings. Periocular embeddings average the two eyes; zero-shot similarity classification and linear SVMs are evaluated.
II. Method
- Three pretrained CLIP backbones map images and text into a shared embedding space for zero-shot transfer through natural-language queries.The evaluated image backbones are ViT-B/16, ViT-L/14, and a ResNet-50 with 16× width scaling.Source: §II · p. 1
- Male/female prompts are defined for full-face and periocular images, and periocular image embeddings average the independently extracted left- and right-eye embeddings.The text encoder produces corresponding male and female prompt embeddings.Source: §II · pp. 1–2
- Figure 1 contrasts the CLIP architecture with its face/periocular adaptation and shows Adience images.The adaptation is shown at top-right and bottom-left, while Adience examples appear at bottom-right.Source: Figure 1 · p. 2
- Table I summarizes the characteristics of the three evaluated CLIP backbones.The backbones are ViT-B/16, ViT-L/14, and a scaled ResNet-50.Source: §II · p. 1, Table I · p. 2
- Table II lists the prompts used for face and periocular gender classification.The prompts include male and female descriptions for both image types.Source: §II · p. 1, Table II · p. 2
- Zero-shot gender classification uses the difference between male and female cosine similarities, assigning images above zero to male and others to female.Linear SVMs are also trained using CLIP similarity scores, image embeddings, or their combination as input features.Source: §II · p. 2
III. Database and Protocol
On 11,299 frontal Adience images, CLIP achieved strong full-face zero-shot gender estimation but showed a male bias on periocular images. Threshold alignment corrected much of that bias, while periocular accuracy remained below full-face performance.
III. Database and Protocol
- 11,299 frontal images from 2,284 Adience subjects were retained from 26,580 originals, with five-fold cross-validation preventing Flickr album overlap between training and test sets.The protocol uses 5,353 males and 5,946 females, aligned face crops, and left/right periocular crops.Source: §III · p. 2, FACE GENDER ESTIMATION · p. 3
FACE GENDER ESTIMATION
- 92.17%-95.54% full-face and 64.59%-75.57% periocular accuracy with averaged prompts; ViT-L/14 performs best and ResNet-50 worst.Across prompts, periocular male accuracy exceeds 95% for most prompts, while female accuracy is often below 60%; no prompt is consistently optimal across classes and backbones.Source: Figure 2 · p. 3, A. Zero-shot face and periocular… · p. 4
- Moderate downsampling causes only minor accuracy reductions; even at 0.25 scale, full-face female accuracy exceeds original-resolution periocular female accuracy, while periocular male accuracy remains above 96%.The results indicate that limited spatial resolution alone cannot explain poor zero-shot female periocular performance.Source: B. Scale analysis · p. 4, Figure 3 · p. 4
- Female periocular scores shift toward positive values and overlap more with male scores, explaining zero-threshold male misclassifications and why alignment cannot close the full-face gap.Full-face male and female scores are well separated around zero; greater periocular score overlap leaves performance lower even after correcting the shift.Source: C. Embeddings alignment · pp. 4–5, Figure 4 · p. 5
- Score alignment substantially raises female periocular accuracy from 33.92%-54.86% to 76.67%-87.64%, while male accuracy drops from 97.7%-98.97% to 75.93%-82.5%, balancing performance per prompt and class.Full-face accuracy changes little because its scores are already well aligned around zero.Source: C. Embeddings alignment · p. 5
- SVMs change full-face per-class accuracy by generally less than 0.5%; periocular gains mainly redistribute class performance, and image-only embeddings remain competitive.In the supervised setting, the text encoder is unnecessary when image embeddings alone are used.Source: D. SVM-trained face and periocular… · p. 5
- On Adience, CLIP’s original zero-shot full-face gender estimator achieves 95.54% accuracy without task-specific training; the best periocular SVM reaches 86.17%, versus the previous best 83.34%.The best full-face accuracy is 95.87%, within 2% of the state of the art.Source: E. Comparison with the state-of-the-art · p. 5
V. Conclusion
CLIP-based gender estimation was competitive on full-face images but less successful on periocular images, where zero-shot predictions showed a male bias. Threshold alignment reduced that bias, while the remaining performance gap points to weaker separation of male and female periocular representations in the evaluated CLIP space.
V. Conclusion
- 95.87% full-face accuracy (within 2% of the state of the art) and 86.17% periocular accuracy (2.8% above previous Adience works) were reported for the CLIP-based approach.Table III compares face and periocular gender-classification methods on Adience and marks the best accuracy per column.Source: Table III · p. 7, §V · p. 7
- Periocular zero-shot scoring showed strong male bias, mainly from a shifted decision boundary and overlapping class scores; threshold alignment reduced bias, while SVMs added modest gains.The reported analysis indicates that lower crop resolution alone does not explain the bias.Source: §V · p. 7
- The study evaluates CLIP for full-face and periocular gender estimation using zero-shot image-text similarity, with threshold alignment and linear SVMs as extensions.Experiments used 11,299 Adience smartphone images.Source: §V · pp. 6–7
- Image embeddings alone were sufficient for competitive SVM performance, allowing the text encoder to be discarded in supervised use.Source: §V · p. 7
- Future work will investigate alignment and prompt-optimization strategies alongside model-compression techniques to make CLIP-based soft-biometric estimation more practical for deployment.Examples include pruning, knowledge distillation, lightweight encoders, and intermediate-layer embeddings.Source: §V · p. 7