Source-linked AI summary
Patch-based Probabilistic Image Quality Assessment for Face Selection and Improved Video-based Face Recognition
Yongkang Wong, Shaokang Chen, Sandra Mau, Conrad Sanderson, Brian C. Lovell
TL;DR
Video face recognition must select useful images despite blur, pose, illumination, shadow, and alignment variation, because poor-quality frames can degrade recognition. The paper proposes a patch-based probabilistic quality score that combines these factors, and reports better-selected subsets and improved verification performance across still-image and surveillance-video evaluations. Its scope is limited by reduced sensitivity to subtle illumination and minor expression changes.
Problem
Surveillance face recognition faces simultaneous variation in pose, illumination, blur, focus, and alignment, while existing selection techniques cannot handle all these factors together.
Method
A patch-based algorithm scores each face by its probabilistic similarity to an ideal-face model, jointly reflecting pose, shadows, sharpness, and alignment errors.
Results
The method identified the best overall faces on FERET and PIE and produced better subsets than prior methods with considerably improved verification accuracy on ChokePoint videos.
Takeaways & Limitations
Quality-based subset selection can outperform using all captured faces and improve video face verification when retaining only top-ranked images.
Takeaways & Limitations
The method is unlikely to detect subtle illumination changes or minor expression variations because of its illumination normalization and reliance on low-frequency information.
Abstract
from arXiv · showhide
In video based face recognition, face images are typically captured over multiple frames in uncontrolled conditions, where head pose, illumination, shadowing, motion blur and focus change over the sequence. Additionally, inaccuracies in face localisation can also introduce scale and alignment variations. Using all face images, including images of poor quality, can actually degrade face recognition performance. While one solution it to use only the "best" subset of images, current face selection techniques are incapable of simultaneously handling all of the abovementioned issues. We propose an efficient patch-based face image quality assessment algorithm which quantifies the similarity of a face image to a probabilistic face model, representing an "ideal" face. Image characteristics that affect recognition are taken into account, including variations in geometric alignment (shift, rotation and scale), sharpness, head pose and cast shadows. Experiments on FERET and PIE datasets show that the proposed algorithm is able to identify images which are simultaneously the most frontal, aligned, sharp and well illuminated. Further experiments on a new video surveillance dataset (termed ChokePoint) show that the proposed method provides better face subsets than existing face selection techniques, leading to significant improvements in recognition accuracy.
1. Introduction
Surveillance face recognition is degraded by multiple interacting image-quality problems, while existing selection methods struggle to combine them into one robust score. The proposed patch-based method addresses these issues with a single quality score reflecting several recognition-relevant characteristics.
- Surveillance imagery suffers from low resolution, blur, pose variation, low contrast, and strong illumination effects such as cast shadows.
- Outlier removal can fail when most sequence images are poor, because genuinely good images may then be classified as outliers.
- Fusion approaches can fail when one component classifier, such as pose estimation, depends on facial localisation that is unreliable in low-resolution CCTV footage.
- Existing quality assessment methods measure properties separately, use multiple classifiers, or learn system-specific fusion models, complicating scoring, efficiency, or portability.
- The proposed patch-based algorithm produces one score without fusion, jointly reflecting alignment errors, pose, shadowing, and sharpness without requiring facial-feature localisation.
2. Probabilistic Face Quality Assessment
The algorithm converts a face into normalized overlapping-patch features, evaluates each patch against a location-specific probabilistic model of ideal faces, and integrates those probabilities into one quality score.
- The pipeline performs pixel normalization, overlapping-patch extraction and normalization, patch-level feature extraction, local probability calculation, and score integration.
- A logarithmic transform reduces dynamic range, amplifying low intensities and compressing high intensities to reduce differences between skin tones.
- Each normalized patch is represented by top low-frequency DCT components that retain generic facial textures while reflecting changes from shadows, pose, and alignment.
- Each patch feature is assigned a location-specific probability using a Gaussian model trained on aligned frontal faces with frontal illumination and neutral expression, although test faces need not be aligned.
- Assuming location independence, the algorithm integrates local probabilities into an overall score representing similarity to an ideal face; higher scores indicate better quality.
3. Face Datasets
The study evaluates face-quality selection on FERET and PIE still images, then uses the ChokePoint surveillance dataset for video-to-video verification experiments.
- Still image datasets: FERET and PIE assess selection of images with desired alignment, sharpness, pose, and illumination characteristics against existing methods.FERET supplies simulated blur and alignment errors, while dedicated FERET and PIE subsets provide pose and cast-shadow variations.
- Still image datasets: FERET alignment experiments vary horizontal and vertical shifts, in-plane rotation, and scale, while sharpness is simulated by downscaling and rescaling images.The tested shifts range from 0 to ±8 pixels, rotations from 0° to ±30°, and scaling factors from 0.7 to 1.3.
- Still image datasets: Dedicated FERET and PIE subsets provide out-of-plane pose rotations, with FERET spanning up to ±60° and PIE up to ±67.5°.These subsets isolate pose variation for evaluating quality-based image selection.
- Surveillance videos: ChokePoint: ChokePoint records people walking through portals with three cameras, producing face sequences that vary in illumination, pose, sharpness, and localisation alignment.The three-camera arrangement makes a near-frontal face set likely for one camera; the dataset contains 48 sequences and 64,204 face images from 54 subjects.
- Surveillance videos: ChokePoint: ChokePoint evaluation uses a video-to-video verification protocol that alternates development and evaluation groups, with sequences serving as galleries and probes.Experiments select frontal-view cameras and report the average verification rate.
4. Experiments on Still Images
Still-image experiments tested whether the proposed quality measure can select faces that are well aligned, sharp, frontal, and minimally shadowed. It generally performed best or near-best across alignment and sharpness variations, while remaining competitive for pose and cast-shadow selection.
- Experimental setup: The experiments compared the proposed method with Asym shrp, Gabor asym, and DFFS on alignment, sharpness, pose, and cast-shadow selection.FERET and PIE subsets were used, with the FERET ‘fa’ subset training the proposed models and selecting the rejection threshold.
- Alignment, scale, and sharpness: The proposed method achieved the best average performance across shift, rotation, scale, and sharpness variations.It was consistently best or near-best across most variations, but performed less well on cast-shadowed images with scale changes.
- Alignment, scale, and sharpness: Asymmetry-based methods could not reliably detect vertical alignment errors and scale variations, while DFFS failed to detect the sharpest images.Combining asymmetry with sharpness measurements improved Asym shrp’s handling of sharpness, but its overall performance remained poor.
- Pose variations: The proposed method ranked second on frontal-face selection for both FERET and PIE, approaching Asym shrp on PIE.On FERET, visual differences between 0° and ±15° faces likely contributed to some −15° faces being classified as frontal.
- Dataset comparison: DFFS performed best on FERET but dropped on PIE, suggesting sensitivity to overlap between FERET training and pose subjects.The authors associate this cross-dataset inconsistency with possible overtraining to the FERET training dataset.
- Cast shadows: Asym shrp ranked first for selecting frontally illuminated faces, with the proposed method a close second under cast-shadow variations.Gabor asym confused illumination subsets, while DFFS often assigned the strongest-shadow subset the highest quality.
5. Experiments on Video: Subset Selection
Video experiments evaluated quality-based subset selection on ChokePoint using multiple set-recognition pipelines. The proposed method consistently improved verification over competing selection methods, especially for small subsets, while using all faces generally was not optimal.
- Experimental setup: ChokePoint video-to-video verification used MRH and LBP features with Mutual Subspace Method classification, plus average MRH signatures for a second experiment.Face sets were ranked by quality scores and truncated to the top N images.
- Subset selection with MRH and LBP: The proposed quality measure consistently outperformed Asym shrp, Gabor asym, and DFFS across feature-extraction algorithms when N ranged from 4 to 16.The improvement was most prevalent at N = 4, indicating more accurate ranking of high-quality images.
- Average MRH signatures: Using all captured faces generally did not produce the best verification performance.This result supports selecting a quality-ranked subset rather than automatically retaining every detected face.
- Average MRH signatures: The proposed method considerably outperformed the other methods for N ≤32 and achieved its best verification performance at N = 16.Even with N = 1, it achieved relatively high verification accuracy, suggesting effective selection of a single strong face.
6. Main Findings
The paper introduces a patch-based face image quality assessment method that simultaneously handles pose, cast shadows, blur, and localisation-related alignment errors. Across still-image and video verification evaluations, it identifies higher-quality face images and improves subset-based verification, while remaining limited for subtle illumination changes.
- The proposed patch-based method simultaneously handles pose variations, cast shadows, blurriness, and alignment errors from automatic face localisation.The alignment errors include in-plane rotations and horizontal and vertical shifts.
- On FERET and PIE, the method identified images that were most frontal, well-aligned, illuminated, and sharp, without dataset-specific parameter tuning or retraining.
- On ChokePoint, selecting only the top-ranked face images considerably improved verification accuracy compared with using all available images.The method ranked each person’s face images by quality before subset selection.
- The selected subsets consistently produced higher verification accuracy than subsets generated by previous image quality assessment algorithms.
- The method assigns low-quality scores to cast-shadow images but is unlikely to detect subtler illumination variations or minor expression changes.Its illumination normalisation supports generalisation, while its use of low-frequency information limits sensitivity to expression changes.