PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Supervised Sparse Representation Based Classification for Face Recognition with Insufficient Labeled Samples

Yuan Gao, Jiayi Ma, Alan L. Yuille

arXiv:1609.03279v2cs.CV

TL;DR

The paper addresses face recognition with very few labeled samples, where linear and non-linear variations separate labeled training images from unlabeled testing images. S3RC removes linear variation with a variation dictionary and learns gallery prototypes using a semi-supervised GMM from labeled and unlabeled data. Across the reported tasks, S3RC improves recognition performance, and combinations with established variation-dictionary methods improve it further.

  • Problem

    Face recognition is challenged by scarce labeled samples and nuisance variations between labeled training and unlabeled testing faces, which existing SRC methods may fail to model.

  • Method

    S3RC removes linear variation with a variation dictionary and learns Gaussian prototype means as a gallery dictionary using semi-supervised EM on labeled and unlabeled data.

  • Results

    S3RC consistently outperforms counterpart methods on the insufficient-sample task, with accuracy around 10% higher than SSRC for 2–4 labeled samples per person.

  • Takeaways & Limitations

    S3RC can improve recognition with strictly limited labeled data and can be combined with methods that learn linear variation dictionaries for further gains.

Abstract

from arXiv · show

This paper addresses the problem of face recognition when there is only few, or even only a single, labeled examples of the face that we wish to recognize. Moreover, these examples are typically corrupted by nuisance variables, both linear (i.e., additive nuisance variables such as bad lighting, wearing of glasses) and non-linear (i.e., non-additive pixel-wise nuisance variables such as expression changes). The small number of labeled examples means that it is hard to remove these nuisance variables between the training and testing faces to obtain good recognition performance. To address the problem we propose a method called Semi-Supervised Sparse Representation based Classification (S$^3$RC). This is based on recent work on sparsity where faces are represented in terms of two dictionaries: a gallery dictionary consisting of one or more examples of each person, and a variation dictionary representing linear nuisance variables (e.g., different lighting conditions, different glasses). The main idea is that (i) we use the variation dictionary to characterize the linear nuisance variables via the sparsity framework, then (ii) prototype face images are estimated as a gallery dictionary via a Gaussian Mixture Model (GMM), with mixed labeled and unlabeled samples in a semi-supervised manner, to deal with the non-linear nuisance variations between labeled and unlabeled samples. We have done experiments with insufficient labeled samples, even when there is only a single labeled sample per person. Our results on the AR, Multi-PIE, CAS-PEAL, and LFW databases demonstrate that the proposed method is able to deliver significantly improved performance over existing methods.

I. INTRODUCTION

Face recognition becomes difficult when labeled samples are scarce and training–testing faces differ through linear and non-linear nuisance variations. S3RC addresses this by rectifying linear variations and learning gallery prototypes from labeled and unlabeled data.

  • Motivation: Shortage of labeled samples leaves SRC unable to model nuisance variables separating training and testing faces, including illumination, occlusion, expression, and resolution changes.The paper highlights settings where only one labeled image may exist, such as recognizing a terrorist from an ID photo.
  • Related limitations: Existing ESRC models additive nuisance variables with a shared variation dictionary and represents each person with a prototype gallery dictionary.This framework handles variations such as shared sunglasses but does not directly address non-additive pixel-wise changes.
  • Related limitations: Non-linear nuisance variables make prototypes difficult to obtain because specific changes such as smiling cannot be generated by simply adding a variation to a neutral image.SSRC still requires multiple labeled samples and uses averaging to form its gallery dictionary.
  • Proposed method: S3RC uses a variation dictionary to remove linear changes, then applies a GMM with semi-supervised EM to learn gallery prototypes from labeled and unlabeled samples.The learned Gaussian means become the gallery dictionary for sparse-representation classification.
  • Contributions: The proposed gallery dictionary learning method exploits unlabeled data to address non-linear variations and can operate with only one labeled sample per person.The model is designed to handle both linear and non-linear variation between labeled training and unlabeled testing samples.
  • Contributions: The gallery-learning component is complementary to existing variation-dictionary methods, allowing combinations such as S3RC with ESRC, SVDL, and RADL.The paper reports that these combinations achieve significantly improved performance on all evaluated tasks.

II. RELATED WORK

Prior SRC-based approaches address nuisance variation through sparse representations and variation dictionaries, but many require sufficient samples or rely on linear prototype construction. S3RC instead learns a precise gallery dictionary with probability-based semi-supervision.

  • SRC-based discriminative-dictionary methods require sufficient samples per subject to model uncontrolled-sample variations.
  • ESRC extends SRC with gallery and variation dictionaries, representing prototypes separately from shared linear nuisance variations.
  • S3RC uses semi-supervised EM and a GMM to learn the gallery dictionary from labeled and unlabeled data within the ESRC framework.
  • The gallery-plus-variation model represents a test image with sparse gallery and variation coefficients, then classifies it using reconstruction residuals.

B. Construct the data from each class as a Gaussian after eliminating linear variations

S3RC first removes linear nuisance variation from each sample using the gallery-plus-variation model. The normalized rectified data are then modeled classwise as Gaussian distributions to capture nonlinear variation.

  • Linear variations are removed by solving the sparse gallery-plus-variation representation for each sample.The gallery and variation dictionaries are used to represent and rectify the data.
  • The selected gallery coefficient identifies the class centroid nearest to the rectified unlabeled image.
  • Normalization removes the scale introduced by the dominant sparse coefficient before Gaussian modeling.
  • Each class is modeled with its own Gaussian noise covariance to estimate gallery prototypes more precisely from insufficient samples.

C. GMM Formulation

The rectified labeled and unlabeled images are represented with a K-component GMM whose class means serve as prototype estimates. Known labels are fixed, while unlabeled assignments remain latent.

  • A GMM models normalized rectified images from labeled and unlabeled samples to estimate each class centroid.
  • Normalized rectifications use rough labeled class centroids for labeled data and variation-dictionary-based estimates for unlabeled data.
  • The unlabeled samples receive latent class indicators, while the model contains class means, covariances, and prior probabilities.

D. Estimation of the gallery dictionary by semi-supervised EM algorithm

Semi-supervised EM estimates the GMM parameters iteratively, keeping labeled assignments fixed while estimating unlabeled responsibilities. The learned means become the gallery prototypes.

  • EM iteratively estimates latent indicators for unlabeled data and maximizes the normalized-data log likelihood.
  • During the E-step, labeled assignments remain fixed at their true labels, unlike in ordinary EM.
  • For unlabeled samples, the E-step uses expectations under the previous model to estimate class indicators.
  • The M-step substitutes estimated indicators into the likelihood and updates the model parameters using maximum-likelihood estimation.
  • Labeled samples are initialized with their class centroids, while each class covariance starts as the identity matrix.

E. Classification using the estimated gallery dictionary and SRC

The estimated gallery dictionary replaces the initial gallery in sparse representation, after which class-specific residuals determine the final label.

  • The estimated gallery dictionary P* substitutes for the original P when recomputing sparse coefficients for classification.
  • Class-specific residuals are computed by retaining only coefficients associated with each class.
  • The test image receives the label of the class with the smallest residual.

F. S3RC model for SLSPP problem with generic dataset

For the single-labeled-sample-per-person setting, S3RC initializes prototype and variation dictionaries, then iteratively refines a GMM with labeled and unlabeled data before SRC classification.

  • The SLSPP formulation uses one labeled gallery sample per subject alongside unlabeled samples and an optional labeled generic dataset.
  • Initial class centers use the sole labeled samples, while generic-data averages initialize the corresponding gallery-related quantities.
  • The algorithm computes prototype and universal linear-variation matrices, applies dimensional reduction, and normalizes columns.
  • Sparse representation estimates coefficients for unlabeled samples, rectifies linear variation, and normalizes each sample to unit ℓ2-norm.
  • EM alternates GMM E- and M-steps until convergence, then estimated means form P* for residual-based SRC labeling.

IV. RESULTS

The experiments evaluate S3RC on insufficient-labeled-sample and single-labeled-sample settings using transductive and inductive protocols.

  • The evaluation covers the insufficient training samples problem on AR, followed by the SLSPP problem on Multi-PIE and CAS-PEAL.
  • The study also redoes experiments to further investigate semi-supervised gallery dictionary learning.
  • All experiments report both transductive and inductive settings, differing in how unlabeled data are used relative to testing.

A. Performance on the insufficient training samples problem

On AR, the study compares S3RC with established methods under transductive and inductive protocols, including session-separated conditions and varying labeled-sample counts.

  • The AR subset contains 100 subjects, 26 cropped images per subject, and variations in illumination, expression, and facial occlusion.
  • Transductive experiments vary labeled images from 2–13 per subject, with combined-session and separated-session configurations.
  • Inductive experiments use the same labeled-sample strategy, splitting remaining images into unlabeled training and testing sets, including separated-session conditions.
  • Comparisons include SRC, ProCRC, DLRD SR, D2L2R2, ESRC, SSRC, and RADL, using 300 PCA components, λ = 0.005, and 20 runs.
  • Around 10% higher accuracy than SSRC is reported for S3RC with 2–4 labeled samples per person, while S3RC-RADL achieves the best performance.
  • Session-separated experiments are harder for all methods, yet S3RC shows higher outperformance in the challenging separated-session transductive condition.

1) The Multi-PIE Database :

On Multi-PIE, S3RC is evaluated for single labeled samples under controlled conditions, with transductive and inductive settings and comparisons against classical and dictionary-learning baselines. It achieves top recognition rates, particularly when expressions vary, while learned galleries better represent nonlinear changes.

  • Multi-PIE contains variations in pose, expression, and illumination across four sessions, providing a challenging face-recognition benchmark.
  • Experimental setup: The controlled single-gallery experiment compares S3RC and its SVDL- and RADL-based variants with NN, SVM, SRC, CRC, ProCRC, ESRC, and SVDL.Transductive experiments use all corresponding-session images as unlabeled testing data; inductive experiments split them into unlabeled training and testing halves.
  • Results: S3RC, S3RC-SVDL, and S3RC-RADL achieve the top recognition rates across the Multi-PIE subsets in the controlled-gallery experiments.The inductive results are comparable to the transductive results, despite using only half the corresponding-session images for unlabeled training.
  • Results: The largest improvements occur for varying expressions, where learned gallery dictionaries can better align testing images than eye-center alignment alone.The paper attributes this to gallery learning capturing expression-related alignment differences.
  • Learned galleries: With neutral labeled galleries and smiling unlabeled images, S3RC learns gallery samples containing smile attributes, especially around the mouth region.This illustrates how semi-supervised gallery learning represents nonlinear variation between labeled and unlabeled faces.
  • CAS-PEAL comparison: In complementary CAS-PEAL experiments, S3RC variants achieve the top three recognition rates except for S3RC-RADL versus RADL in the inductive setting.The exception is attributed to having only three unlabeled samples per subject in that inductive experiment.

C. The performance on the SLSPP problem using uncontrolled image as gallery

The uncontrolled-gallery Multi-PIE experiment tests S3RC when each subject has a randomly selected uncontrolled labeled image. S3RC consistently outperforms the compared baselines, while learned galleries better capture nonlinear differences such as squint-to-smile changes.

  • Experimental setup: Each Multi-PIE subject contributes one randomly selected uncontrolled labeled image from subsets excluding the unlabeled/testing subset.The experiment uses both transductive and inductive settings; inductive results average 20 replicates.
  • Results: S3RC consistently outperforms NN, SVM, SRC, CRC, and ProCRC in the uncontrolled-gallery Multi-PIE results.Overall accuracy decreases relative to controlled galleries, but the conclusions from the controlled experiment remain supported.
  • Learned galleries: Using a squint image as the sole labeled gallery input, S3RC learns galleries that better represent smile testing images in the eye and mouth regions.The paper relates this improvement to pixel-wise alignment between labeled and unlabeled/testing images.
  • LFW results: On LFW, S3RC significantly outperforms baseline methods with PCA features, although none of the methods achieves satisfactory performance in that setting.The LFW experiment uses a single randomly selected gallery image per subject and evaluates transductive and inductive learning.
  • LFW results: With 4096-dimensional VGG-face features, semi-supervised gallery dictionary learning further boosts LFW identification performance.The comparison uses NN, SVM, SRC, CRC, and ProCRC alongside the proposed method.

D. Analysis of the influence of different amounts of labeled and/or unlabeled data

The paper studies how labeled and unlabeled sample quantities affect S3RC on AR. S3RC remains effective with one unlabeled sample per subject, improves as unlabeled data increase, and retains its advantage across labeled-data amounts.

  • Experimental setup: The AR analysis varies unlabeled samples from 1 to 13 per subject and uses 2, 4, or 6 labeled samples from another session.Results are averaged over 20 runs and compared with SRC, ESRC, and SSRC under the same configurations.
  • Results: S3RC outperforms the state-of-the-art even with only 1 unlabeled sample per subject.
  • Results: S3RC accuracy increases significantly as more unlabeled samples are added, whereas state-of-the-art accuracies change little.The paper links this difference to the alternatives not using unlabeled information.
  • Results: S3RC maintains better performance than the alternatives across different amounts of labeled data.The comparison covers 2, 4, and 6 labeled samples per subject.
  • Learned galleries: Learned galleries increasingly represent smiling unlabeled faces as more unlabeled training samples are used.The paper interprets this as nonlinear pixel-wise or local alignment between neutral labeled galleries and smiling faces.

E. The performance of our method with different alignments

S3RC integrates with automatic face detection and alignment, while maintaining strong recognition performance across alignment settings. Its gallery-dictionary learning also complements variation-dictionary methods and benefits from deep features.

  • Automatic alignment: S3RC can be used after Viola-Jones detection and MRR alignment in a fully automatic recognition pipeline.The pipeline detects faces with Viola-Jones, aligns them with MRR, and classifies them using S3RC.
  • Alignment robustness: The alignment experiments indicate that S3RC’s strong performance is not dependent on a particular face alignment method.The authors explicitly evaluate robustness to different alignments and state that performance remains strong under automatic alignment.
  • Alignment robustness: S3RC and S3RC-SVDL achieve the two highest recognition rates in the automatically aligned Multi-PIE experiments.Table III reports recognition rates for Viola-Jones detection and MRR alignment, with the paired comparisons defined relative to ESRC and SVDL.
  • Complementary variation modeling: The learned gallery dictionary is complementary to methods that learn linear variation dictionaries, including ESRC, SVDL, and RADL.Combining these components produced significantly improved performance across the reported tasks.
  • Deep features: With VGG-face features on LFW, S3RC outperforms other baselines by 1.98%.The paper reports less improvement from adding deep features to SRC, CRC, ProCRC, and ESRC than to the basic nearest-neighbor classifier.
Loading 1609.03279v2…