PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Imbalanced Learning for Face Recognition and Attribute Prediction

Chen Huang, Yining Li, Chen Change Loy, Xiaoou Tang

arXiv:1806.00194v2cs.CV

TL;DR

Face-analysis data are highly imbalanced, raising questions about how conventional re-sampling and cost-sensitive methods support deep representation learning. The paper systematically evaluates these schemes and proposes CLMLE, which enforces angular margins between local cluster distributions and uses a k-nearest-cluster metric. CLMLE reports improved performance across imbalanced face recognition and face attribute prediction tasks, including state-of-the-art results on face benchmarks.

  • Problem

    Face-analysis data are highly imbalanced, while the effectiveness of conventional re-sampling and cost-sensitive methods for deep representation learning had not been systematically studied.

  • Method

    CLMLE learns representations by enforcing angular margins between intra-class and inter-class cluster distributions, then evaluates them with a soft k-nearest-cluster metric.

  • Results

    CLMLE achieves state-of-the-art performance on several face-recognition datasets and superior balanced accuracy across multiple face attributes, outperforming reported comparison methods.

  • Takeaways & Limitations

    Cluster-aware local margins provide a representation-learning approach for imbalanced face recognition and attribute prediction, together with validated conventional re-sampling and cost-sensitive schemes.

  • Takeaways & Limitations

    Triplet and quintuplet approaches face sampling growth and inconsistency that can hinder convergence and learning quality; CLMLE generalizes sampling to entire clusters.

Abstract

from arXiv · show

Data for face analysis often exhibit highly-skewed class distribution, i.e., most data belong to a few majority classes, while the minority classes only contain a scarce amount of instances. To mitigate this issue, contemporary deep learning methods typically follow classic strategies such as class re-sampling or cost-sensitive training. In this paper, we conduct extensive and systematic experiments to validate the effectiveness of these classic schemes for representation learning on class-imbalanced data. We further demonstrate that more discriminative deep representation can be learned by enforcing a deep network to maintain inter-cluster margins both within and between classes. This tight constraint effectively reduces the class imbalance inherent in the local data neighborhood, thus carving much more balanced class boundaries locally. We show that it is easy to deploy angular margins between the cluster distributions on a hypersphere manifold. Such learned Cluster-based Large Margin Local Embedding (CLMLE), when combined with a simple k-nearest cluster algorithm, shows significant improvements in accuracy over existing methods on both face recognition and face attribute prediction tasks that exhibit imbalanced class distribution.

1 INTRODUCTION

Face-analysis data are naturally imbalanced, challenging deep representation learning and motivating systematic study of re-sampling, cost-sensitive learning, and cluster-based margins. CLMLE targets local class-boundary imbalance by separating within- and between-class cluster distributions, improving face recognition and attribute prediction.

  • Motivation: Face-analysis datasets naturally contain highly skewed class distributions, biasing conventional methods toward majority classes.Examples include more negative than positive face pairs and more normal-sized-nose than big-nose images.
  • Existing strategies: Re-sampling balances class priors, whereas cost-sensitive learning assigns higher misclassification costs to minority classes.These are the two principal conventional strategies discussed for deep imbalanced learning.
  • Existing strategies: These schemes have limitations: over-sampling can add noise, computation, and overfitting risk; under-sampling can discard information; cost design often relies on heuristics.The paper also notes that their implications for deep representation learning had not been systematically studied.
  • Proposed approach: CLMLE separates cluster distributions within and between classes to reduce local imbalance and form boundaries less sensitive to remaining class-sample imbalance.The method uses margins derived on a hypersphere feature space and studies re-sampling and cost-sensitive learning in this setting.
  • Results: CLMLE combines learned representations with a soft k-nearest-cluster metric and improves performance on face recognition and balanced-accuracy evaluation of multiple attributes.The reported face-recognition evaluations include small-training-data settings and comparisons against softmax, triplet, and recent imbalanced-learning methods.
  • Contribution: The expanded work generalizes LMLE from quintuplet sampling to entire clusters, adds customized re-sampling and cost-sensitive techniques, and introduces angular cluster margins.It also validates the approach on both face attribute prediction and face recognition, including an open-set scenario.

2 RELATED WORK

Prior imbalanced-learning methods mainly alter data sampling or classifier costs, but generally overlook the internal structure of imbalanced classes. CLMLE addresses this by enforcing margins between local intra-class and inter-class clusters.

  • Conventional approaches: Re-sampling changes the training distribution through under-sampling, over-sampling, or both, while cost-sensitive learning adjusts algorithmic misclassification costs.The two families aim to learn classifiers that perform equally well across classes.
  • Conventional approaches: Replication-based over-sampling can overfit and does not add minority-class information; SMOTE instead synthesizes examples by interpolating neighboring minority instances.The paper notes that synthetic border examples can still produce error-prone broadened decision regions.
  • Conventional approaches: Determining cost representations remains open, with common practices using inverse class frequencies or predefined misclassification costs.These costs are applied through methods including SVMs, decision trees, boosting, and classifier ensembles.
  • Structure-aware learning: Earlier methods generally ignore imbalanced-class data structure, while CRL regularizes hard minority classes without guaranteeing equal learning for all classes.CLMLE instead enforces large margins between intra-class and inter-class clusters to draw balanced local boundaries for every class.
  • Structure-aware learning: CLMLE uses cluster distributions to characterize class structure globally, supporting faster and better convergence than purely online methods such as CRL.The cluster separation rule applies during both training and testing with negligible clustering cost during training.
  • Deep face analysis: Existing deep face-recognition embeddings combine softmax with contrastive, center, marginal, or triplet losses, while imbalance methods modify images, features, centers, norms, or intra-class ranges.The paper positions CLMLE as complementary through its data-structure-aware local margin enforcement.

3 LEARNING DEEP REPRESENTATION FROM CLASS-IMBALANCED DATA

Existing embedding losses impose class-level margins that can mishandle multimodal, imbalanced distributions, whereas CLMLE models local cluster structure and separates entire cluster distributions on a hypersphere.

  • Challenges with Existing Embedding Methods: Triplet and A-Softmax losses assume each class is captured by a single mode, potentially causing overlap when majority classes have larger variation.Triplet loss imposes Euclidean class-level margins, while A-Softmax imposes angular class-level margins.
  • Challenges with Existing Embedding Methods: LMLE preserves local discrimination by enforcing Euclidean margins among intra- and inter-class clusters, making local boundaries less sensitive to class size.Its quintuplet constraints retain locality across same-class clusters while separating classes.
  • Challenges with Existing Embedding Methods: Quintuplet-based LMLE can suffer exponential sampling growth and inconsistent sampled constraints, hindering convergence rate and quality.These issues motivate sampling entire cluster distributions instead of individual quintuplets.
  • Cluster-based Large Margin Local Embedding: CLMLE samples all examples in contextual clusters using spherical k-means, rather than only hard quintuplet examples, to learn cluster-based local representations.The feature vectors are normalized onto the unit hypersphere, supporting cluster-based angular margins.
  • Cluster-based Large Margin Local Embedding: CLMLE makes clusters inter-distinct and intra-compact, collapsing each cluster distribution into a small region and forming margins between adjacent hypersphere regions.Marginal clusters near class bounds implicitly produce balanced classification boundaries.
  • Cluster-based Large Margin Local Embedding: CLMLE uses linear sampling O(N), avoiding triplet O(N^3) and quintuplet O(N^5) sampling complexity while converging faster and achieving better performance.The entire-distribution loss provides contextual neighborhoods and more coherent, globally consistent separation.

4 FAST EVALUATION WITH NEAREST CLUSTERS

CLMLE supports evaluation by retrieving nearby cluster representations rather than individual examples, enabling a simple non-parametric classifier that scales efficiently. The method is applied to imbalanced face-recognition and attribute-prediction datasets.

  • Fast Evaluation with Nearest Clusters: CLMLE evaluates queries with a simple k-nearest-cluster algorithm consistent with its learned feature representation.The nearest-neighbor rule is non-parametric and can extend to new classes without retraining.
  • Fast Evaluation with Nearest Clusters: For each query, the method retrieves the nearest clusters from all training classes using inner-product similarity.The predicted class is selected from the relative similarities of the retrieved clusters.
  • Fast Evaluation with Nearest Clusters: Cluster-wise search reduces the search dependence from the number of examples L to the number of clusters ⌊L/l⌋.A KD-tree further provides runtime O(L/l log(L/l)).
  • Fast Evaluation with Nearest Clusters: The evaluated settings include long-tailed face-recognition classes and 40 binary face attributes with imbalanced positive and negative samples.These distributions are shown for CASIA-WebFace and CelebA, respectively.
  • Fast Evaluation with Nearest Clusters: The cluster-based search achieves up to three orders of magnitude speedup over standard example-wise search in practice.The paper reports this result while using a simple testing algorithm to demonstrate the learned features.

5 EXPERIMENTS

Experiments evaluate CLMLE on imbalanced face recognition and attribute prediction, comparing it with established and recent imbalanced-learning methods. CLMLE achieves strong results with small training data and a single or smaller network in the reported comparisons.

  • Face Recognition: CLMLE is evaluated on LFW, YTF, and MegaFace for face verification and identification under imbalanced, including open-set, settings.The reported protocols include verification accuracy, rank-1 identification accuracy, and TAR at fixed FAR.
  • Face Recognition: 99.62% on LFW and 96.5% on YTF are achieved using 0.45M CASIA-WebFace images and a single model.This setting uses publicly available small training data and compares favorably with systems using much larger training sets or model ensembles.
  • Face Recognition: Under matched WebFace data and a shared 64-layer CNN, CLMLE outperforms softmax variants, metric-learning losses, and recent imbalanced-learning methods.The comparison includes L-Softmax, A-Softmax, LMCL, contrastive, triplet, center, augmentation, range, center-invariant, and LMLE methods.
  • Face Recognition: 76.26% rank-1 accuracy and 89.41% TAR under 10^-6 FAR are reported for MegaFace at the 1M-distractor scale.The results are reported for the top-performing methods under small and large protocols, and CLMLE outperforms LMLE and range loss in the cited comparison.
  • Face Attribute Prediction: On CelebA, CLMLE exceeds traditional re-sampling, cost-sensitive learning, LMLE, and CRL-I, and averages about 10% and 6% over MOON and AFFACT.These comparisons use balanced accuracy; MOON and AFFACT use substantially larger networks, while CLMLE does not assume a target attribute distribution.
  • Face Attribute Prediction: CLMLE remains comparable in two metrics to AttCNN and SSP without increasing network capacity.The cited figure also reports challenging highly imbalanced attributes correctly predicted by CLMLE but not SSP.
  • Ablation Study: Ablations show non-uniform cluster sampling is preferable, cost sensitivity matters more for CelebA attributes, and instance-wise kNN gives 88.59% with lower speed and slightly worse results.The cost-sensitive scheme helps because batches balanced for one multi-label attribute can remain imbalanced for others.

6 CONCLUSION

The paper validates classic imbalance strategies and proposes CLMLE, which enforces angular margins between intra-class and inter-class clusters. Combined with a simple k-nearest cluster classifier, CLMLE achieves reported state-of-the-art performance on face benchmarks.

  • Extensive experiments validate class re-sampling and cost-sensitive learning for imbalanced deep representation learning.
  • CLMLE maintains inter-cluster angular margins within and between classes to carve more balanced local class boundaries.
  • Combined with a simple k-nearest cluster classifier, CLMLE converges fast and achieves state-of-the-art performance on face recognition and attribute benchmarks.
Loading 1806.00194v2…