Source-linked AI summary
Class Rectification Hard Mining for Imbalanced Deep Learning
Qi Dong, Shaogang Gong, Xiatian Zhu
TL;DR
Large-scale facial and clothing attribute recognition is challenged by severe class imbalance and subtle distinctions between fine-grained attributes. The paper introduces an end-to-end Class Rectification Loss framework with batch-wise minority-class hard mining, outperforming competing methods on CelebA and X-Domain. Its hard-mining procedure assumes minority classes have at least two samples per batch.
Problem
Large-scale person-attribute datasets have imbalanced class distributions and subtle visual differences, limiting reliable recognition of sparsely labelled minority classes.
Method
Class Rectification Loss jointly learns features and multi-label classifiers while incrementally mining hard positives and hard negatives from minority classes.
Results
CRL outperforms competing attribute-recognition and imbalanced-learning methods, including LMLE by 2% in average accuracy on CelebA.
Takeaways & Limitations
Batch-wise minority-class hard mining with loss rectification improves recognition across imbalanced facial and clothing attribute benchmarks.
Takeaways & Limitations
The mining strategy considers only minority classes with at least two sample images in each training batch.
Abstract
from arXiv · showhide
Recognising detailed facial or clothing attributes in images of people is a challenging task for computer vision, especially when the training data are both in very large scale and extremely imbalanced among different attribute classes. To address this problem, we formulate a novel scheme for batch incremental hard sample mining of minority attribute classes from imbalanced large scale training data. We develop an end-to-end deep learning framework capable of avoiding the dominant effect of majority classes by discovering sparsely sampled boundaries of minority classes. This is made possible by introducing a Class Rectification Loss (CRL) regularising algorithm. We demonstrate the advantages and scalability of CRL over existing state-of-the-art attribute recognition and imbalanced data learning models on two large scale imbalanced benchmark datasets, the CelebA facial attribute dataset and the X-Domain clothing attribute dataset.
1. Introduction
Person-attribute recognition is difficult because large-scale clothing and facial datasets are highly imbalanced and contain subtle fine-grained distinctions. The paper proposes end-to-end batch-wise hard mining with Class Rectification Loss to improve minority-class learning.
- Power-law attribute distributions bias models toward well-labelled majority classes and hurt sparsely labelled minority classes.This imbalance is reported for both clothing and facial attributes.
- Subtle visual differences between attributes such as “Woollen-Coat” and “Cotton-Coat” further complicate recognition.The paper notes that similar visual appearances require extensive balanced training data.
- Re-sampling can introduce noise, overfit, or discard valuable data, while classical methods generally rely on hand-crafted features.Deep learning can learn more expressive representations from large imagery pools but is especially vulnerable to imbalance.
- The proposed model jointly learns end-to-end representations and multi-label classifiers using batch-wise incremental hard positives and hard negatives from minority classes.Class Rectification Loss uses these samples with batch-balanced feature updates for incremental imbalanced learning.
2. Class Rectification Deep Learning
The paper addresses imbalanced multi-label person-attribute recognition by mining minority-class hard samples within training batches and adding Class Rectification Loss to standard deep CNN learning. CRL uses hard positives and negatives to refine minority-class boundaries and margins through several comparison-based formulations.
- Problem: The task jointly learns end-to-end features and classifiers for weakly labelled, multi-label facial or clothing attributes with highly unequal class frequencies.Attributes may be multivalued, such as clothing categories, or binary-valued, such as facial characteristics.
- Minority Class Identification: Minority classes are selected per batch as the classes with the smallest sample counts, collectively contributing less than half of the batch.Classes with only one or zero samples are ignored because the method requires at least two samples for triplet-loss learning.
- Minority Class Hard Mining: The method mines minority-class hard positives and hard negatives at either the class level or the individual-instance level.Class-level positives are poorly scored examples from the target class, while instance-level positives and negatives are defined using classification errors and feature-space distances.
- Minority Class Hard Mining: Within each batch, K hard positives are the lowest-scored or most distant examples, while K hard negatives are the highest-scored or nearest confusing examples.Selecting only these difficult samples concentrates optimisation on poor recognitions and obvious mistakes while reducing constraints and computation.
- Class Rectification Loss: Class Rectification Loss regularises conventional cross-entropy learning using hard positive and negative samples from minority classes.The stated objective is to reduce class bias by discovering sparsely sampled minority boundaries and improving margins against confusing classes.
- Class Rectification Loss: CRL is formulated through relative comparison, absolute comparison, and distribution comparison of mined minority-class samples.Relative comparison uses triplet ranking; absolute comparison constrains positive pairs to be close and negative pairs to be far, with incremental separation from overlapping majority instances.
3. Experiments
Experiments evaluate CRL on highly imbalanced facial and clothing attribute datasets using class-balanced accuracy and comparisons with existing models. CRL improves recognition, particularly for severely imbalanced attributes, while requiring substantially less training time than LMLE.
- Datasets & Performance Metric: CelebA contains 202,599 images with 40 binary facial attributes, while X-Domain contains 245,467 clothing images spanning 178 attribute values in 9 categories.Both datasets are highly imbalanced; X-Domain uses class-imbalanced accuracy, defined as mean sensitivity, for evaluation.
- Facial Attribute Evaluation: CRL(I) improves average CelebA accuracy by 5% over DeepID2 and by 2% over LMLE.Down-Sampling performs worse because it discards useful data for balancing class distributions.
- Facial Attribute Evaluation: CRL(I) gains 7% over LMLE on the 20 most imbalanced CelebA attributes, compared with 2% across all 40 attributes.On individual attributes, CRL(I) exceeds LMLE by 21% on Mustache and 26% on Blurry.
- Model Training Time: LMLE requires 388 training hours, whereas CRL(C) and CRL(I) require 27 and 35 hours, respectively, yielding an 11-times training-cost advantage.LMLE’s total comprises four rounds each of quintuplet construction and deep model learning.
- Clothing Attribute Evaluation: On X-Domain Sleeve Shape, CRL(C) and CRL(I) gain 8% and 7% over MTCT, while LMLE gains only 2%.The comparison targets clothing attributes with increasing imbalance ratios.
- Hard Mining and Loss Analysis: CRL improves both facial and clothing recognition, with CRL(I+R) performing best and instance-level mining superior in most same-loss comparisons.Most CRL variants obtain larger gains on the more imbalanced X-Domain dataset than on CelebA.
4. Conclusion
The proposed CRL framework targets very large-scale, imbalanced clothing and facial attribute recognition through batch-wise hard mining of minority classes. It outperforms existing imbalanced-learning and attribute-recognition methods while substantially reducing training time.
- CRL performs batch-wise incremental hard positive and negative mining on minority classes to regularise learning under severe class imbalance.The framework is designed for very large-scale clothing and facial attribute recognition.
- 2% average accuracy improvement on CelebA and 4% on X-Domain are reported over the state-of-the-art LMLE model.
- Over threes time faster model training time advantage is reported alongside CRL’s recognition gains.