Striking the Right Balance with Uncertainty

Salman Khan; Munawar Hayat; Waqas Zamir; Jianbing Shen; Ling Shao

Source-linked AI summary

Striking the Right Balance with Uncertainty

Salman Khan, Munawar Hayat, Waqas Zamir, Jianbing Shen, Ling Shao

arXiv:1901.07590v3Jan 22, 2019cs.CV

TL;DR

Imbalanced datasets make reliable boundaries difficult to learn for rare classes, while existing distributional and cost-sensitive remedies have limitations. The paper develops Bayesian-uncertainty-based class-imbalance learning at class and sample levels, modeling samples as Gaussian feature distributions and adjusting margins. The method reports significant gains across six benchmark datasets spanning several vision tasks.

Problem
Rare classes have limited training data, making unbiased classification boundaries difficult to learn, while existing data-level and cost-sensitive approaches are prone to over-fitting or require careful weight selection.
Method
The framework uses Bayesian uncertainty to enlarge margins for rare classes and difficult samples, representing each sample with a multivariate Gaussian mean and covariance.
Results
The proposed uncertainty-based technique reports significant performance gains across six benchmark datasets covering face verification, attribute prediction, digit/object classification, and skin lesion detection.
Takeaways & Limitations
Uncertainty information supports learning robust features and classifiers with more generalizable decision regions for less frequent classes and challenging samples.
Takeaways & Limitations
Cost-sensitive alternatives require difficult class-specific cost tuning, and their costs do not change learned boundaries when training samples are separable.

Abstract

from arXiv · show

Learning unbiased models on imbalanced datasets is a significant challenge. Rare classes tend to get a concentrated representation in the classification space which hampers the generalization of learned boundaries to new test examples. In this paper, we demonstrate that the Bayesian uncertainty estimates directly correlate with the rarity of classes and the difficulty level of individual samples. Subsequently, we present a novel framework for uncertainty based class imbalance learning that follows two key insights: First, classification boundaries should be extended further away from a more uncertain (rare) class to avoid overfitting and enhance its generalization. Second, each sample should be modeled as a multi-variate Gaussian distribution with a mean vector and a covariance matrix defined by the sample's uncertainty. The learned boundaries should respect not only the individual samples but also their distribution in the feature space. Our proposed approach efficiently utilizes sample and class uncertainty information to learn robust features and more generalizable classifiers. We systematically study the class imbalance problem and derive a novel loss formulation for max-margin learning based on Bayesian uncertainty measure. The proposed method shows significant performance improvements on six benchmark datasets for face verification, attribute prediction, digit/object classification and skin lesion detection.

1. Introduction

Real-world imagery produces long-tail datasets in which rare classes have too little training data for reliable classification boundaries. The paper uses Bayesian uncertainty at class and sample levels to learn more balanced representations and improve generalization.

Long-tail imagery datasets contain a few abundant classes and several rare classes, making optimal feature-space boundaries difficult to learn.
Existing imbalance methods either modify data distributions or re-weight class errors, but each has important training drawbacks.Data-level methods are prone to over-fitting, while cost-sensitive learning requires careful weight selection.
The framework integrates Bayesian uncertainty at category and individual-sample levels because rare classes have higher uncertainty and lower classifier confidence.
It expands decision regions for less frequent classes and optimizes margins according to individual-sample uncertainty.The same principle assigns larger classification regions to more uncertain rare samples or classes.

2. The Imbalance Problem

Class imbalance makes empirical-loss minimization favor boundaries biased toward minority classes, while cost-sensitive losses may not improve generalization. The paper motivates extending minority-class boundaries and enforcing uncertainty-aware margins.

2.1. Bias due to Empirical Loss Minimization: Imbalanced class frequencies can make the learned classification boundary biased toward the minority class.The bias becomes more likely as the imbalance rate increases.
2.1. Bias due to Empirical Loss Minimization: The optimal separator under hidden class distributions can induce a larger minority-class region than the empirically learned boundary.Theorem 1 attributes this discrepancy to empirical-loss minimization under high imbalance ratios.
2.2. Cost-sensitive Loss: Cost-sensitive losses re-weight minority-class errors but require difficult class-specific cost tuning and do not change separable training boundaries.The costs are usually fixed initially rather than dynamically adjusted during training.
2.2. Cost-sensitive Loss: Because cost-sensitive costs are generally absent at test time, they leave class boundaries effectively unchanged and do not enhance the learned model’s generalization capability.The paper relates this limitation to the distinction between empirical and test-set distributions.
2.2. Cost-sensitive Loss: The proposed direction combines uncertainty-based minority-boundary expansion with margin constraints for less represented classes.The framework uses Bayesian uncertainty estimates at category and individual-sample levels.

3. Bayesian Uncertainty Estimates

The paper uses Bayesian uncertainty estimates to connect prediction confidence with class representation and obtain uncertainty from dropout-based deep CNNs. Rare classes are associated with higher uncertainty, while output variance provides an uncertainty estimate.

Under-represented classes produce higher uncertainty and wider confidence intervals, whereas well-represented classes produce lower uncertainty and compact intervals.
Deep CNNs with dropout provide Bayesian uncertainty estimates by approximating a Gaussian process.
The Bayesian formulation treats the Gaussian process as a prior distribution over functions updated according to observed labels.
At inference, multiple model configurations generate outputs whose expectation forms the prediction and whose variance estimates uncertainty.
Model precision and the class-count identity matrix contribute to the uncertainty formulation.

4. Uncertainty based Max-margin Learning

This section develops uncertainty-based max-margin learning from the softmax formulation. It imposes stricter margins on uncertain or rare classes and converts the resulting relations into differentiable forms for optimization.

The softmax loss includes the final fully connected layer and uses the dot product between class weights and features.
Rare classes receive stricter separation constraints because class uncertainty is inversely proportional to training frequency.
The method imposes a larger margin on more uncertain classes to improve generalization performance.
The margin-based formulation uses a continuous, monotonically decreasing function ψ(·) over the angle range [0, π].
Chebyshev-polynomial expansions replace cos(mα_j) so the max-margin loss can be differentiated and optimized by backpropagation.

5. Sample-level Uncertainty Modeling

The paper models each sample's feature representation as a multivariate Gaussian to quantify misclassification probability from uncertainty. These probability estimates re-weight the loss and impose stronger penalties when uncertainty is higher.

Each sample is represented by a multivariate Gaussian with feature mean μ_f and covariance Σ_f.
Sample-level uncertainty is incorporated because examples within the same class can have different difficulty levels.
The method introduces a simpler error function to approximate the otherwise intractable misclassification probability.
Because the error function is linear in Gaussian features, its error distribution is univariate Gaussian with statistics determined by μ_f and Σ_f.
The misclassification probability is obtained from the error distribution using the condition E(f) > 0 and a complementary cumulative distribution function.
Probability estimates re-weight loss values, and the modified ψ(·) function produces an improved uncertainty-aware loss.
The loss applies a stricter margin penalty as uncertainty increases and reduces to ordinary softmax when uncertainty is zero and m = 1.

6. Experiments

Experiments evaluate the uncertainty-based approach across face verification, skin lesion detection, digit and attribute classification, and imbalanced CIFAR-10. Results report competitive face verification, gains in skin lesion detection, and stronger performance across diverse imbalance-aware metrics.

6.1 Face Verification: Face verification uses VGG2-trained models evaluated on LFW, YTF, AgeDB, and CFP datasets.The study compares 20 recent methods on LFW and YTF and reports additional CFP-FP and AgeDB-30 evaluations.
6.1 Face Verification: The approach achieves competitive performance on saturated LFW and YTF face-verification benchmarks.Some comparison methods use as many as 200M images or ensembles of 25 models.
6.2 Results and Comparisons: 15.5% and 14.3% absolute gains over a cost-sensitive CNN are reported for skin lesion Experiments 1 and 2.The experiments use five and ten classes, respectively, with three-fold cross-validation.
6.2 Results and Comparisons: The method outperforms other imbalance-learning approaches on imbalanced MNIST and reports results separately from balanced-split comparisons.The balanced and imbalanced settings are not directly comparable because one comparison group uses approximately 45% less training data.
6.2 Results and Comparisons: CelebA attribute prediction is evaluated against nine recent methods despite majority-to-minority ratios reaching 1:43.The compared methods include both general approaches and methods specifically focused on class imbalance learning.
6.2 Results and Comparisons: 80.6 accuracy is reported for This Paper on imbalanced CIFAR-10, alongside 80.8±6.1, 80.6±9.3, 80.4±6.3, 88.7±5.2, and 77.6±9.6 across the reported metrics.The method is compared with data-level and algorithm-level imbalance-removal techniques.
6.2 Results and Comparisons: Recall and F1 improve by 3.9 and 4.5, respectively, while individual-class score deviations are lower than those of other techniques.These metrics give equal importance to rare classes and are presented as suitable for imbalanced-learning scenarios.
6.2 Results and Comparisons: Uncertainty scores improve performance across the evaluated margin-based losses and help distinguish difficult classes.The comparison uses identical features and substitutes uncertainty estimates for the original margin parameter m.

7. Conclusion

The paper links Bayesian uncertainty to both sample difficulty and class scarcity, then uses that information to enlarge classification regions for rare and challenging examples. Across several application domains, the approach reports significant performance gains and improved classifier generalization.

7. Conclusion: Bayesian uncertainty estimates are associated with individual-sample difficulty and under-represented-class scarcity.The conclusion states that classifier confidence levels directly relate to both factors.
7. Conclusion: The method uses uncertainty to enforce larger classification regions for rare classes and challenging training samples.This design is intended to improve generalization of the learned classifier to new samples from less frequent classes.
7. Conclusion: Significant performance gains are reported across face verification, attribute prediction, object and digit classification, and skin lesion detection.The conclusion summarizes results across several datasets and application domains.