Source-linked AI summary
Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, Qi Tian
TL;DR
Label-insufficient learning can reduce prediction diversity when entropy minimization favors majority categories. The paper analyzes batch output matrices and proposes Batch Nuclear-norm Maximization (BNM), which improves discriminability and diversity and performs well across semi-supervised learning, domain adaptation, and open domain recognition.
Problem
Label-insufficient learning can degrade decisions near dense boundaries, while entropy minimization can reduce prediction diversity by favoring majority categories.
Method
BNM maximizes the nuclear norm of a batch classification-output matrix, using its relationship to the Frobenius norm and matrix rank to improve discriminability and diversity.
Results
BNM performs well in semi-supervised learning, domain adaptation, and open domain recognition, cooperates with existing methods, and can outperform entropy minimization significantly under rich domain discrepancy.
Takeaways & Limitations
BNM is suitable for classification tasks under label-insufficient situations and can maintain minority-class prediction possibilities while improving batch-level discriminability and diversity.
Takeaways & Limitations
BNM may sacrifice some majority-class hit rate and encourage minority-class assignments, with classification loss needed to penalize wrongly encouraged diversity.
Abstract
from arXiv · showhide
The learning of the deep networks largely relies on the data with human-annotated labels. In some label insufficient situations, the performance degrades on the decision boundary with high data density. A common solution is to directly minimize the Shannon Entropy, but the side effect caused by entropy minimization, i.e., reduction of the prediction diversity, is mostly ignored. To address this issue, we reinvestigate the structure of classification output matrix of a randomly selected data batch. We find by theoretical analysis that the prediction discriminability and diversity could be separately measured by the Frobenius-norm and rank of the batch output matrix. Besides, the nuclear-norm is an upperbound of the Frobenius-norm, and a convex approximation of the matrix rank. Accordingly, to improve both discriminability and diversity, we propose Batch Nuclear-norm Maximization (BNM) on the output matrix. BNM could boost the learning under typical label insufficient learning scenarios, such as semi-supervised learning, domain adaptation and open domain recognition. On these tasks, extensive experimental results show that BNM outperforms competitors and works well with existing well-known methods. The code is available at https://github.com/cuishuhao/BNM.
1. Introduction
Label-insufficient learning can produce ambiguous predictions, while entropy minimization may reduce prediction diversity by favoring majority categories. The paper proposes BNM, which uses batch output-matrix structure to enhance both discriminability and diversity, with promising results across three tasks.
- Deep models rely on large labeled datasets, and label insufficiency can cause ambiguous or spurious predictions under train–test discrepancy.
- Entropy minimization improves discriminability near dense decision boundaries but can reduce prediction diversity by pushing samples toward majority categories.
- BNM maximizes the nuclear-norm of the batch output matrix, leveraging its Frobenius-norm upper bound and convex approximation of matrix rank.
- The paper theoretically links prediction discriminability to the Frobenius-norm and prediction diversity to the rank of a batch classification-output matrix.
- BNM achieves promising performance in semi-supervised learning, domain adaptation, and open domain recognition, and works with existing methods.
2. Related Work
Related work addresses label-insufficient classification through entropy minimization, imbalanced learning, pseudo-labels, and matrix-based methods. BNM instead maximizes nuclear-norm to promote prediction diversity without requiring prior category-distribution knowledge.
- Semi-supervised learning, domain adaptation, and open domain recognition address different forms of label insufficiency involving unlabeled or unseen categories.
- Entropy-based methods are widely used to reduce uncertainty and obtain more reliable predictions near dense decision boundaries.
- Imbalanced-learning methods maintain minority-category predictions but require prior knowledge of category distributions, whereas pseudo-label approaches estimate that distribution.
- Matrix-analysis methods often minimize nuclear-norm to suppress extra components, while this paper maximizes it to seek prediction diversity.
- Unlike feature-focused BSP, BNM analyzes the batch classification-response matrix to increase both prediction discriminability and diversity.
3. Method
The method analyzes a batch classification output matrix to separately capture prediction discriminability and diversity, then maximizes its nuclear norm to improve both. BNM addresses entropy minimization’s tendency to favor majority categories while remaining computationally practical.
- Batch output matrix: The batch output matrix A ∈ R^B×C represents predictions for B randomly selected unlabeled samples across C categories.Each row corresponds to a sample prediction, and the rows satisfy the classification-response constraints described in the method.
- Measuring discriminability: Prediction discriminability is measured by the Frobenius norm, whose maximum coincides with entropy’s minimum and yields fully determined predictions.The paper establishes strict opposite monotonicity between H(A) and ∥A∥F, so maximizing ∥A∥F enhances discriminability.
- Entropy-minimization trade-off: Entropy or Frobenius-norm maximization can push ambiguous samples toward majority categories, reducing prediction diversity and potentially harming overall accuracy.BNM instead seeks to preserve predictions for minority categories in batches with imbalanced category representation.
- Measuring diversity: Prediction diversity is approximated by rank(A), because linearly independent prediction vectors correspond approximately to different predicted categories when ∥A∥F is near its upper bound.The maximum possible rank is min(B, C); when B ≥ C, it reaches C.
- Batch Nuclear-norm Maximization: The nuclear norm is used as a convex approximation of rank and is linked to the Frobenius norm, so maximizing it promotes both diversity and discriminability.The resulting objective is Batch Nuclear-norm Maximization, applied to the classification responses of a randomly selected batch.
- Toy example: In a toy batch with three zebras and one goat, Entropy Minimization and BNM reach entropy 0.50, while BNM better preserves the minority-class prediction.The example contrasts majority-category assignment under direct entropy minimization with BNM’s tendency to assign the infrequent correct label.
- Computational cost: Singular-value decomposition costs O(min(B^2C, BC^2)), and the small batch size makes the nuclear-norm computation nearly negligible during deep-network training.The computational cost is assessed for A ∈ R^B×C.
4. Experiments
Experiments evaluate BNM across semi-supervised learning, domain adaptation, and unsupervised open domain recognition, finding consistent gains and improved prediction diversity. BNM also complements existing methods and is especially effective on difficult domain-adaptation tasks and open-domain recognition.
- Experiments cover semi-supervised learning, unsupervised domain adaptation, and unsupervised open domain recognition across CIFAR-100, Office-31, Office-Home, and I2AwA.The study reports reproduced and previously published results under the listed experimental settings.
- 4.1. Semi-supervised Learning: BNM outperforms entropy minimization in all reported semi-supervised learning cases and cooperates effectively with existing SSL methods.Its improvement is more significant when combined with VAT than when applied to a simple pretrained ResNet alone.
- 4.2. Domain Adaptation: BNM substantially improves average domain-adaptation performance over entropy-based and popular alignment-based methods on Office-31 and Office-Home.The gains are particularly evident on difficult tasks with relatively low baseline accuracy.
- 4.2. Domain Adaptation: BNM produces higher diversity ratios than EntMin on Office-Home transfers Ar → Cl and Ar → Pr.The difference is larger on Ar → Cl; the paper attributes the smaller Ar → Pr gap to fewer samples near its decision boundary.
- 4.3. Unsupervised Open Domain Recognition: On I2AwA, BNM improves accuracy over zGCN by 11.4% on known categories, 19.0% on unknown categories, and 13.3% overall.It also reports a 15.2% average improvement over zGCN and a 4.8% improvement over UODTN.
- 4.3. Unsupervised Open Domain Recognition: During I2AwA training, BNM outperforms EntMin and Balance on all, known-category, and unknown-category accuracy while preserving minority-category prediction ratios.Balance also protects minority-category ratios, but its accuracy is lower and its results are less stable.
- 4.4. Discussion: Across the three tasks, the paper concludes that BNM works well in label-insufficient settings and surpasses entropy minimization especially with rich domain discrepancy and imbalanced categories.The discussion presents this as the overarching applicability pattern across the experiments.
5. Conclusion
The paper concludes that BNM maximizes batch nuclear norm to improve prediction discriminability and diversity across several label-insufficient classification settings, while identifying broader applications as future work.
- BNM maximizes the batch nuclear-norm to ensure higher prediction discriminability and diversity.Nuclear-norm is described as both an upperbound of the Frobenius-norm and a convex approximation of matrix rank.
- Prediction discriminability and diversity are separately represented by the Frobenius-norm and rank of the batch output matrix.
- BNM is suitable for classification under semi-supervised learning, domain adaptation, and open domain recognition scenarios.
- The authors identify the effect of BNM on other settings and tasks as future work.