PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy-based Active Learning for Object Detection with Progressive Diversity Constraint

Jiaxi Wu, Jiaxin Chen, Di Huang

arXiv:2204.07965v1cs.CV

TL;DR

Object detection requires costly bounding-box annotations, while redundant instance predictions and the need to make image-level acquisition decisions from instance-level information complicate active learning. The method uses entropy-based NMS to remove redundant instances within images, then class-specific prototypes to manage intra-class and inter-class diversity across images. The method achieves state-of-the-art results on MS COCO and Pascal VOC; in the last cycle with 40% annotated images, it reaches 32.87% AP, 1.25% above uniform random sampling.

  • Problem

    Object detection requires costly bounding-box annotations, while redundant instance predictions and the need to make image-level acquisition decisions from instance-level information complicate active learning.

    Source: §1 · p. 1, §4.2 · p. 4, §4.3 · p. 5

  • Method

    The method uses entropy-based NMS to remove redundant instances within images, then class-specific prototypes to manage intra-class and inter-class diversity across images.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    The method achieves state-of-the-art results on MS COCO and Pascal VOC; in the last cycle with 40% annotated images, it reaches 32.87% AP, 1.25% above uniform random sampling.

    Source: §5.2 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    The method reaches 73.68% mAP using 7k images, versus 8k for CDAL and MIAL and 10k for Learn Loss.

    Source: §5.2 · p. 7

Abstract

from arXiv · show

Active learning is a promising alternative to alleviate the issue of high annotation cost in the computer vision tasks by consciously selecting more informative samples to label. Active learning for object detection is more challenging and existing efforts on it are relatively rare. In this paper, we propose a novel hybrid approach to address this problem, where the instance-level uncertainty and diversity are jointly considered in a bottom-up manner. To balance the computational complexity, the proposed approach is designed as a two-stage procedure. At the first stage, an Entropy-based Non-Maximum Suppression (ENMS) is presented to estimate the uncertainty of every image, which performs NMS according to the entropy in the feature space to remove predictions with redundant information gains. At the second stage, a diverse prototype (DivProto) strategy is explored to ensure the diversity across images by progressively converting it into the intra-class and inter-class diversities of the entropy-based class-specific prototypes. Extensive experiments are conducted on MS COCO and Pascal VOC, and the proposed approach achieves state of the art results and significantly outperforms the other counterparts, highlighting its superiority.

1. Introduction

Object detection depends on costly bounding-box annotations, while active-learning methods for it are less developed than those for image classification. The paper combines instance-level uncertainty and diversity in a two-stage acquisition method.

1. Introduction

  • Bounding-box annotation is expensive and time-consuming, motivating active learning to select informative images for labeling.Active learning is expected to improve accuracy with fewer annotated images than random selection.Source: §1 · p. 1
  • For object detection, image-level acquisition must account for predictions of both object category and location across varying numbers of instances.Existing detection methods leave room for improvement because diversity-based approaches underuse categorical information and uncertainty-based approaches incompletely measure sample discrepancy.Source: §1 · p. 1
  • The framework depicts detector inference on unlabeled images, entropy-based instance uncertainty, ENMS pruning, DivProto prototype selection, and oracle labeling of acquired images.The figure illustrates acquisition priority with three example images, I1, I2, and I3.Source: Figure 1 · p. 2, §1 · p. 2
  • The proposed method combines instance-level uncertainty and diversity through ENMS followed by DivProto, which forms class-specific prototypes to manage intra-class and inter-class diversity.ENMS removes redundant instances using entropy in feature space; DivProto uses prototypes to reject redundant images and acquire images from minority classes.Source: §1 · p. 2

2. Related Work

Prior active-learning work for image classification includes uncertainty-based, diversity-based, and hybrid approaches; object-detection methods adapt these ideas but retain limitations in instance-level diversity and complexity. The paper situates its approach within these two lines of work.

2.1. Active Learning on Image Classification

  • Diversity-based classification methods represent the data distribution through clustering or matrix partitioning, while Core-set [27] uses k-center approximation and CDAL uses KL divergence.These methods can be computationally inefficient on high-dimensional data.Source: §2.1 · p. 2
  • Uncertainty-based classification methods select ambiguous samples, using approaches such as Bayesian CNNs, deep ensembles and Monte-Carlo dropout, and loss prediction in Learn Loss.These approaches are efficient but can select redundant samples.Source: §2.1 · p. 2
  • Hybrid classification methods combine uncertainty and diversity scores or estimators, including BADGE, which uses gradient magnitudes and directions to model them, respectively.Other cited hybrids use the minimum score, shift emphasis from diversity to uncertainty over cycles, or re-weight scores.Source: §2.1 · pp. 2–3

2.2. Active Learning on Object Detection

  • Detection methods adapt diversity and uncertainty to instance predictions: CDAL uses spatially pooled features, Learn Loss uses image-level features, and MIAL re-weights top-K instances.Holistic representations can be too coarse, while comparing all instance pairs is costly; existing approaches also leave instance-level redundancy or class imbalance insufficiently addressed.Source: §2.2 · p. 3

3. Problem Statement

The section formulates budgeted image acquisition in cycles and explains why object-detection acquisition needs instance-level evaluation. It introduces entropy as an image-level uncertainty measure and motivates diversity constraints to address redundant instances.

3. Problem Statement

  • Each active-learning cycle infers on unlabeled images, acquires and annotates a budgeted batch, then trains and evaluates the detector on the expanded labeled set.The detector training and evaluation procedure is held fixed so acquisition methods can be compared.Source: §3 · p. 3
  • Given candidate images and a labeled subset, the algorithm acquires a batch of size b, obtains oracle labels, and adds those images to the labeled set for training.The labeled subset is initially selected at random.Source: §3 · p. 3
  • Object-detection core-set loss sums instance-level losses, but unordered encoded spatial features make the image-level loss difficult to define as Lipschitz continuous.The detector is decomposed into an encoder for spatial-position features and a predictor for bounding boxes, classes, and confidence scores.Source: §3 · p. 3
  • The basic image-level detection entropy replaces instance loss with instance entropy, sums the candidate-instance entropies, and ranks images for top-K acquisition.The confidence score represents foreground probability for a category, with its complement representing background.Source: §3 · p. 3
  • Visually similar boxes create redundancy within and across images, motivating instance-level diversity constraints that connect ENMS-based intra-image pruning with prototype-based inter-image diversity.The hierarchy further distinguishes intra-class and inter-class diversity across images.Source: §3 · p. 3, Figure 2 · p. 4

4. Method

The framework combines instance-level uncertainty and diversity in a staged process: ENMS reduces redundant instances within images, while DivProto promotes diversity across images using class-specific prototypes.

4.1. A Hybrid Framework

  • The hybrid framework combines detection entropy, ENMS, and DivProto to estimate image uncertainty and strengthen instance-level diversity within and across images.Its diversity constraints are applied progressively, separating intra-image diversity from inter-image diversity and then dividing the latter into intra-class and inter-class diversity.Source: §4.1 · p. 4

4.2. ENMS for Intra-Image Diversity

  • ENMS accumulates entropy from informative instances while removing similar same-class instances, addressing redundancy that ordinary NMS does not remove.It compares instances using feature similarity and can reduce memory cost and the imbalance in the number of instances per image through redundant-instance removal.Source: §4.2 · pp. 4–5

4.3. Diverse Prototype for Inter-Image Diversity

  • DivProto sorts unlabeled images by ENMS entropy and uses entropy-weighted, class-specific prototypes to promote diversity across images.Unlike confidence-weighted prototypes, this formulation gives uncertain instances greater contribution than instances with high classification confidence.Source: §4.3 · p. 5
  • DivProto rejects candidate images whose class-prototype similarity to the acquired set exceeds Tintra, reducing intra-class redundancy across images.The method compares prototypes rather than all instance pairs, which the paper says remarkably reduces computational complexity.Source: §4.3 · p. 5
  • To counter majority-class bias, DivProto assigns larger acquisition quotas to minority classes and updates those quotas as qualifying images are selected.The process stops balancing when the minority-class set is exhausted, then fills the remaining selection budget with other unlabeled images.Source: §4.3 · pp. 5–6

5. Experiments

Experiments on MS COCO and Pascal VOC report gains over comparison methods, with the strongest performance depending on annotation budget and cycle. Ablations and analyses examine the components, computational cost, prototype diversity, and class balance.

5.1. Experimental Settings

  • MS COCO experiments use 20%–40% labeled training subsets and acquire 5% of total images per cycle; Pascal VOC starts with 1,000 labeled images and adds 1,000 per cycle.MS COCO is evaluated with AP over IoU thresholds 0.5–0.95, while Pascal VOC uses mAP at IoU 0.5.Source: §5.1 · p. 6

5.2. Experimental Results

  • 32.87% AP at 40% annotated images is a 1.25% increase over uniform random sampling on MS COCO.The method also consistently performs best across the reported active-learning cycles.Source: §5.2 · p. 7
  • At budgets of no more than 10%, the method outperforms counterpart methods by a large margin, although detection performance remains below real-world application levels.The passage describes the gains relative to MIAL as remarkable.Source: §5.2 · p. 7
  • The method reaches 73.68% mAP using 7k images, versus 8k for CDAL and MIAL and 10k for Learn Loss, saving annotations.It does not perform better at the 2k and 3k cycles, which the authors attribute to insufficient detector training; MIAL uses unlabeled images for semi-supervised learning.Source: §5.2 · p. 7

5.3. Ablation Study

  • ENMS and DivProto separately outperform uniform sampling and entropy-only selection, while their combination further boosts accuracy through complementary diversity constraints.At the 30% MS COCO cycle, the reported best settings are α = 0.5 and β = 0.75.Source: §5.3 · p. 7
  • The acquisition strategy still outperforms uniform sampling and basic entropy with Faster R-CNN using ResNet-101, which generally improves detection performance over ResNet-50.The comparison is reported in Table 3.Source: Table 3 · p. 7, §5.3 · p. 8

5.4. Analysis

  • Progressive instance-level diversity constraints significantly reduce acquisition time relative to raw instance-pair comparison, and the method is faster than CDAL.The raw pairwise upper-bound computation has unacceptable time cost; the comparison is reported at the 20% MS COCO cycle.Source: §5.4 · p. 8, Table 4 · p. 8
  • DivProto prototypes are more representative and diverse, while ENMS and DivProto both reduce class-count standard deviation and their combination further improves performance.The prototype visualization covers six MS COCO classes; class-count statistics cover 80 classes in active-acquisition subsets.Source: §5.4 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

6. Conclusion

The paper concludes that its hybrid method combines instance-level uncertainty and diversity for object detection, with experiments on MS COCO and Pascal VOC outperforming state-of-the-art counterparts. The appendix adds component analyses and further experimental comparisons.

6. Conclusion

  • The proposed method combines instance-level uncertainty and diversity in a bottom-up manner, using ENMS and DivProto, and reports outperforming state-of-the-art results on MS COCO and Pascal VOC.ENMS estimates instance-level uncertainty; DivProto enhances intra-class and inter-class diversity through entropy-based class-specific prototypes.Source: §6 · p. 8

Appendix

  • The supplementary material analyzes DivProto components, compares with MDN, explores a state-of-the-art detector and semi-supervised learning, and adds MS COCO results.Source: Appendix · p. 11

A. Ablation Study of DivProto

DivProto combines intra-class redundancy rejection with inter-class balanced selection. Both components improve on Basic Entropy, while their combination performs especially well at two annotation percentages.

A. Ablation Study of DivProto

  • The combined DivProto strategy further promotes AP at 25% and 35% annotated percentages and remains highly competitive in other cases.Table A reports AP (%) on MS COCO using Faster R-CNN with a ResNet-50 backbone.Source: A. Ablation Study of DivProto · p. 11
  • Intra-class redundancy rejection and inter-class balanced selection each improve the Basic Entropy baseline under various annotation percentages.Source: A. Ablation Study of DivProto · p. 11
  • DivProto combines intra-class redundancy rejection and inter-class balanced selection.Source: A. Ablation Study of DivProto · p. 11

B. Comparison with the latest work MDN

The comparison with MDN focuses on acquisition methods while holding the detector setting constant. Under that setting, the paper reports that its method outperforms MDN on VOC07+12.

B. Comparison with the latest work MDN

C. Results on SOTA Detectors

The method is designed for active acquisition independent of the detector, and the paper reports an additional evaluation using YOLOv5 on MS COCO.

C. Results on SOTA Detectors

D. Combining with Semi-supervised Learning

The paper combines its method with the semi-supervised method UBT on MS COCO. It reports better results for the combination and describes the methods as complementary.

D. Combining with Semi-supervised Learning

E. Comprehensive Results on MS COCO

The MS COCO results compare the method with four active-learning counterparts using AP across IoU thresholds and size-specific AP metrics. The method outperforms counterparts in most cases, while Core-set performs better on large objects.

E. Comprehensive Results on MS COCO

  • The comparison reports AP over IoU thresholds from 0.5 to 0.95, plus AP50, AP75, and AP for small, medium, and large objects.AP50 and AP75 use IoU thresholds of 0.5 and 0.75, respectively; APS, APM, and APL correspond to small, medium, and large objects.Source: E. Comprehensive Results on MS COCO · p. 11
  • Our method outperforms counterparts in most cases and achieves higher averaged AP across scales than Core-set, though Core-set performs better on large objects.Core-set's spatial pooling strengthens large-object features; our more balanced integration of instance-level features comes at the cost of slightly lower AP for large objects.Source: E. Comprehensive Results on MS COCO · pp. 11–12
  • Table E compares active-learning methods using Faster R-CNN with a ResNet-50 backbone.The table includes Core-set, CDAL, Learn Loss, MIAL, and the proposed method.Source: E. Comprehensive Results on MS COCO · pp. 11–12

Related papers

Loading 2204.07965v1…