PaperlayerView exact revision on arXiv

Source-linked AI summary

Localization-Aware Active Learning for Object Detection

Chieh-Chi Kao, Teng-Yok Lee, Pradeep Sen, Ming-Yu Liu

arXiv:1801.05124v1cs.CV

TL;DR

Object-detection active learning has received limited attention, in part because assessing the informativeness of object localization is difficult. This paper adds localization tightness and stability metrics to classification-based active learning. Its experiments report better performance than classification-only active learning and passive learning when localization uncertainty is considered.

  • Problem

    Existing object-detection active-learning approaches mainly assess classification information, despite detection accuracy also depending on localization.

    Source: §1 · p. 2

  • Method

    The paper augments classification-based active learning with localization uncertainty, measured by proposal-to-final-box tightness and box stability under Gaussian image noise.

    Source: §1 · p. 2

  • Results

    Considering both localization and classification uncertainty outperforms classification-only active learning and passive learning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For object detection, image selection for annotation should consider localization as well as classification uncertainty.

    Source: §1 · p. 2

    Localization tightness reduces to classification uncertainty when perfect candidate windows are available, while localization stability may not help when feature extractors resist Gaussian noise.

    Source: §5 · p. 8

Abstract

from arXiv · show

Active learning - a class of algorithms that iteratively searches for the most informative samples to include in a training dataset - has been shown to be effective at annotating data for image classification. However, the use of active learning for object detection is still largely unexplored as determining informativeness of an object-location hypothesis is more difficult. In this paper, we address this issue and present two metrics for measuring the informativeness of an object hypothesis, which allow us to leverage active learning to reduce the amount of annotated data needed to achieve a target object detection performance. Our first metric measures 'localization tightness' of an object hypothesis, which is based on the overlapping ratio between the region proposal and the final prediction. Our second metric measures 'localization stability' of an object hypothesis, which is based on the variation of predicted object locations when input images are corrupted by noise. Our experimental results show that by augmenting a conventional active-learning algorithm designed for classification with the proposed metrics, the amount of labeled training data required can be reduced up to 25%. Moreover, on PASCAL 2007 and 2012 datasets our localization-stability method has an average relative improvement of 96.5% and 81.9% over the baseline method using classification only.

1. Introduction

Object-detection annotation is costly, and the paper argues that active learning for detection must account for localization as well as classification. It introduces two localization-uncertainty metrics for selecting informative images.

1. Introduction

  • Bounding-box annotation can take 10 times longer than answering an image-class question, and its quality is harder to monitor and control [20].Object-detection annotation also requires annotators to specify bounding boxes.Source: §1 · p. 1
  • Active learning selects informative unlabeled samples for annotation and can reach the same accuracy as passive learning with fewer labeled samples.A baseline is trained on a small labeled set, then updated with selected, newly labeled samples.Source: §1 · p. 1
  • Prior object-detection active-learning methods mainly measure classification information, although detection accuracy also depends on localization.The proposed approach evaluates uncertainty in both classification and localization.Source: §1 · p. 2
  • Localization tightness estimates uncertainty from changes between an intermediate proposal and the final class-specific box, while localization stability measures box variation under Gaussian image noise.Tightness applies to propose-then-classify detectors; stability can apply to all detector types, especially those without an explicit proposal stage.Source: §1 · p. 2
  • Considering localization and classification uncertainty outperforms active learning based on classification alone and passive learning.The metrics are intended to assess localization uncertainty when object-location ground truth is unknown.Source: §1 · p. 2

2. Related Works

Related work covers classification-oriented active learning, its applications to object detection, and other approaches to reducing annotation effort.

2. Related Works

  • The paper uses maximum uncertainty as its classification-active-learning baseline because it is efficient to compute.The method has also been used in classification and video annotation and with decision trees, SVMs, and Gaussian processes.Source: §2 · p. 2
  • Prior active-learning approaches for satellite- and vehicle-image detection consider classifier information, whereas this paper also considers localization.A separate approach actively crawls web images to train a part-based linear SVM detector.Source: §2 · p. 2
  • Deep-learning detectors either generate proposals before classification and geometry refinement [23] or directly output locations and classes, as in YOLO [22] and SSD [18].This distinction motivates localization stability for both detector categories.Source: §2 · p. 2
  • Other annotation-cost approaches use video-frame temporal coherence, transfer classifier knowledge without bounding-box annotation [10], simplify box labeling to a yes/no question [20], or combine computer-vision and human inputs.Source: §2 · p. 2

3. Active Learning for Object Detection

The method extends object-detection active learning to consider localization as well as classification when selecting images. It measures localization uncertainty through proposal-to-box refinement and box variation under image noise.

3. Active Learning for Object Detection

  • Each active-learning round selects a subset of newly collected images for annotation, adds them to the training set, and retrains the detector.The cycle repeats with the updated detector.Source: §3.1 · p. 3

3.1. Active Learning with Classification Uncertainty

  • For classification, box uncertainty is 1 − Pmax(B), and image uncertainty is the maximum box uncertainty among its detections.Pmax(B) is the highest class probability for box B.Source: §3.1 · p. 3

3.2. Localization Tightness

  • Localization Tightness estimates box fit without ground truth using IoU between an intermediate region proposal and its refined, class-specific box.Images with inconsistent classification and localization are preferred, including certain but loose boxes or tight but class-uncertain boxes.Source: §3.2 · pp. 3–4

3.3. Localization Stability

  • Localization Stability measures IoU between original-image reference boxes and corresponding detections across Gaussian-noise levels; images with larger localization variation are selected.The image score weights reference-box stability by each box’s highest class probability.Source: §3.3 · pp. 4–5

4. Experimental Results

Across PASCAL 2012 and 2007, adding localization cues to classification-based active learning improved annotation savings over classification-only selection, with localization stability especially beneficial for difficult categories. On MS COCO and SSD, localization-aware selection also reduced annotation needs relative to passive learning or the classification-only baseline.

4. Experimental Results; 4.1. FRCNN on PASCAL 2012

  • On PASCAL 2012, LT/C(GT) performed best in most cases, while estimated LT/C approached its upper bound where passive learning matched performance with 500–1,100 labeled images.Experiments compared random selection, classification-only selection, LS+C, LT/C, and variants using ground-truth tightness or combined cues.Source: §4.1 · p. 5, §4 · p. 5

4.1. FRCNN on PASCAL 2012; 4.2. FRCNN on PASCAL 2007

  • 96.5% and 81.9% relative improvement over classification-only selection in average saving came from LS+C on PASCAL 2012 and 2007, respectively.On PASCAL 2007, LT/C achieved 45.2% relative improvement over C; LS+C and LT/C both outperformed C in mAP.Source: §4.1 · pp. 5–6, Figure 5 · p. 6, §4.2 · p. 6
  • 3.95% versus 0.38% average AP improvement for LS+C over C on difficult versus other PASCAL 2012 categories; the corresponding PASCAL 2007 values were 3.94% versus 0.95%.Difficult categories were defined as those with passive-learning AP below 40%.Source: §4.1 · p. 6, §4.2 · p. 6, Figure 6 · p. 6

4.3. FRCNN on MS COCO

  • 5% relative annotation saving over passive learning came from LS+C on MS COCO, while classification-only selection did not improve over passive learning.Results average three trials; the experiments used Faster R-CNN on a dataset with 80 categories.Source: Figure 8 · p. 7, §4.3 · p. 7

4.4. SSD on PASCAL 2007

  • 12–22% versus 6–15% image reduction for LS+C versus C at equal passive-learning performance on SSD, with LS+C averaging 29.0% better saving than C.The comparison covers passive learning with 2,300–3,500 labeled images; SSD has no proposal stage, so localization-tightness tests were skipped.Source: Figure 9 · p. 7, §4.4 · pp. 7–8

5. Discussion

The discussion qualifies localization-aware active learning by identifying cases where its localization cues lose value, explaining when estimated tightness is reliable, and noting the computation cost of stability scoring.

5. Discussion

  • Perfect candidate windows make LT/C reduce to classification uncertainty, while noise-resilient feature extractors make LS+C reduce to classification uncertainty.The authors note perfect windows may be possible for a few special classes, but report no perfect region-proposal algorithm for all object types; tested AlexNet, VGG16, and ResNet101 were not noise-resilient.Source: §5 · p. 8
  • Estimated localization tightness is most accurate when proposals closely match final detection boxes and can diverge when they do not.This discrepancy is offered as a partial explanation for why estimated tightness performs below ground-truth-based tightness.Source: §5 · p. 8
  • Localization-stability scoring is slower than other metrics because it requires multiple detector runs, although the metrics are automatic to calculate.For Faster R-CNN, forwarding propagation took 137 ms per image, or 82.5% of the time considering classification uncertainty; the authors still describe the annotation savings as cost efficient.Source: §5 · p. 8

6. Conclusion

The conclusion presents an object-detection active-learning algorithm that considers both classification and localization uncertainty. The reported experiments show improved selection over classification-only active learning, allowing the same accuracy with fewer annotated images.

6. Conclusion

  • Considering localization uncertainty alongside classification improves classification-only active learning, enabling object detectors to reach the same accuracy with fewer annotated images.The two localization metrics assess bounding-box tightness around true objects and stability under image noise.Source: §6 · p. 8

Supplementary Materials

The supplementary document contains proposed-method data and analyses not included in the main paper. Its abbreviations are used throughout the document.

Supplementary Materials

A. Design of Localization-Tightness Metric

The section compares ways to score images using localization tightness and classification confidence. LT/C performs best among the evaluated localization-tightness metrics, although its advantage narrows later in the experiment.

A. Design of Localization-Tightness Metric

  • Using localization tightness alone performs worse than other metrics because it tends to select images with tiny objects from easier classes the detector already handles well.Figure 10 contrasts examples selected using localization-only and classification-uncertainty-only scores.Source: A. Design of Localization-Tightness… · p. 11, Figure 10 · p. 11
  • The design tests image-level aggregation and box-level scoring; P is a box's highest class probability, and T is its localization tightness.The tested image scores use either the lowest box score or a Pmax-weighted sum.Source: A. Design of Localization-Tightness… · p. 11
  • LT/C uses min(|T+P-1|) to select boxes with classification–localization disagreement, including boxes uncertain in both outputs.Source: A. Design of Localization-Tightness… · p. 11
  • The min(-|P-T|) alternative selects disagreement but not boxes uncertain in both outputs; the weighted-sum variant uses LT/C's box score with a different image score.The weighted-sum and LT/C image selections overlap by only 17.9% on average over five trials, showing that image-score definition affects sampling.Source: A. Design of Localization-Tightness… · pp. 11–12
  • LT/C outperforms the other localization-tightness metrics clearly in the first half of the experiment, with a smaller gap in the second half.Figure 11 reports mAP curves averaged over five trials.Source: A. Design of Localization-Tightness… · p. 11, Figure 11 · p. 12

B. Discussion of Extreme Cases

The discussion identifies extreme cases in which the proposed methods may not help. Tests of feature extractors found that their classification accuracy fell under Gaussian noise rather than showing noise resilience.

B. Discussion of Extreme Cases

  • LT/C may not help when perfect candidate windows are available, while LS+C may not help when feature extractors are resilient to Gaussian noise.Source: B. Discussion of Extreme Cases · p. 12
  • 23–37%: With the largest Gaussian-noise standard deviation, tested feature extractors' top-1 accuracy drops by 23–37%, and none is noise-resilient.The test uses AlexNet, VGG16, and ResNet101 on ImageNet validation data.Source: B. Discussion of Extreme Cases · p. 12

C. Full Experimental Results

The full experiments compare localization-stability and combined-cue selection with classification-only sampling, then examine whether adding localization tightness improves selection. Results also include trial variability and overlap between methods’ selected images.

Results of Using 3 Cues

  • On PASCAL 2012, 3in1 does not outperform LS+C or LT/C; on PASCAL 2007, it is compatible with LS+C and better than LT/C.The 3in1 score combines classification uncertainty, localization stability, and localization tightness, with both localization terms weighted by 1.Source: C. Full Experimental Results · p. 12
  • 3in1 selections overlap by about 30% with C and LS, about 10% with LT/C, and over 60% with LS+C, despite no improvement over LS+C.The authors suggest that the roughly one-third of images selected differently by 3in1 and LS+C may explain their performance difference.Source: C. Full Experimental Results · p. 13, Table 5 · p. 15, Table 6 · p. 15

mAP Plots with Error Bars

  • The added mAP plots show FRCNN on MS COCO and SSD on PASCAL 2007, adding error bars for minimum and maximum values across multiple trials to show trial-result distributions.Source: C. Full Experimental Results · p. 14

Results of Using Localization Stability Only

D. Visualization of The Selection Process

The visualization section maps PASCAL 2012 images into a two-dimensional feature space and overlays active-learning selections. It uses these views to examine class distribution and how sampling relates to difficult categories.

Visualization of the PASCAL 2012 Dataset

Visualization of Different Active Learning Methods

Related papers

Loading 1801.05124v1…