PaperlayerView exact revision on arXiv

Source-linked AI summary

Self Paced Deep Learning for Weakly Supervised Object Detection

Enver Sangineto, Moin Nabi, Dubravko Culibrk, Nicu Sebe

arXiv:1605.07651v3cs.CV

TL;DR

Object detection needs costly box-level annotations, while weakly supervised methods use image-level labels but can drift when classifier errors produce false positives. This paper uses self-paced selection of reliable images and boxes in an end-to-end deep-network training protocol. It reports state-of-the-art results on ILSVRC 2013 and Pascal VOC 2007 and 2010.

  • Problem

    Bounding-box annotations are more costly to collect than image-level labels, and iterative MIL methods can drift when immature classifiers select false positives.

    Source: §1 · p. 1

  • Method

    The protocol iteratively uses the network to select reliable images and boxes, with inter-classifier competition and progressive class selection.

    Source: §1 · p. 1, §8 · p. 12

  • Results

    The method achieves state-of-the-art weakly supervised detection results on ILSVRC 2013 and Pascal VOC 2007 and 2010.

    Source: §1 · p. 2, §8 · p. 13

  • Takeaways & Limitations

    The authors show that self-paced selection can be used in end-to-end deep-network training and report that reduced noise boosts accuracy over different initialization methods.

    Source: §8 · pp. 12–13

    The training setup uses image-level class labels without bounding-box annotations; the passage also notes relaxations of inter-classifier competition for multiple labels and instances.

    Source: §3 · p. 4, §4 · p. 5

Abstract

from arXiv · show

In a weakly-supervised scenario object detectors need to be trained using image-level annotation alone. Since bounding-box-level ground truth is not available, most of the solutions proposed so far are based on an iterative, Multiple Instance Learning framework in which the current classifier is used to select the highest-confidence boxes in each image, which are treated as pseudo-ground truth in the next training iteration. However, the errors of an immature classifier can make the process drift, usually introducing many of false positives in the training dataset. To alleviate this problem, we propose in this paper a training protocol based on the self-paced learning paradigm. The main idea is to iteratively select a subset of images and boxes that are the most reliable, and use them for training. While in the past few years similar strategies have been adopted for SVMs and other classifiers, we are the first showing that a self-paced approach can be used with deep-network-based classifiers in an end-to-end training pipeline. The method we propose is built on the fully-supervised Fast-RCNN architecture and can be applied to similar architectures which represent the input image as a bag of boxes. We show state-of-the-art results on Pascal VOC 2007, Pascal VOC 2010 and ILSVRC 2013. On ILSVRC 2013 our results based on a low-capacity AlexNet network outperform even those weakly-supervised approaches which are based on much higher-capacity networks.

1 INTRODUCTION

The paper adapts self-paced learning to weakly supervised object detection, where image-level labels are available but bounding-box annotations are costly. It selects reliable images and boxes for deep-network training and reports state-of-the-art results on three benchmarks.

1 INTRODUCTION

  • The motivation is that object-level annotations take more time and expense to collect than image-level labels, while weakly supervised detection can use image-labeled datasets.In the MIL formulation, an image is a bag of boxes with at least one positive box for its label; iterative classifier errors can introduce false positives and drift.Source: §1 · p. 1
  • The work presents self-paced sample selection as useful for end-to-end deep-network training, extending prior applications mainly involving SVMs and other classifiers.It assumes image-level labels rather than bounding-box annotations at training time.Source: §1 · pp. 1–2
  • The self-paced protocol uses the evolving network to localize positive samples and select images whose pseudo-ground truth is most reliable.It is designed to reduce noisy training data and is embedded in an end-to-end deep-network training protocol.Source: §1 · pp. 1, 3
  • Fast-RCNN’s spatial regression dynamically changes the bag of boxes, and the method selects from current predictions rather than only initial proposals.The authors report that selecting over current predictions can boost final WSD accuracy.Source: §1 · pp. 2–3
  • The method uses class-specific confidence and competition among classifiers to reduce the probability of selecting incorrect samples.It also selects classes progressively, beginning with those estimated to be easiest.Source: §1 · pp. 2–3
  • The approach achieves state-of-the-art weakly supervised detection results on Pascal VOC 2007, Pascal VOC 2010, and ILSVRC 2013.On ILSVRC 2013, its AlexNet-based results exceed those of some weakly supervised methods using much larger-capacity networks.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work uses curriculum or self-paced sample selection and several deep-learning approaches to weakly supervised detection. This paper distinguishes its learner-driven, end-to-end training protocol from those approaches and presents it as usable with compatible architectures.

2 RELATED WORK

  • Curriculum learning orders samples using additional supervisory information, whereas self-paced learning computes the sample order automatically and a priori unknown.The cited curriculum approaches use metadata such as dataset difficulty, program complexity, or object scale.Source: §2 · p. 3
  • Prior self-paced work uses varied selection cues, including clustering, SVM objectives, exemplar classifiers, objectness, context, and saliency.The paper's class-selection goal is related to [30], but its approach is described as radically different.Source: §2 · p. 3
  • The paper embeds self-paced selection in an end-to-end deep-network training protocol, unlike earlier self-paced methods using CNN features without a deep classifier or end-to-end training.The protocol is presented as applicable to different architectures with Fast-RCNN-like region pooling and image-based mini-batch sampling.Source: §2 · pp. 3–4
  • Deep WSD approaches include max-pooling, soft-assignment, context, and attention designs, while this paper proposes a training protocol rather than a specific architecture.Some earlier ILSVRC approaches also used bounding-box annotations for half of the categories.Source: §2 · pp. 3–4
  • Compared with [34], this approach does not require an auxiliary object-size training set, and it updates pseudo-ground truth during Fast-RCNN training rather than using it only for initialization.The size-regression approach in [34] requires ground-truth object sizes; its Fast-RCNN variant uses pseudo-ground truth selected before detector training.Source: §2 · p. 4

3 FAST-RCNN AND NOTATION

This section establishes Fast-RCNN's image-and-box input, per-box class scores and regressed boxes, and its mini-batch training setup. It contrasts the box-level supervision used by standard Fast-RCNN with the image-level labels available in this paper's weakly supervised setting.

3 FAST-RCNN AND NOTATION

  • Fast-RCNN uses RoI pooling to extract box-specific information from convolutional maps for its classification and regression branches.The network takes an image and external region proposals as input and produces detections for those boxes.Source: §3 · p. 4
  • For each input box and class, the network outputs a score and a predicted box, which may differ from the original proposal after spatial regression.RoI pooling enables efficient computation over the set of boxes.Source: §3 · p. 4
  • Standard Fast-RCNN trains from mini-batches drawn from two images, matching annotated boxes to proposals by spatial overlap to define positives and background negatives.In the weakly supervised setting, training data provide image-level label sets, with object counts per class unknown.Source: §3 · p. 4
  • Because box-level annotations are unavailable, the current network estimates object locations, which are used to update the training set.The weakly supervised training set retains each image and its associated labels rather than box annotations.Source: §3 · p. 4

4 SELF-PACED LEARNING PROTOCOL

The protocol repeatedly uses the current detector to select reliable classes, images, and latent boxes, then trains on the resulting subset. Selection is governed by classifier competition and a gradually expanding pace; the section also describes initialization and a less effective alternative.

4 SELF-PACED LEARNING PROTOCOL

  • At each self-paced iteration, the current network selects easy classes and samples to form a new training set, then Fast-RCNN trains on that set for one epoch.The sequence of self-paced iterations is distinct from the mini-batch SGD iterations within each one.Source: §4 · p. 5
  • For each image, the highest-scoring detection supplies a latent box, but the image is included only when an annotated class out-scores competing classes.A predicted box may be outside the original proposal set because the detector's regression layer adjusts it; predictions are recomputed from original proposals in the next iteration.Source: §4 · p. 5
  • Competition permits only one pseudo-ground-truth box per image, regardless of its labels or number of same-class instances; the paper later considers relaxations.The constraint applies to the selected box for a given image.Source: §4 · p. 5
  • Class easiness is the fraction of accepted samples for that class relative to its dataset frequency, and only the easiest rtC classes are trained in that iteration.The class set grows as rt increases across iterations.Source: §4 · p. 6
  • After removing samples from difficult classes, the protocol ranks remaining images by confidence and selects up to min(rtN, |P|); rejection often leaves fewer than the nominal cap.The authors report that classifier competition, rather than the increasing sample cap, usually governs the pace.Source: §4 · p. 6
  • The protocol starts with half the data and uses four iterations to reach the full nominal pace, while keeping its hyperparameters fixed across Pascal VOC and ILSVRC experiments.The authors report similar results with five iterations and use Fast-RCNN hyperparameters except for one learning-rate reduction after the first self-paced iteration.Source: §4 · p. 6
  • The alternative of fully training on successively larger datasets is slower and less effective, with preliminary runs overfitting on the initial small set and achieving lower final accuracy.The proposed protocol instead updates the dataset by epoch.Source: §4 · p. 6
  • Initialization uses an image-level classification network to create fixed pseudo-ground truth for a one-shot MIL pretraining stage before self-paced training.For Pascal VOC, the paper also describes initialization from a Pascal-trained classifier or an ILSVRC classifier adapted to VOC classes.Source: §4 · p. 7

5 DISCUSSION

The discussion frames selective training as a way to reduce noisy pseudo-ground truth in MIL, even when that means not using every image. It also notes that datasets can change non-monotonically while overall mean AP remains nearly monotonic across iterations.

5 DISCUSSION

  • The protocol favors cleaner samples over using all available images, since noisy pseudo-ground truth can contribute to drift in MIL.Classifier competition may cause training to terminate without using the full dataset; the authors contrast this with the usual preference for more training data.Source: §5 · p. 7
  • Using each selected dataset for one SGD epoch is intended to avoid overfitting, especially when the initial training set is small.This is the stated rationale for the epoch-based update strategy.Source: §5 · p. 7
  • Although successive training sets are non-monotonic and class-specific AP oscillates, overall mean AP is almost monotonic, with oscillations typically below 1%.The authors attribute the overall pattern to weight sharing among classes and the common background class, alongside progressively increased training data.Source: §5 · pp. 7–8

6 DETECTION PERFORMANCE

The paper reports state-of-the-art comparisons on ILSVRC 2013 and Pascal VOC 2010, and substantial self-paced gains on Pascal VOC 2007. Results are reported using both AlexNet-like and VGG-16 networks.

6 DETECTION PERFORMANCE

  • The Pascal VOC 2007 comparison separates AlexNet-like and VGG-16-like methods and marks network ensembles because prior approaches vary in architecture and capacity.The default experimental architecture, including initialization, is AlexNet unless otherwise stated.Source: §6 · p. 8
  • The ILSVRC 2013 evaluation uses image-level labels and evaluates on val2 after training on val1; the reported protocol follows prior detection experiments.The dataset has 200 classes, and object-level annotations are not used.Source: §6 · p. 8
  • The method is reported to outperform prior weakly supervised approaches on ILSVRC 2013 and achieve state-of-the-art results on Pascal VOC 2010, including with VGG-16.The VGG-16 Pascal VOC 2010 results are reported to exceed prior results from an ensemble of three networks.Source: Table 1 · p. 8, §6 · pp. 8–9
  • On Pascal VOC 2007 trainval, the method’s CorLoc is second best, behind Shi et al. [34].The comparison uses a single AlexNet-like capacity network; Shi et al. use additional bounding-box annotations.Source: Table 3 · p. 9, §6 · p. 9
  • On Pascal VOC 2007, mAP reaches 38.11 with hP; self-paced training adds +7.36 mAP with hI and +6.18 with hP.The paper reports that 38.11 is higher than the other weakly supervised approaches tested on this dataset.Source: §6 · pp. 8–9, Table 4 · p. 9
  • With VGG-16, the Pascal VOC 2007 results are reported as state of the art and as evidence that the protocol is not inclined to overfit with a larger-capacity network.The comparison is against works using the same single network.Source: §6 · p. 9

7 ANALYSIS OF DIFFERENT ASPECTS OF THE PRO-

The analysis tests how protocol components and multi-label variants affect training performance, and evaluates the precision of selected training subsets.

7.1 Simplified versions of the training protocol

  • SP improves mAP over iterations, while standard MIL ends below initialization and Curriculum oscillates without improving on its starting point.Curriculum uses fixed scores from the initialization network, whereas SP selects images dynamically using the current detector.Source: §7.1 · pp. 10–11
  • Removing class selection lowers final mAP on both datasets, and random class selection performs worse than retaining all classes on Pascal VOC.The SP-all-cls variant removes class selection while retaining inter-classifier competition.Source: Table 8 · p. 11, Table 9 · p. 11, §7.1 · p. 11
  • 7.57 mAP, 4.56 points below the best result on ILSVRC, when the regression layer is disabled at both training and testing.The authors report a similarly large accuracy gap when repeating this experiment on ILSVRC.Source: Table 9 · p. 11, §7.1 · p. 11

7.2 Multi-label versions of the training protocol

  • The analysis compares SP’s single-box selection with variants that relax inter-classifier competition by collecting boxes for multiple labels or instances.SP-SIML collects one box per label; SP-MIML can collect multiple boxes of the same category from an image.Source: §7.2 · pp. 11–12
  • SP outperforms SP-SIML and SP-MIML on Pascal VOC 2007 despite those variants collecting more boxes, and their mined boxes have lower CorLoc.The authors report that many additional boxes in the relaxed variants are inaccurately localized or completely wrong.Source: Table 10 · p. 12, §7.2 · p. 12

7.3 Precision of the selected subsets of training data

  • Precision in T4 is much higher than in T1 and than the result reported by [17], even though [17] used object-level annotations for 100 of 200 categories.Precision is the percentage of correct image samples among the images included in the training set Tt.Source: §7.3 · p. 12

8 CONCLUSIONS

The paper introduces a self-paced protocol for deep networks that combines inter-classifier competition, class selection, and regression-based changes to the box proposals. It reports state-of-the-art results across three weakly supervised detection benchmarks.

8 CONCLUSIONS

  • The protocol combines inter-classifier competition to reduce noise, class selection to train easier classes first, and Fast-RCNN regression to modify the box bag implicitly.The goal is to reduce noise while training deep networks for weakly supervised detection.Source: §8 · p. 12
  • The protocol achieves state-of-the-art weakly supervised detection results on ILSVRC 2013 and Pascal VOC 2007 and 2010.The authors also report that the protocol benefits from reduced noise and boosts final accuracy relative to different initialization methods.Source: §8 · p. 13

APPENDIX

The appendix examines per-category detection performance on ILSVRC 2013 val2 and reports that improvement extends across many categories.

APPENDIX

  • The authors report that learning is spread across most categories rather than dominated by initially strong classifiers.They suggest that a weak classifier can win competition on its own samples and gain new training examples when stronger classifiers score those images poorly.Source: APPENDIX · p. 15
  • 147 of 200 categories have higher AP with W4 than with the initialization model W0.The table reports per-class AP on the ILSVRC13 detection val2 set.Source: APPENDIX · p. 15, Table 12 · p. 16

Related papers

Loading 1605.07651v3…