PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Loss for Active Learning

Donggeun Yoo, In So Kweon

arXiv:1905.03677v1cs.CVcs.LG

TL;DR

Limited annotation budgets motivate active learning, but existing deep-network methods can be task-specific or computationally inefficient. The paper attaches a loss prediction module to a target network to predict losses for unlabeled inputs and select examples for annotation. Across image classification, object detection, and human pose estimation, the method consistently outperforms previous methods.

  • Problem

    Annotation budgets are limited, while many active learning methods require task-specific design or are inefficient for recent deep networks.

  • Method

    A loss prediction module attached to a target network predicts losses for unlabeled inputs so high-loss examples can be selected for annotation.

  • Results

    The method consistently outperforms previous methods across image classification, object detection, and human pose estimation.

  • Takeaways & Limitations

    The method provides a simple active learning approach applicable to deep networks across three major visual recognition tasks.

  • Takeaways & Limitations

    The method does not consider data diversity or density, and loss prediction accuracy is relatively low for complex tasks such as object detection and human pose estimation.

Abstract

from arXiv · show

The performance of deep neural networks improves with more annotated data. The problem is that the budget for annotation is limited. One solution to this is active learning, where a model asks human to annotate data that it perceived as uncertain. A variety of recent methods have been proposed to apply active learning to deep networks but most of them are either designed specific for their target tasks or computationally inefficient for large networks. In this paper, we propose a novel active learning method that is simple but task-agnostic, and works efficiently with the deep networks. We attach a small parametric module, named "loss prediction module," to a target network, and learn it to predict target losses of unlabeled inputs. Then, this module can suggest data that the target model is likely to produce a wrong prediction. This method is task-agnostic as networks are learned from a single loss regardless of target tasks. We rigorously validate our method through image classification, object detection, and human pose estimation, with the recent network architectures. The results demonstrate that our method consistently outperforms the previous methods over the tasks.

1. Introduction

Deep networks remain data-hungry while annotation budgets are limited, motivating active learning methods that select informative unlabeled data. The paper proposes a simple, task-agnostic loss prediction module and validates it across three recognition tasks.

  • Annotation costs vary by task, from class labels to bounding boxes, pixel masks, and specialist medical annotations.
  • Active learning selects uncertain or informative unlabeled examples for annotation instead of relying on random sampling.
  • Existing deep-network methods often require task-specific designs or become computationally inefficient at scale.
  • The proposed loss prediction module learns to predict input losses, enabling selection of data expected to be more informative for the current model.
  • The method is evaluated on image classification, human pose estimation, and object detection using current deep-network architectures.
  • The method consistently outperforms previous methods across the evaluated recognition tasks.

2. Related Research

Prior active learning research includes uncertainty, distribution, and expected-model-change approaches, but scaling and task generality remain concerns. This paper predicts loss from input features without annotations, distinguishing its approach from output-based uncertainty and hard-example mining.

  • Classical uncertainty methods use posterior probabilities, margins, entropy, decision-boundary distances, or committee disagreement.
  • Distribution methods select representative or locally informative points, while expected model change estimates effects on parameters, errors, or outputs.
  • Deep-network approaches can remain task-specific, computationally inefficient, or insufficiently validated beyond classification.
  • The proposed method predicts loss from input contents rather than statistically estimating uncertainty from network outputs.
  • Unlike hard-example mining, it selects high-loss examples without requiring their annotations.

3. Method

The method attaches a small loss prediction module to a target network and jointly learns it to estimate target losses from multi-level features. It uses predicted losses to select informative unlabeled samples for annotation while avoiding task-specific uncertainty design.

  • 3.2. Loss Prediction Module: The loss prediction module predicts an input loss from features extracted from several hidden layers of the target model.This multi-level design uses intermediate representations rather than task-specific prediction outputs.
  • 3.1. Overview: The active-learning cycle trains a target model and loss prediction module, evaluates unlabeled samples, and annotates the K samples with highest predicted losses.The cycle repeats until satisfactory performance is reached or the annotation budget is exhausted.
  • 3.3. Learning Loss: The resulting loss prediction module supports task-agnostic active learning by selecting informative data without separate task-specific assumptions.The approach is designed to use rich mid-level representations with few additional parameters and to select samples for the next active-learning stage.
  • 3.2. Loss Prediction Module: The module is much smaller than the target model and is jointly learned with it, reducing the additional memory and computation required.Its features are globally pooled, projected, concatenated, and mapped to a scalar predicted loss.
  • 3.3. Learning Loss: For each labeled input, the target loss becomes the ground-truth signal for learning the module to predict loss.The target model computes l = Ltarget(ˆy, y), while the module computes a predicted loss from h; both objectives are combined in the final loss.
  • 3.3. Learning Loss: Because target-loss scale changes during training, the method compares paired samples and uses a margin-based loss to learn their relative loss ordering.When li > lj, the prediction incurs no loss only when ˆli exceeds ˆlj by the positive margin ξ.

4. Evaluation

The method is evaluated across image classification, object detection, and human pose estimation using deep-network architectures and standard active-learning baselines. It achieves the strongest reported performance across these tasks while predicting losses with useful ranking accuracy.

  • Evaluation setup: The evaluation covers image classification, object detection, and human pose estimation as classification, hybrid, and regression tasks.The experiments use CIFAR-10, PASCAL VOC, and MPII, respectively.
  • Evaluation setup: The experiments compare loss-based learning against random, entropy-based, and core-set sampling.The comparison includes entropy computed from softmax outputs and a K-Center-Greedy implementation for core-set sampling.
  • Image Classification: 0.9101 accuracy is achieved by the loss-learning method in the final CIFAR-10 active-learning cycle, exceeding entropy and core-set by 0.42% and 0.91%.Entropy and core-set reach 0.9059 and 0.9010, while the random baseline reaches 0.8764.
  • Loss Prediction: The loss prediction module reaches 0.9074 ranking accuracy as more labeled data are added and predicts regression loss with about 75% ranking accuracy.For human pose estimation, predicted loss correlates with real loss at 0.68, compared with 0.45 for entropy.
  • Object Detection: 0.7338 mAP is achieved on object detection, exceeding the random baseline by 2.21% and entropy and core-set by 1.15% and 1.63%.The results average 3 trials with different initial labeled datasets.
  • Human Pose Estimation: 0.8046 PCKh@0.5 is achieved on human pose estimation, exceeding entropy and core-set by 1.47% and 0.61%.The random baseline reaches 0.7862, and the results average 3 trials.

5. Limitations and Future Work

The method is applicable across visual recognition tasks, but its current design does not account for data distribution and has relatively low loss-prediction accuracy on complex tasks.

  • The method does not consider the diversity or density of data when providing uncertainty scores.
  • Loss prediction accuracy is relatively low for complex tasks such as object detection and human pose estimation.
  • Future work will incorporate data distribution and improve the module’s architecture and objective function.
Loading 1905.03677v1…