PaperlayerView exact revision on arXiv

Source-linked AI summary

Cost-Effective Active Learning for Deep Image Classification

Keze Wang, Dongyu Zhang, Ya Li, Ruimao Zhang, Liang Lin

arXiv:1701.03551v1cs.CV

TL;DR

Deep image classification requires many costly annotations, while traditional active learning leaves CNN feature learning under-supported and assumes fixed representations. CEAL integrates CNN training with complementary selection of uncertain samples for annotation and high-confidence samples for pseudo-labeling, and experiments on CACD and Caltech-256 report better accuracy with fewer labeled samples.

  • Problem

    Traditional active learning selects only a few uncertain samples and assumes fixed features, leaving CNNs with insufficient data for joint feature and classifier learning.

  • Method

    CEAL combines CNNs with complementary selection of low-confidence samples for user annotation and high-confidence samples for automatic pseudo-labeling during incremental model updates.

  • Results

    CEAL outperforms compared methods in recognition accuracy and annotation efficiency on CACD and Caltech-256.

  • Takeaways & Limitations

    CEAL needs fewer labeled samples while supporting CNN classifier training and feature learning through complementary manually labeled and pseudo-labeled data.

Abstract

from arXiv · show

Recent successes in learning-based image classification, however, heavily rely on the large number of annotated training samples, which may require considerable human efforts. In this paper, we propose a novel active learning framework, which is capable of building a competitive classifier with optimal feature representation via a limited amount of labeled training instances in an incremental learning manner. Our approach advances the existing active learning methods in two aspects. First, we incorporate deep convolutional neural networks into active learning. Through the properly designed framework, the feature representation and the classifier can be simultaneously updated with progressively annotated informative samples. Second, we present a cost-effective sample selection strategy to improve the classification performance with less manual annotations. Unlike traditional methods focusing on only the uncertain samples of low prediction confidence, we especially discover the large amount of high confidence samples from the unlabeled set for feature learning. Specifically, these high confidence samples are automatically selected and iteratively assigned pseudo-labels. We thus call our framework "Cost-Effective Active Learning" (CEAL) standing for the two advantages.Extensive experiments demonstrate that the proposed CEAL framework can achieve promising results on two challenging image classification datasets, i.e., face recognition on CACD database [1] and object categorization on Caltech-256 [2].

I. INTRODUCTION

The paper introduces CEAL, an active learning framework that integrates CNN feature learning with classifier training by combining manually labeled uncertain samples and pseudo-labeled high-confidence samples. Experiments on CACD and Caltech-256 report improved accuracy and reduced annotation needs.

  • Existing active learning methods select few uncertain samples, leaving CNNs with insufficient labeled data for effective feature representation learning.
  • Traditional active learning assumes fixed feature representations, whereas CNNs jointly optimize feature learning and classifier training, creating a potential divergence problem.
  • CEAL combines CNNs with complementary sample selection to fine-tune the model using progressively annotated informative samples and unlabeled training data.
  • CEAL updates both the CNN feature representation and classifier incrementally as annotated uncertain and pseudo-labeled certain samples enter training.
  • Minority low-confidence samples receive user annotations for classifier training, while majority high-confidence samples receive automatic pseudo-labels for feature learning.
  • Experiments on CACD and Caltech-256 show higher classification accuracy and reduced human annotation compared with other methods.

II. RELATED WORK

The related work reviews uncertainty-based active learning and motivates CEAL's use of high-confidence unlabeled samples and incremental joint optimization for deep image classification.

  • Uncertainty-based active learning selects unlabeled instances from classifier predictions, including entropy, decision-boundary distance, and committee disagreement.
  • These methods generally ignore the majority of certain unlabeled samples and can therefore be sensitive to outliers.
  • CEAL progressively selects complementary samples for model updating while handling manually annotated and automatically pseudo-labeled data together.
  • The problem setting assumes that most labels are initially unknown and that unlabeled data may arrive incrementally as the dataset grows.
  • The objective uses CNN softmax probabilities and alternates updates of pseudo-labels in DU with updates of network parameters W.The indicator function selects label-consistent terms in the objective, while softmax outputs represent category probabilities.

A. Initialization.

CEAL initializes the CNN from a small manually labeled set, then separates unlabeled samples into uncertain instances for annotation and high-confidence instances for pseudo-labeling.

  • A few samples from each class are manually annotated to initialize the CNN parameters W.
  • Unlabeled samples are ranked using active-learning criteria, with the most uncertain samples manually annotated and added to D_L.
  • CEAL can select K uncertain samples using least confidence, margin sampling, or entropy criteria.
  • High-confidence samples are selected when their entropy is below threshold δ and assigned predicted pseudo-labels.

C. CNN fine-tuning

CEAL fine-tunes CNN parameters using manually labeled and pseudo-labeled samples, repeating this process within an iterative learning algorithm.

  • The algorithm initializes W with D_L and iteratively adds K uncertainty samples while selecting high-confidence samples D_H.
  • Every t iterations, W is updated by fine-tuning on the union D_H ∪ D_L.
  • The fine-tuning loss uses N samples from D_H ∪ D_L and updates W through standard backpropagation.
  • After fine-tuning, pseudo-labeled high-confidence samples are returned to D_U and their pseudo-labels are erased.

D. Threshold updating

CEAL updates the high-confidence selection threshold during incremental learning to preserve pseudo-label reliability as classifier capability improves.

  • As classifier capability improves, more high-confidence samples are selected and incorrect automatic annotation may decrease.
  • At the end of each iteration t, CEAL updates the threshold to guarantee reliable high-confidence sample selection.
  • The threshold uses δ0 as its initial value, while dr controls the threshold decay rate.
  • Figure 2 shows sample images from Caltech-256 and the Cross-Age Celebrity Dataset for the evaluated classification tasks.

A. Datasets and Experiment settings

Experiments evaluate CEAL on CACD face recognition and Caltech-256 object categorization using CNN configurations tailored to each dataset and varying annotation budgets.

  • Datasets and Experiment settings: Experiments use CACD for face recognition and Caltech-256 for object categorization.CACD experiments use 500 individuals from a selected subset, while Caltech-256 contains 30,607 images across 256 categories.
  • Datasets and Experiment settings: CACD faces are resized to 200 × 150, whereas Caltech-256 images are resized to 256 × 256.The Caltech-256 setup uses an AlexNet architecture with an ImageNet-pretrained starting model and a 256-way softmax classifier.
  • Datasets and Experiment settings: The CACD CNN takes 200 × 150 × 3 inputs and produces 500-way class predictions.The detailed architecture is summarized in Table I.
  • Datasets and Experiment settings: The Caltech-256 CNN takes 256 × 256 × 3 inputs, randomly crops them to 227 × 227 during training, and produces 256-way predictions.The architecture is summarized in Table II.
  • Datasets and Experiment settings: Learning rates are 0.01 for all CACD layers and 0.001 for Caltech-256 layers except the softmax layer, which uses 0.01.Training CACD with 44,708 images requires an average of 17 hours on the reported desktop system.
  • Datasets and Experiment settings: Figure 3 reports classification accuracy across different percentages of annotated training samples on both datasets.The caption states that CEAL MS performs consistently better than TCAL and AL RAND.

3) Comparison Methods:

The comparison includes fully labeled, random-selection, and active-learning baselines, with CEAL evaluated using common informative-sample criteria and fusion variants.

  • Comparison Methods: The experiments compare CEAL with TCAL, AL ALL, and AL RAND.The comparison is designed to assess classification performance with less labeled data.
  • Comparison Methods: AL ALL manually labels every training sample and serves as the CNN upper bound.It represents the best performance attainable when all training labels are used.
  • Comparison Methods: AL RAND randomly selects samples for annotation and serves as the lower-bound baseline.This baseline discards active-learning selection techniques.
  • Comparison Methods: TCAL jointly evaluates uncertainty, diversity, and density to select minority informative samples.It is treated as a relevant competitor because it is described as a state-of-the-art active-learning approach requiring fewer annotations.
  • Comparison Methods: CEAL MS uses margin sampling to measure uncertainty, while Figure 4 studies least confidence, margin sampling, and entropy criteria.Figure 5 additionally compares individual criteria with their fusion on CACD and Caltech-256.

B. Comparison Results and Empirical Analysis

CEAL improves recognition accuracy and reduces annotation requirements relative to the compared methods, with the strongest annotation savings reported at target accuracies on CACD and Caltech-256.

  • Comparison Results: CEAL MS outperforms compared methods at the same annotation percentage, especially when few samples are annotated.The comparison covers recognition accuracy across annotation percentages on both datasets.
  • Comparison Results: CEAL performs consistently better than TCAL in recognition accuracy and user annotation amount under the reported fair comparisons.The paper attributes TCAL’s disadvantage to mining only minority informative samples, which provides insufficient training data for deep feature learning.

2) Component Analysis:

Component analyses show that pseudo-labeling high-confidence samples complements uncertainty-based selection, while fusing multiple criteria performs best among the tested selection variants.

  • Component Analysis: CEAL variants use least confidence, margin sampling, or entropy for uncertain-sample selection, with corresponding versions that omit pseudo-labeled samples.These variants are evaluated on CACD and Caltech-256.
  • Component Analysis: Pseudo-labeling majority high-confidence samples alone achieves performance gains similar to several informative-sample criteria.CEAL RAND uses only pseudo-labeled majority samples and performs similarly to AL LC, AL MS, and AL EN at equal labeled percentages.
  • Component Analysis: CEAL variants combining pseudo-labeled samples with LC, MS, or EN outperform both CEAL RAND and versions excluding pseudo-labeled samples.The comparison supports complementarity between pseudo-labeling and common informative-sample criteria.
  • Component Analysis: CEAL FUSION selects samples using combined criteria and performs better than CEAL LC, CEAL MS, and CEAL EN.The individual criteria show similar performance, while the fusion improves recognition accuracy.
  • Component Analysis: The pseudo-label heuristic assigns reliable labels to high-confidence unlabeled samples under an acceptable average error rate.Figure 6 plots average pseudo-label error rate against learning iteration on both datasets.

C. Reliability of CEAL

CEAL’s pseudo-labeling remains reliable across learning iterations, with low average error rates on both datasets. Its performance is also relatively insensitive to the tested threshold settings, although the framework remains intended for broader evaluation in future work.

  • Pseudo-label reliability: Less than 3% average pseudo-labeling error on CACD and less than 5.5% on Caltech-256, even at early iterations.The error remains low along the learning iterations, supporting reliable pseudo-label assignment under an acceptable average error rate.
  • Threshold sensitivity: Threshold δ and decay rate dr do not substantially affect overall system performance.The first row of Fig. 7 analyzes δ, while the second row analyzes dr.
  • Threshold sensitivity: 10%–60% of samples are pseudo-labeled across the tested threshold and decay-rate ranges.The tested parameter ranges were chosen to preserve the paper’s majority-high-confidence assumption.
  • Scope: The framework is evaluated on two challenging benchmarks, while broader large-scale and multi-label applications are reserved for future work.Planned extensions include ImageNet-scale recognition, additional CACD participants, and PASCAL VOC multi-label recognition.
Loading 1701.03551v1…