PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Learning for Convolutional Neural Networks: A Core-Set Approach

Ozan Sener, Silvio Savarese

arXiv:1708.00489v4stat.MLcs.CVcs.LG

TL;DR

CNNs require costly large labeled datasets, motivating better selection under a labeling budget. The paper reframes batch active learning as geometric core-set selection, derives a bound, and chooses subsets by minimizing it. Across image-classification experiments, the method significantly outperforms existing approaches by a large margin.

  • Problem

    CNNs need large labeled datasets, making it important to choose which points to label when the labeling budget is fixed.

  • Method

    The paper formulates batch active learning as unlabeled core-set selection, bounds subset performance using data geometry, and selects subsets through the resulting k-Center problem.

  • Results

    The proposed method significantly outperforms existing active-learning approaches by a large margin in image-classification experiments.

  • Takeaways & Limitations

    Geometric subset selection is reported as an effective alternative to uncertainty-based heuristics for batch active learning with CNNs.

  • Takeaways & Limitations

    The theoretical study uses an l2 loss and zero-training-error assumption, while experiments use cross-entropy loss; the bound does not extend to cross-entropy.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) have been successfully applied to many recognition and learning tasks using a universal recipe; training a deep model on a very large dataset of supervised examples. However, this approach is rather restrictive in practice since collecting a large set of labeled images is very expensive. One way to ease this problem is coming up with smart ways for choosing images to be labelled from a very large collection (ie. active learning). Our empirical study suggests that many of the active learning heuristics in the literature are not effective when applied to CNNs in batch setting. Inspired by these limitations, we define the problem of active learning as core-set selection, ie. choosing set of points such that a model learned over the selected subset is competitive for the remaining data points. We further present a theoretical result characterizing the performance of any selected subset using the geometry of the datapoints. As an active learning algorithm, we choose the subset which is expected to yield best result according to our characterization. Our experiments show that the proposed method significantly outperforms existing approaches in image classification experiments by a large margin.

1 INTRODUCTION

CNNs achieve strong vision performance but require large labeled datasets, making labeling cost a central constraint. This motivates batch active learning and the paper’s core-set formulation for selecting informative subsets.

  • Motivation: Large labeled datasets are costly, creating a fixed-budget problem of choosing which unlabeled points to label for highest accuracy.The paper frames active learning as a response to CNNs’ need for extensive supervision and continued gains from more data.
  • Problem: Classical active-learning heuristics are ineffective for CNNs in batch settings because acquiring correlated batches weakens their usefulness.Single-point acquisition is impractical because each point has little effect under local optimization and each iteration would require full retraining; larger batches introduce correlation.
  • Contribution: The paper reformulates batch active learning as unlabeled core-set selection, seeking a small subset whose trained model remains competitive on the full dataset.The proposed approach uses a geometry-based bound and selects a subset that minimizes it, yielding a k-Center optimization problem with an efficient approximate solution.
  • Contribution: Experiments on three image-classification datasets report state-of-the-art performance by a large margin.

2 RELATED WORK

Related work spans classical active learning, model-specific and optimization-based batch methods, weakly supervised deep learning, and core-set selection. The paper differs by directly addressing core-set selection for CNNs with theoretical guarantees and without extra assumptions.

  • Active Learning: Classical active-learning research includes information-theoretic, ensemble, uncertainty-based, and Bayesian acquisition strategies, but several do not scale directly to large CNN datasets.The paper reports that uncertainty-based methods are not effective for CNNs, while Bayesian approaches based on Gaussian processes do not directly scale to large CNN datasets.
  • Batch Active Learning: Optimization-based batch methods trade uncertainty against diversity, but some use n^2 variables and therefore do not scale to large datasets.Related approaches also include algorithms specialized to particular learners and hypothesis-space set-cover methods.
  • Theory: The paper also contrasts with prior theory showing impossibility in algorithm- and data-agnostic settings and positive results under assumptions about realizability or concept-space disagreement.
  • Core-Set Selection: Core-set selection chooses a subset on which training should remain close to training on the full labeled dataset, but prior methods target SVMs, k-Means, or k-Medians rather than CNNs.
  • Core-Set Selection: The closest unsupervised subset method uses facility location for diverse coverage, whereas this paper uses a minimax formulation and provides CNN-specific theoretical guarantees.The paper applies this formulation to active learning for the first time, according to the cited discussion.
  • Deep Learning: Weakly supervised deep-learning methods and CNN-specific active-learning approaches provide related alternatives, but reported effectiveness can depend on the task, including CAPTCHA recognition versus image classification.

3 PROBLEM DEFINITION

The paper defines batch pool-based active learning for multiclass classification over unlabeled data with a limited labeling budget. Each iteration selects points for oracle labeling and trains a classifier using labeled data, either fully or weakly supervised.

  • Setup: The problem concerns C-class classification over a compact input space, with class probabilities modeled by Lipschitz regression functions and data sampled i.i.d.
  • Pool-Based Active Learning: The learner observes all inputs but labels only an initial uniformly sampled pool, then receives a budget b for additional oracle queries.
  • Objective: An active-learning algorithm selects b extra points for labeling so the resulting model minimizes future expected loss.The formulation includes a learning algorithm that returns model parameters from a labeled set.
  • Batch Setting: The formulation uses batch acquisition because a single queried point has negligible effect in deep-learning regimes, while retraining after every point is intractable.The paper notes that classical active learning commonly uses budget b = 1 and may involve multiple rounds.
  • Iteration: Each iteration first selects points for an oracle and then trains with newly and previously labeled data, using either fully supervised or weakly supervised learning.

4 METHOD

The method reframes batch active learning as unlabeled core-set selection, bounds the resulting loss using geometric coverage, and solves the induced k-Center problem with greedy and MIP-based methods.

  • 4 METHOD: Batch active learning selects a subset whose trained model should perform competitively on the full dataset, avoiding one-point-at-a-time training that is impractical for CNNs.The formulation targets a batch budget because single examples have little effect and training one model per candidate is infeasible at scale.
  • 4 METHOD: The population-risk bound decomposes into training error, generalization error, and core-set loss, defined as the difference between subset and full-dataset average empirical losses.The method assumes zero training error for the core-set when applying the bound.
  • 4.2 CORE-SETS FOR CNNS: A subset’s core-set loss is bounded through its covering radius plus a term that vanishes with n, so additional labels help only when they reduce the covering radius.The theorem assumes Lipschitz loss and regression functions, bounded loss, and zero loss on selected points.
  • 4.2 CORE-SETS FOR CNNS: For CNNs, the analysis proves Lipschitz continuity for an l2 loss with max-pooling and ReLU, while experiments use cross-entropy despite the theory not extending to that loss.The CNN Lipschitz constant depends on the maximum sum of input weights per neuron, which can be reduced without changing labels of selected points.
  • 4.3 SOLVING THE K-CENTER PROBLEM: Minimizing the bound becomes k-Center selection, choosing centers that minimize the largest distance to a data point; the implementation uses final-layer activation distances.Greedy k-Center provides a 2-OPT solution, while a robust MIP improves it by checking feasible radii and allowing up to Ξ outliers.
  • 4.3 SOLVING THE K-CENTER PROBLEM: The robust MIP encodes centers, coverage, and outliers with binary variables, and its variables are illustrated as center choice, point coverage, and outlier assignment.The practical implementation uses Gurobi and sets the outlier bound to Ξ = 1e−4 × n for n unlabelled points.

5 EXPERIMENTAL RESULTS

Experiments across CIFAR and SVHN compare the proposed core-set method with random, uncertainty-based, clustering, and other active-learning baselines in fully and weakly supervised settings. The method outperforms all baselines, especially for weakly supervised models, while geometric coverage explains its advantage and class count limits performance.

  • Experimental setup: The evaluation covers coarse- and fine-grained CIFAR classification and SVHN digit classification using several active-learning baselines.Baselines include random selection, empirical uncertainty methods, k-Median, BMDR, and CEAL.
  • Active learning results: The proposed method outperforms all baselines across the experiments, with a large margin for weakly supervised models.The experiments evaluate fully supervised and weakly supervised models using average classification accuracy over five random initializations.
  • Active learning results: The method’s weakly supervised advantage is attributed to better feature spaces, whereas its reduced effectiveness on CIFAR-100 follows from a bound that scales with the number of classes.The geometric method benefits from more accurate feature-space geometry, while fewer classes yield a tighter core-set-loss bound.
  • Baseline analysis: BMDR does not necessarily outperform greedy methods, while k-Medoids fails to sample distributional tails because initial iid samples already cover cluster centers.The authors relate BMDR’s behavior to its reliance on uncertainty and softmax probabilities, which are not a good uncertainty proxy.
  • Active learning results: Uncertainty-based methods remain ineffective in batch settings because queried samples are correlated, allowing random sampling to outperform them.The tSNE analysis attributes this behavior to poor coverage of the representation space by uncertainty-selected queries.
  • Solver analysis: The optimal MIP solver is tractable in practice for 50k images, and the 2-OPT approximation scales further with a small accuracy drop while still outperforming other baselines.Figure 6 compares optimal k-Center and 2-OPT accuracy on CIFAR-100; Table 1 reports runtime for the algorithm.

6 CONCLUSION

The paper finds that classical uncertainty-based active-learning methods have limited applicability to CNNs because batch sampling creates correlations. It instead formulates active learning as core-set selection and reports state-of-the-art performance across three datasets by a large margin.

  • 6 CONCLUSION: Across three datasets, the proposed core-set approach achieves state-of-the-art performance by a large margin.The conclusion summarizes an extensive empirical validation of the algorithm.
  • 6 CONCLUSION: The paper reformulates CNN active learning as core-set selection after finding that classical uncertainty methods are limited by batch-sampling correlations.The core-set formulation targets a subset whose learned model remains competitive on the full dataset.

A PROOF FOR LEMMA 1

The proof establishes Lipschitz properties for CNN components and uses them to characterize the loss variation between inputs through their layer representations. These bounds support the paper’s geometric core-set analysis.

  • A PROOF FOR LEMMA 1: The softmax Jacobian is analyzed with the Frobenius norm, whose optimum occurs when all class probabilities satisfy f_i = 1/C.The proof first defines softmax and its Jacobian, then identifies the uniform probability vector as the optimal solution.
  • A PROOF FOR LEMMA 1: The layerwise bound assumes |w_i,j| ≤ α for convolutional and fully connected weights, enabling a uniform Lipschitz characterization.This assumption applies across the indexed weights and layers in the network.
  • A PROOF FOR LEMMA 1: The proof bounds CNN loss variation by combining Lipschitz constants for softmax, ReLU, max-pooling, convolutional, and fully connected layers.The argument uses bounded weights, the ReLU inequality, and the representation at an intermediate layer.
  • A PROOF FOR LEMMA 1: The proof applies the reverse triangle inequality and a Jacobian-norm result to connect differentiable network mappings with their Lipschitz bounds.The Jacobian norm is used as the local quantity underlying the Lipschitz argument.

B PROOF FOR THEOREM 1

The proof bounds expected loss at each point by comparing it with a nearby point having zero loss, then applies regularity assumptions and concentration to obtain the theorem’s probabilistic guarantee.

  • B PROOF FOR THEOREM 1: The proof begins by bounding the expected loss E_{y_i∼η(x_i)}[l(x_i,y_i;A_s)] using a nearby x_j within a δ-ball whose loss is zero.This nearby-point condition supplies the starting comparison for the bound.
  • B PROOF FOR THEOREM 1: Claim 1, the Lipschitz property of the regression function, and a loss bound control the remaining terms in the decomposition.The proof rewrites class-label probabilities using η_k(x_i) and applies these ingredients to derive the intermediate inequality.
  • B PROOF FOR THEOREM 1: The final intermediate bound uses the assumption that the trained classifier has zero loss on the training points.This assumption justifies the last step of the bound before the terms are combined.
  • B PROOF FOR THEOREM 1: Hoeffding’s bound converts the loss bound into a statement that holds with probability at least 1 − γ.The proof concludes by applying concentration after combining the preceding inequalities.
Loading 1708.00489v4…