PaperlayerView exact revision on arXiv

Source-linked AI summary

Gone Fishing: Neural Active Learning with Fisher Embeddings

Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, Sham Kakade

arXiv:2106.09675v2cs.LG

TL;DR

Neural active learning needs practical batch-selection methods because deep networks often require considerable labeled data, while existing approaches can have limited explanations or fail in some settings. The paper revisits Fisher-based selection and proposes BAIT, which improves over baselines in deep classification and is applicable to regression and convex models.

  • Problem

    Deep networks often require considerable labeled data, while practical neural active learning must address batch querying, computational cost, and methods that may not transfer across settings.

    Source: §1 · p. 1

  • Method

    BAIT revisits a classic Fisher-based selection objective from a probabilistic perspective and makes it tractable for neural active learning.

    Source: §1 · p. 2

  • Results

    BAIT improves over baselines in deep classification across varying batch sizes, architectures, and datasets, and also performs well in regression and convex-model settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    BAIT offers a general-purpose active-learning approach that extends to regression and performs in both convex and non-convex scenarios.

    Source: §1 · p. 2, §7 · p. 10

    BAIT’s classification selection is roughly k times slower than BADGE, although retraining largely dominates total run time and the added computation may suit high label-cost settings.

    Source: §7 · p. 10

Abstract

from arXiv · show

There is an increasing need for effective active learning algorithms that are compatible with deep neural networks. This paper motivates and revisits a classic, Fisher-based active selection objective, and proposes BAIT, a practical, tractable, and high-performing algorithm that makes it viable for use with neural models. BAIT draws inspiration from the theoretical analysis of maximum likelihood estimators (MLE) for parametric models. It selects batches of samples by optimizing a bound on the MLE error in terms of the Fisher information, which we show can be implemented efficiently at scale by exploiting linear-algebraic structure especially amenable to execution on modern hardware. Our experiments demonstrate that BAIT outperforms the previous state of the art on both classification and regression problems, and is flexible enough to be used with a variety of model architectures.

1 Introduction

Active learning must choose informative batches when labels are costly, but neural networks make principled selection challenging because many foundational methods assume convexity, are computationally expensive, or do not scale to batch querying. The paper presents BAIT as a tractable Fisher-based approach and reports strong results across several settings.

1 Introduction

  • Deep neural networks often need considerable labeled data, making active learning relevant to reducing sample-complexity requirements in tasks such as medical diagnostics [2] and image labeling.Source: §1 · p. 1
  • Neural active learning must address nonconvex models, computational expense, and batch acquisition rather than only single-point queries [5].Foundational methods often assume convexity, while adapting expensive methods to real-world use can be unclear [4].Source: §1 · p. 1
  • BADGE performs robustly, but its behavior is not well explained; it also performs poorly with convex models and cannot be used for regression [6].Source: §1 · p. 1
  • BAIT adopts a probabilistic perspective and revisits a classic Fisher-based selection objective, aiming to make it tractable for overparameterized neural networks.The paper frames this as bridging performant but less theoretically understood algorithms and theoretically transparent but computationally intractable ones.Source: §1 · p. 2
  • BAIT improves over baselines in deep classification, with the trend robust across batch sizes, model architectures, and datasets, and it extends to regression and convex models.The paper also reports in-depth empirical comparisons of neural and convex regimes and discusses simplifying assumptions for neural settings.Source: §1 · p. 2

2 Related work

Related work spans diversity-based selection, uncertainty sampling, batch loss-bound optimization, and methods that combine uncertainty with diversity. Their suitability varies with batch size and model setting.

2 Related work

  • Diversity and uncertainty sampling have different strengths: diversity is successful for large batches, while uncertainty sampling is successful for small batches.Source: §2 · p. 2
  • Diversity methods select representative batches, often by embedding unlabeled examples in a neural network’s penultimate layer; a GAN-based approach instead seeks samples indistinguishable from the unlabeled pool [10] [11] [12].Source: §2 · p. 2
  • Batch active-learning methods commonly formulate selection as minimizing an upper bound on a notion of model loss [17].Source: §2 · p. 2
  • In linear models, uncertainty methods often query near the decision boundary or use predictive entropy or expected gradient size, but the gradient-based criterion cannot account for batch diversity.Source: §2 · p. 2
  • Neural uncertainty methods estimate uncertainty through dropout-based weight sampling, adversarial boundary distance, or predictive variance across ensembles [23] [24].Source: §2 · p. 2
  • Hybrid methods combine uncertainty and diversity through metalearning or bandit selection; BADGE also combines them and is considered state of the art for deep neural networks.Source: §2 · p. 2

3 Notation and Setup

The setup is pool-based batch active learning: a model is updated with labeled examples, then a batch is selected from an unlabeled pool for labeling. Experiments target low loss on the data distribution, while algorithm development uses a transductive objective over the pool.

3 Notation and Setup

  • The learner has an unlabeled pool U sampled from the instance distribution and can request labels for any point in that pool.Source: §3 · p. 3
  • After selecting a batch, the learner uses its newly labeled examples to update the neural network before proceeding to the next round.Source: §3 · p. 3
  • Experiments aim to minimize expected loss on the data distribution, while algorithm development uses a transductive objective that treats the unlabeled pool as the distribution; the objectives can typically be related by generalization arguments.Source: §3 · p. 3

4 Probabilistic Perspective

The paper connects active selection to probabilistic modeling and classical estimation: Fisher information characterizes MLE behavior, while Bayesian linear regression yields a subset-selection risk objective. BAIT uses a greedy approach to make Fisher-based selection usable in neural settings.

4 Probabilistic Perspective

  • The paper models neural networks as conditional label distributions and uses negative log likelihood, with cross-entropy for classification and square loss corresponding to Gaussian regression.Source: §4 · p. 3
  • In Bayesian linear regression, the setup assumes a Gaussian prior on parameters and Gaussian label noise, with the MAP estimate given by ridge regression using regularizer λσ^2.Source: §4 · p. 3
  • The Bayesian linear-regression design goal is to select a subset of unlabeled points that gives the resulting MAP estimate the lowest Bayes risk.The Bayes-risk expectation is over label noise and the prior on the true parameter.Source: §4 · p. 3
  • For Bayesian linear regression, the subset-selection objective does not depend on labels, so minimizing it is feasible and optimal under the stated criterion; this does not extend to neural settings.BAIT is designed to approximately minimize this objective.Source: §4 · p. 3
  • Classical MLE analysis uses Fisher information, defined as the expected loss Hessian, to characterize the estimator’s asymptotic distribution under model-specification and regularity assumptions.The paper notes that these conditions do not hold in the neural setting.Source: §4 · p. 4
  • An idealized classical active-learning strategy selects a batch to optimize a Fisher-based objective after an initial random batch and likelihood fit, but the combinatorial optimization is intractable [4].Source: §4 · p. 4
  • Because solving the SDP is infeasible in high dimensions, the paper provides a new greedy algorithm intended to minimize the objective in the neural regime.Prior work studied the objective theoretically or through relaxations, but some approaches do not ensure batch diversity and none had been extended to neural models.Source: §4 · p. 4
  • The Fisher formulation generalizes the linear-regression risk objective, differing there only in the regularizer, which gives a shared basis for active selection.Source: §4 · p. 4

5 BAIT

BAIT adapts Fisher-based active selection to neural networks by approximating its objective greedily, with practical updates for changing representations and regression. In the MNIST comparisons, BAIT outperforms determinantal objectives, especially with uninformed features.

5 BAIT

  • BAIT uses last-layer Fisher information and recomputes it each round because the learned representation changes during active learning.This framework can guide sampling when the top-layer representation begins to approximate a convex model.Source: §5 · p. 4
  • BAIT approximates the Fisher objective greedily; each batch first oversamples 2B points, then prunes B, a trade-off chosen for computational cost and batch quality.The trace objective is not submodular, and larger oversampling multipliers did not improve performance.Source: §5 · pp. 4–5
  • For regression, the pointwise Fisher reduces to a representation covariance term paired with the estimator’s inverse noise covariance, enabling rank-one updates instead of rank-k matrices.The representation is the penultimate-layer feature; for linear models it is the input x.Source: §5 · p. 6

5.1 BADGE comparison

  • BAIT selects samples using full pointwise Fisher matrices and the Fisher over all samples, whereas BADGE uses rank-one gradient embeddings and cannot handle regression.The authors state that the all-sample Fisher term is essential for minimizing risk and bounding MLE error.Source: §5.1 · p. 6
  • BAIT outperforms determinantal objectives in both MNIST representations, with the largest advantage on uninformed features; the full-rank Fisher helps later and I(θL) helps earlier.The comparison used forward greedy selection to isolate objective differences.Source: §5.1 · p. 7
  • In the orthonormal regression simulation, BAIT significantly outperforms determinantal maximization, which cycles through coordinates instead of using their occurrence probabilities.Both methods perform almost identically in the Gaussian simulation; the data have poorly conditioned covariance with quadratic spectral decay.Source: §5.1 · p. 7

6 Experiments

Across linear and deep classification and regression experiments, BAIT performs strongly against active-learning baselines. Its relative performance holds across varied deep-classification settings, and it remains competitive on regression tasks.

6 Experiments

  • The experiments compare BAIT with classic and recent active-learning approaches across linear classification, deep classification, and regression.The considered baselines include BADGE, CORESET, CONFIDENCE, and RANDOM sampling.Source: §6 · p. 7

6.1 Linear Classification

  • In unstructured random-feature MNIST, BAIT outperforms baselines by a large margin, while differences are subdued with learned representations.The random representation mimics the convex case, where the model cannot control data representations.Source: §6.1 · p. 8
  • Removing BAIT’s Fisher term yields performance on par with other baselines, and the authors suggest that including the term lets BAIT succeed in poorly structured feature spaces.The authors also suggest penultimate-layer representations may behave more like learned than random representations.Source: §6.1 · p. 8

6.2 Deep Classification

  • BAIT significantly outperforms baseline approaches in aggregate deep-classification comparisons and is as good as or better than the strongest approach across varied settings.Figure 5 aggregates experiments; additional pairwise plots suggest BAIT remains higher-performing across environments.Source: Figure 5 · p. 8, §6.2 · pp. 9–10

6.3 L2 Regression

  • Across linear and nonlinear regression tasks, BAIT consistently performs as well as or better than the strongest baseline and runs about as long as CORESET.The regression experiments include yearbook-photo year prediction with an 18-layer ResNet and rainfall prediction with a linear model; CONFIDENCE and BADGE are omitted because their uncertainty measures require classification.Source: §6.3 · p. 10, Figure 9 · p. 10

7 Discussion

BAIT succeeds across convex and non-convex settings and across classification and regression. Its classification selection is more computationally intensive than BADGE, a trade-off the authors consider justified when label acquisition is costly.

7 Discussion

  • BAIT is successful in both convex and non-convex scenarios, and in both classification and regression settings.The paper describes BAIT as a generalized, high-performing, and tractable approach to neural active learning.Source: §7 · p. 10
  • Classification BAIT selection is roughly k times slower than BADGE, though retraining models after acquisition largely dominates total run time.The authors say this added computation is justified when label acquisition costs are high relative to computation.Source: §7 · p. 10
  • The authors suggest estimating the Fisher with only the lowest-norm r < k columns of Vx as a possible computation–performance trade-off, leaving its analysis for future work.Source: §7 · p. 10

A.1 Bayesian Linear Regression

The appendix gives supporting derivations for Bayesian linear regression and Fisher information in classification and regression models. In the regression objective, the computation can use Kronecker-product structure.

A.1 Bayesian Linear Regression

  • In Bayesian linear regression, the MAP estimate is the ridge-regression solution with ridge regularizer λσ2.The setup assumes a Gaussian prior on the parameters and a Gaussian conditional response distribution.Source: §A.1 · p. 13

A.2 Fisher Information for Multi-class Logistic Regression

  • For multi-class logistic regression, the appendix computes the model log-likelihood and its derivatives to obtain the Fisher information.Source: §A.2 · p. 14

A.3 Fisher Information for Multi-output Regression

  • For multi-output regression, the Fisher information is the negative Hessian of the log-likelihood with respect to W.Source: §A.3 · p. 14

A.3.1 The Fisher Objective For Regression

  • The regression Fisher computation can be reduced using properties of the Kronecker product.The appendix introduces this reduction in its treatment of the Fisher objective for regression.Source: §A.3.1 · p. 15

A.4 Finding the Minimizing Sample in Regression

The regression derivation defines F from unlabeled examples and reduces the computation of a single point’s contribution in Algorithm 1.

A.4 Finding the Minimizing Sample in Regression

  • The regression derivation defines F using an average over unlabeled examples and their feature vectors.The supplied expression is fragmentary, so the exact matrix formula is not stated here.Source: §A.4 · p. 15
  • Using F, the contribution of one point in Algorithm 1 can be reduced to a simpler computation.Source: §A.4 · p. 15

A.5 Optimality of Greedy

The appendix establishes greedy optimality for a particular trace-optimization setting and describes determinantal sampling procedures. These procedures use gradient or Fisher information terms and update the matrix inverse efficiently.

A.5 Optimality of Greedy

  • For the trace objective with standard-basis support, batch size B, and regularizer λ > 0, the forward greedy algorithm is optimal for any distribution p.The setup is the infinite-unlabeled-data setting; the appendix notes that the log-det objective has a similar optimality proof but is different from the trace objective.Source: §A.5 · pp. 15–16

B Determinantal Algorithms

  • The determinantal algorithms select points by maximizing a determinant using BADGE gradient embeddings or Fisher information, then efficiently update the inverse matrix with the Woodbury identity.The Fisher information is represented as VxVx⊤, while the BADGE embedding gx is effectively one column of Vx without the corresponding class-probability scaling.Source: B Determinantal Algorithms · p. 17

C Additional Experimental Results (Classification)

This appendix reports full learning curves for the deep-classification setting and the experimental conditions used to produce them. The curves span several datasets, architectures, and batch sizes, alongside pairwise distance matrices for selected experiments.

C Additional Experimental Results (Classification)

  • Experiments ran five random seeds on an NVIDIA Tesla P100, stopping when the dataset was labeled or runtime exceeded 14 days.Models were trained with Adam to at least 99% training accuracy per round; CIFAR-10 used data augmentation, with BAIT initialized at λ = 0.01 versus λ = 1 for other experiments.Source: C Additional Experimental Results… · p. 17

C.1 Learning curves

Related papers

Loading 2106.09675v2…