PaperlayerView exact revision on arXiv

Source-linked AI summary

SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios

Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, Rishabh Iyer

arXiv:2107.00717v2cs.LGcs.CV

TL;DR

Active learning seeks informative labels efficiently, but existing methods struggle with realistic data containing rare classes, redundancy, and out-of-distribution examples. SIMILAR applies submodular information measures in a unified framework for these settings. The paper reports gains over existing baselines across realistic image-classification scenarios.

  • Problem

    Existing active-learning methods do not work well with realistic unlabeled data containing rare classes, redundancy, or out-of-distribution examples.

    Source: §1 · pp. 1–2

  • Method

    SIMILAR uses submodular information measures and selects query and conditioning sets to adapt the framework to different scenarios.

    Source: §1.2 · pp. 2–3, §2 · p. 3, §3 · p. 4

  • Results

    SIMILAR's LOGDET and FL variants yield approximately 5%–18% gains over existing baselines, particularly for rare classes, and approximately 5%–10% in OOD scenarios.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The paper presents SIMILAR as a common framework for standard active learning and the rare-class, OOD, and redundancy settings.

    Source: §1.2 · p. 3

    The approach depends on good feature representations, and its functions perform less well in early selection stages.

    Source: G Societal Impacts and Limitations · p. 23

Abstract

from arXiv · show

Active learning has proven to be useful for minimizing labeling costs by selecting the most informative samples. However, existing active learning methods do not work well in realistic scenarios such as imbalance or rare classes, out-of-distribution data in the unlabeled set, and redundancy. In this work, we propose SIMILAR (Submodular Information Measures based actIve LeARning), a unified active learning framework using recently proposed submodular information measures (SIM) as acquisition functions. We argue that SIMILAR not only works in standard active learning, but also easily extends to the realistic settings considered above and acts as a one-stop solution for active learning that is scalable to large real-world datasets. Empirically, we show that SIMILAR significantly outperforms existing active learning algorithms by as much as ~5% - 18% in the case of rare classes and ~5% - 10% in the case of out-of-distribution data on several image classification tasks like CIFAR-10, MNIST, and ImageNet. SIMILAR is available as a part of the DISTIL toolkit: "https://github.com/decile-team/distil".

1 Introduction

Active learning seeks to reduce labeling costs, but existing methods have been tested mainly on clean, balanced data rather than the imbalanced, redundant, and out-of-distribution data common in real-world settings. SIMILAR proposes a unified framework for standard active learning and these realistic scenarios.

1 Introduction

  • Real-world data can contain rare classes, redundancy, and out-of-distribution examples, while existing active-learning methods struggle with these conditions.The introduction cites cancer imaging, video-frame datasets, and unlabeled data irrelevant to the task as examples.Source: §1 · pp. 1–2

1.1 Related Work

  • Earlier approaches address diversity or biased data with limitations including failures across selection rounds, computational expense, tuning demands, or unrealistic data assumptions.The related work notes that Sinha et al.'s approach is computationally expensive and requires extensive tuning, while FISHER assumes a large labeled validation set and balanced unlabeled pretraining data.Source: §1.1 · p. 2

1.2 Our Contributions

  • SIMILAR uses submodular information measures as a common framework for standard active learning and realistic settings, including rare classes, OOD data, and redundancy.The contributions study different SIM function instantiations across these scenarios and report improvements on imbalanced and OOD image-classification data.Source: §1.2 · pp. 2–3

2 Background

The background defines submodular functions and three related information measures, then shows how their relationships support a unified active-learning framework.

2 Background

  • A submodular function assigns a score to a set with diminishing returns, and cardinality-constrained maximization admits a 1 − 1/e guarantee.Variants of greedy algorithms can maximize a submodular function in near-linear time.Source: §2 · p. 3
  • SMI scores similarity between a query set and a selected set, SCG measures gain over a conditioning set, and SCMI jointly models query similarity and dissimilarity from that set.SMI favors points similar to the query set while remaining diverse; SCG favors points dissimilar to the conditioning set while remaining diverse.Source: §2 · p. 3
  • SCMI unifies submodular functions, SMI, and SCG through choices of query set Q and conditioning set P.Setting Q to U and P to the empty set gives a submodular function; setting P to the empty set gives SMI, while setting Q to the empty set gives SCG.Source: §2 · p. 3, Table 1 · p. 3
  • SIM acquisition functions include Facility Location, Graph Cut, and Log Determinant variants with named SMI, SCG, or SCMI forms.Facility Location also has FLQMI and FLVMI variants; the table caption identifies the functions' applications by choices of Q and P.Source: §2 · p. 3, Table 1 · p. 3

3 SIMILAR: Our Unified Active Learning Framework

SIMILAR unifies active learning by deriving acquisition functions from submodular information measures and choosing query and conditioning sets for the scenario. It extends from standard selection to rare classes, redundancy, OOD, combined settings, and scalable implementation.

3 SIMILAR: Our Unified Active Learning Framework

  • SIMILAR uses relationships among submodular information measures to choose scenario-specific query and conditioning sets for a unified acquisition framework.The framework computes gradient-based similarities and optimizes a submodular conditional mutual information function.Source: §3 · pp. 3–4

3.1 Standard Active Learning

  • Standard active learning is the special case with Q=U and P=∅, reducing SCMI to a submodular function over the unlabeled set.The approach is closely related to BADGE's selection based on diverse gradients.Source: §3.1 · pp. 4–5

3.2 Rare Classes

  • A held-out rare-class query set R guides selection toward relevant, diverse samples through the SMI function If(A; R).The text says the validation set must be very small for this setting to be realistic.Source: §3.2 · p. 5

3.3 Redundancy in Unlabeled Data

  • Conditioning on the labeled set L with SCG targets redundancy across batches, which within-batch diversity alone may not prevent.BADGE is cited as an example of a method that enforces diversity only within the current batch.Source: §3.3 · p. 5

3.4 Out of Distribution Data

  • For OOD data, SIMILAR conditions on labeled OOD points O and queries labeled in-distribution points I to select ID samples while avoiding OOD points.The method adds an OOD class and ignores its prediction at test time.Source: §3.4 · pp. 5–6

3.5 Multiple Co-occurring Realistic Scenarios

  • SCMI combines rare-class targeting with OOD avoidance (Q=R, P=O) or redundancy avoidance (Q=R, P=L−R~).For redundancy plus OOD, the text proposes FLCMI with P=O∪I and an exponential-kernel penalty on highly similar ID samples.Source: §3.5 · p. 6

3.6 Realizing Realistic Scenarios in Applications

  • Rare classes or OOD points may be identified from the initial labeled set, after standard active-learning rounds, deployment errors, or domain knowledge.Until such scenarios are discovered, the authors suggest standard active learning with a diversity-based function such as LOGDET.Source: §3.6 · pp. 6–7

3.7 Scalability and Computational Aspects of SIMILAR

  • Scalability depends on kernel construction and greedy selection; FLQMI uses an n×q kernel and O(nqB) greedy selection, linear in n.Partitioning reduces kernel construction from O(n^2) to O(n^2/p) for selected functions; last-layer gradients make cosine-similarity kernels tractable for deep models.Source: §3.7 · p. 7

4 Experimental Results

Experiments test SIMILAR across rare classes, redundancy, and unlabeled OOD settings; its submodular acquisitions outperform baseline methods in these scenarios.

4 Experimental Results

  • The experiments compare SCMI-based acquisitions with uncertainty-, diversity-based, and random baselines across the three realistic scenarios.The evaluation also examines selection behavior and includes standard active learning and additional scenarios outside this section.Source: §4 · p. 7

4.1 Rare Classes

  • ≈5–10% overall accuracy and ≈10–18% average rare-class accuracy gains over uncertainty and diversity baselines; LOGDETMI and FLQMI outperform relevance-only GCMI.On ImageNet, FLQMI performs best among the reported SMI variants.Source: Figure 3 · p. 8, §4.1 · p. 9

4.2 Redundancy

  • ≈3–5% higher performance toward later rounds, with FLCG and LOGDETCG selecting more unique points than baselines, including BADGE.The authors report that conditioning on the updated labeled set is required in batch active learning.Source: Figure 4 · p. 9, §4.2 · p. 9

4.3 Out-Of-Distribution

  • ≈5–10% gains over existing AL approaches on ID accuracy; SCMI beats SMI by 2–3% in later rounds and has lower variance.SCMI also selects more ID points than SMI and other baselines.Source: Figure 5 · p. 10, §4.3 · p. 10

5 Conclusion

SIMILAR applies submodular information functions to active learning in rare-class, redundancy, and OOD scenarios. Reported gains vary by setting, and the approach depends on good representations for computing similarity.

5 Conclusion

  • SIMILAR is a unified active learning framework using submodular information functions, evaluated in rare-class, redundancy, and OOD scenarios.The reported real-world experiments use MNIST, CIFAR-10, and ImageNet.Source: §5 · p. 10
  • ≈5–18% gains over existing baselines in rare-class scenarios and ≈5–10% in OOD scenarios are reported across the experiments.The conclusion highlights LOGDET and FL variants among the functions yielding these gains.Source: §5 · p. 10
  • SIMILAR depends on good representations to compute similarity.Source: §5 · p. 10

Supplementary Material for SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios

The supplement analyzes costs for submodular information functions and describes partitioning unlabeled data into chunks to scale acquisition.

A.1 Computational complexity for selection using each function in SMI and baselines

  • FLQMI has O(n|Q|) kernel construction and optimization costs, while FLVMI, FLCG, and FLCMI involve quadratic kernel or optimization costs.The listed costs depend on the query or conditioning set sizes for the conditional variants.Source: §A.1 · p. 15
  • LogDetMI requires O(n^2) kernel computation and storage, with stochastic-greedy optimization of ~O(B^2n); conditional variants add P and Q to their costs.Source: §A.1 · p. 15
  • GCMI uses O(n|Q|) kernel construction and stochastic-greedy optimization costs.Source: §A.1 · p. 15
  • The complexity analysis assumes stochastic greedy; naive or lazy greedy has worst-case cost a factor B larger.The stochastic-greedy bound omits log factors; constructing and optimizing FL, LogDet, and GC functions can be derived from their CG versions with P = ∅.Source: §A.1 · p. 15

A.2 Details on Partitioning Approach

  • Partitioning divides the unlabeled set into p chunks, acquires a proportional share of the batch from each, and combines those selections.Keeping n/p roughly constant makes the stated complexity linear in n; ImageNet acquisition used 50 chunks because full-set selection exhausted compute resources.Source: §A.2 · p. 15

B.1 Datasets description in each scenario

The supplement documents dataset scenarios, training and comparison procedures, licenses, and baseline code; it also reports standard active learning results.

B.1 Datasets description in each scenario

  • The dataset suite includes MNIST, CIFAR-10, SVHN, and down-sampled ImageNet, with simulated rare classes, duplicated points, and OOD-heavy unlabeled pools.Source: §B.1 · pp. 16–17

B.2 Experimental setup

  • Experiments use SGD with cosine annealing, LeNet for MNIST and ResNet18 for other datasets, training up to 150 epochs or 99% accuracy.All experiments ran on a single V100 GPU.Source: §B.2 · p. 17

B.3 Details on computation of penalty matrix

  • The penalty matrix records the fraction of training rounds in which one selection algorithm significantly beats another, using two-tailed t-tests.Rows with high values indicate stronger performance; columns with high values indicate weaker performance.Source: §B.3 · p. 17

B.4 Licensing details

  • Dataset licenses include MIT, CC BY-SA 3.0, CC0, and ImageNet's custom research, non-commercial license.Source: §B.4 · p. 17

B.5 Baselines and Code

  • Baseline implementations draw on author code for BADGE and FISHER and the DISTIL repository for methods such as entropy sampling and CORESET.The experiments also use PyTorch and CORDS for gradient computations.Source: §B.4 · p. 17, §B.5 · p. 18

C Results with Standard Active Learning

D Additional Experiments and Takeaways for Active Learning with Rare Classes

Across rare-class experiments, LOGDETMI and FLQMI generally lead, with LOGDETMI strongest under extreme imbalance; FISHER can be more variable, while FLQMI is more scalable.

Additional rare-class experiments

Variance comparison

Penalty matrix

  • LOGDETMI and FLQMI had the smallest penalty-matrix column sums and highest row sums for rare-class and overall accuracy.The matrices combine MNIST and CIFAR-10 results at ρ = 20, B = 125; the sums indicate they are less often significantly outperformed and more often significantly outperform other approaches.Source: Figure 9 · p. 20, D Additional Experiments and Takeaways… · p. 21

Experimental caveat

Takeaways

E Additional Experiments and Takeaways from Active Learning with Redundancy

Under redundancy, the CG functions outperform baselines, including BADGE, and LOGDETCG leads FLCG; experiments cover multiple redundancy levels and datasets.

Additional experiments

Penalty matrix

Experimental takeaways

F Additional Experiments and Takeaways for Active Learning with OOD Data

In OOD experiments, CMI functions are more stable and generally stronger in later rounds, while SMI functions remain comparable and are also considered good choices.

Penalty matrix

Takeaways

  • CMI functions had the smallest variance and generally outperformed SMI counterparts in later rounds; SMI functions, particularly LOGDETMI and FLQMI, were often comparable and remained good OOD choices.Source: F Additional Experiments and Takeaways… · p. 23

G Societal Impacts and Limitations

SIMILAR could amplify biases when mining large datasets, but the authors also identify fair active learning on underrepresented slices as a potential positive use; they note graph-based functions and feature dependence as limitations.

Limitations

  • The MI functions are graph-based, and all but FLQMI have quadratic complexity; partitioning reduces cost at a performance cost.The authors also note dependence on good features and weaker performance in early stages.Source: G Societal Impacts and Limitations · p. 23

Societal impacts

  • Using SIMILAR to mine large datasets could perpetuate or amplify data biases, while rare-class or rare-slice experiments may support fair active learning for underrepresented groups.The authors propose studying applications such as improving performance on underrepresented Asian women in face, gender, and age recognition.Source: G Societal Impacts and Limitations · p. 23

H Experiments on Real-world Medical Dataset

On Pneumonia-MNIST, the framework is tested with pneumonia as a rare class. SMI functions outperform baselines on rare-class and overall accuracy.

H Experiments on Real-world Medical Dataset

  • With imbalance factor ρ = 20, the experiment treats pneumonia as the rare class in pediatric chest X-ray classification.The task distinguishes pneumonia from benign images.Source: H Experiments on Real-world Medical… · p. 23
  • Using misclassified data points in R was beneficial for acquiring subsets that lead to higher accuracy gains.Source: H Experiments on Real-world Medical… · p. 23
  • For active learning on real-world medical image classification, SMI functions outperform baselines by approximately 10%–12% in rare-classes accuracy and 8%–10% in overall accuracy, as reported in Figure 13.Source: Figure 13 · p. 24

I Experiments on Multiple Realistic Scenarios

The experiment evaluates active learning when rare classes and redundancy co-occur on CIFAR-10. SCMI and SMI functions perform better than the baselines.

I Experiments on Multiple Realistic Scenarios

Loading 2107.00717v2…