PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Aware Variational Adversarial Active Learning

Kwanyoung Kim, Dongwon Park, Kwang In Kim, Se Young Chun

arXiv:2002.04709v2cs.LGstat.ML

TL;DR

High labeling costs motivate active learning, but task-agnostic selection overlooks task structure while task-aware selection may underuse overall data distribution. TA-VAAL combines VAAL with normalized ranking-loss information through RankCGAN, and it outperforms state-of-the-art methods across balanced and imbalanced classification and semantic segmentation benchmarks.

  • Problem

    Active learning must balance task-aware difficulty information with task-agnostic use of overall labeled and unlabeled data under costly annotation.

  • Method

    TA-VAAL relaxes loss prediction into ranking-loss prediction and embeds normalized ranking-loss information into VAAL using RankCGAN.

  • Results

    TA-VAAL outperforms state-of-the-art active-learning methods across balanced and imbalanced classification benchmarks and Cityscapes semantic segmentation.

  • Takeaways & Limitations

    The framework combines task-aware model information with task-agnostic data-distribution structure while accommodating generic task learners.

Abstract

from arXiv · show

Often, labeling large amount of data is challenging due to high labeling cost limiting the application domain of deep learning techniques. Active learning (AL) tackles this by querying the most informative samples to be annotated among unlabeled pool. Two promising directions for AL that have been recently explored are task-agnostic approach to select data points that are far from the current labeled pool and task-aware approach that relies on the perspective of task model. Unfortunately, the former does not exploit structures from tasks and the latter does not seem to well-utilize overall data distribution. Here, we propose task-aware variational adversarial AL (TA-VAAL) that modifies task-agnostic VAAL, that considered data distribution of both label and unlabeled pools, by relaxing task learning loss prediction to ranking loss prediction and by using ranking conditional generative adversarial network to embed normalized ranking loss information on VAAL. Our proposed TA-VAAL outperforms state-of-the-arts on various benchmark datasets for classifications with balanced / imbalanced labels as well as semantic segmentation and its task-aware and task-agnostic AL properties were confirmed with our in-depth analyses.

1. Introduction

Active learning reduces annotation demands by selecting informative samples, but task-agnostic and task-aware methods each omit information used by the other. TA-VAAL combines data-distribution structure with task-related loss ranking to select difficult and influential samples.

  • High annotation cost limits deep learning applications, especially when labels require experts or lengthy processing.
  • Active learning addresses limited labeling budgets by selecting data for annotation that can improve performance.
  • Task-agnostic methods exploit input distribution to identify influential, representative samples, whereas task-aware methods model task dependence to find difficult samples.
  • Task-aware selection may rely on an inaccurate early-stage learner and does not directly model how labeled samples influence the full dataset.
  • TA-VAAL relaxes accurate loss prediction into loss ranking prediction and embeds normalized ranking-loss information into VAAL using RankCGAN.
  • The proposed method combines task-aware difficulty with task-agnostic influence and is reported to outperform state-of-the-art methods across classification settings and semantic segmentation.

2. Related Works

Related active-learning methods differ in whether selection uses task information or overall data distribution. TA-VAAL extends this combination by incorporating direct task-related ranking loss without restricting the task learner’s architecture.

  • Task-aware methods select uncertain or ambiguous samples using metrics such as decision-boundary distance, empirical risk, or disagreement.
  • Task-agnostic methods use labeled and unlabeled data to select diverse, representative samples from the broader data distribution.
  • SRAAL combines task-aware and task-agnostic approaches through an uncertainty indicator and unified labeled-unlabeled representation.
  • Unlike SRAAL, TA-VAAL incorporates direct task-related ranking loss and does not restrict the task learner to VAE-type networks with latent spaces.
  • TA-VAAL is designed to accommodate local task-related information alongside global data-distribution structure.

3. Method

TA-VAAL combines task-agnostic VAAL’s modeling of labeled and unlabeled data distributions with task-aware loss-ranking information. It embeds normalized ranking predictions into VAAL through a Ranker and conditional adversarial framework to guide sample selection.

  • Task loss prediction module as “Ranker”: TA-VAAL relaxes loss prediction from estimating accurate loss values to estimating relative loss rankings.The ranking objective uses pairwise comparisons rather than emphasizing absolute loss values.
  • Task loss prediction module as “Ranker”: The Ranker predicts normalized relative loss rankings that condition VAAL’s latent representation, decoder, and discriminator.This allows the latent space to be reshaped using task-related ranking information.
  • Task loss prediction module as “Ranker”: The ranking-loss training was empirically more stable and performed better than the original learning-loss approach.The paper attributes this comparison to the ablation study in Section 5.1.
  • Training pipeline: TA-VAAL trains a task learner, Ranker, conditional VAE, and discriminator using labeled and unlabeled data pools.The training pipeline initializes parameters for these components and updates them with task, ranking, VAE, and adversarial losses.
  • Proposed task-aware VAAL (TA-VAAL): Removing rank information reduces TA-VAAL to VAAL, preserving global data-distribution modeling while discarding task information.VAAL models the influence of labels across the overall distribution but is agnostic to the task labels.
  • Proposed task-aware VAAL (TA-VAAL): TA-VAAL addresses the limitation that individual loss estimates can be sensitive to noise and outliers by combining task information with data-distribution modeling.Its conditional GAN connects predicted loss rankings with representations learned from labeled and unlabeled pools.

4. Experimental Results

Experiments evaluate TA-VAAL on balanced and imbalanced image classification datasets and Cityscapes semantic segmentation. TA-VAAL consistently outperformed compared active-learning methods across sampling stages, including challenging imbalance settings.

  • Balanced classification: TA-VAAL was evaluated on balanced CIFAR10 and CIFAR100, with 1,000 initial labeled samples and 1,000 queried samples per stage.
  • Balanced classification: TA-VAAL outperformed other state-of-the-art methods at every CIFAR10 sampling stage.
  • Balanced classification: After 3k labeled samples, TA-VAAL substantially outperformed all compared methods on CIFAR100.
  • Imbalanced classification: TA-VAAL outperformed all other methods across modified CIFAR10 imbalance ratios, with larger improvement margins at the ×100 ratio.
  • Imbalanced classification: TA-VAAL substantially outperformed other state-of-the-art methods at every Caltech101 sampling stage in the naturally imbalanced setting.
  • Semantic segmentation: TA-VAAL outperformed all compared methods at every Cityscapes sampling stage on mean IoU.

5. Empirical Analyses

The ablations and analyses indicate that TA-VAAL benefits from ranking-based task information integrated with VAAL’s data-distribution modeling. Compared with VAAL, it selected more class-balanced and higher-loss samples, supporting more reliable acquisition.

  • Ablation studies: Learning loss_v2 performed substantially better at later active-learning stages and comparably at early stages to the original learning-loss method.The comparison used five trials and reported means with standard deviations.
  • Ablation studies: VAAL+learning loss substantially outperformed VAAL across stages but did not surpass the original learning-loss method at later stages.
  • Selected samples: TA-VAAL selected samples with high data class-count entropy on the severely imbalanced dataset, indicating more uniform class coverage.The analysis connects low entropy in learning-loss selection with lower performance at the 2k stage.
  • Selected samples: VAAL’s concentrated discriminator likelihood distribution made selection almost random, whereas TA-VAAL’s wider distribution made selection more reliable.TA-VAAL’s discriminator used latent values together with ranking information.
  • Selected samples: TA-VAAL selected samples with relatively high real loss, while task-agnostic VAAL selected samples across a wide range of real-loss values.Real loss represents task-model uncertainty, while unlabeled likelihood represents task-agnostic data distribution.

6. Conclusion

TA-VAAL combines task-agnostic distribution modeling with task-aware uncertainty from a generic task learner. Across balanced and imbalanced classification benchmarks and semantic segmentation, it outperformed state-of-the-art active-learning methods, while analyses supported both intended properties.

  • Conclusion: TA-VAAL combines VAAL’s labeled-and-unlabeled data-distribution modeling with task-aware model uncertainty from a generic task learner.The framework incorporates LPM and RankCGAN concepts into VAAL while relaxing loss prediction to ranking-loss prediction.
  • Conclusion: TA-VAAL outperformed state-of-the-art active-learning methods on balanced and imbalanced classification benchmarks and Cityscapes semantic segmentation.
  • Conclusion: In-depth analyses confirmed that TA-VAAL uses both task-aware and task-agnostic active-learning properties.

7. Supplementary Section

The supplementary material details TA-VAAL’s sampling algorithm, ablations, absolute performance curves, hyperparameters, larger-budget validation, and selected-image examples. Its sampling rule ranks unlabeled samples, embeds normalized ranking information with VAE representations, and selects a budget-sized acquisition set.

  • Supplementary contents: The supplementary document covers sampling-strategy summaries, ranking-loss comparisons, absolute accuracies and mIOU, larger-budget CIFAR100 results, hyperparameters, and selected images.
  • Active sampling algorithm: TA-VAAL predicts loss rankings for the unlabeled pool, normalizes them to preserve relative ranking, and embeds the information into the VAE latent space.
  • Active sampling algorithm: The acquisition operation selects b samples from the unlabeled pool using the discriminator’s comparison of ranking-conditioned latent representations.The algorithm’s goal is to collect b samples and update the labeled pool.
  • Active sampling algorithm: The method uses the discriminator output so that smaller values indicate greater likelihood of belonging to the unlabeled pool.
  • Supplementary analyses: The modified ranking loss was more stable and yielded better performance than the original learning loss in the reported ablation study.
  • Larger-budget validation: Under CIFAR100’s 5000 / 2500 initial-size / budget setting, TA-VAAL still outperformed other state-of-the-art methods at all stages.This setting matches the reported VAAL experiment configuration.
Loading 2002.04709v2…