PaperlayerView exact revision on arXiv

Source-linked AI summary

ViewAL: Active Learning with Viewpoint Entropy for Semantic Segmentation

Yawar Siddiqui, Julien Valentin, Matthias Nießner

arXiv:1911.11789v2cs.CVcs.LG

TL;DR

Semantic-segmentation annotation is costly, while conventional uncertainty sampling largely ignores geometric consistency in multi-view data. ViewAL selects labels using viewpoint entropy and superpixel-level uncertainty, achieving strong performance with less labeled data and reduced labeling time.

  • Problem

    High-quality ground-truth annotation is costly, and existing uncertainty-sampling techniques operate almost exclusively on single images despite widespread multi-view data.

  • Method

    ViewAL measures uncertainty from prediction inconsistency across views and combines viewpoint entropy with superpixel-based scoring and label acquisition.

  • Results

    ViewAL selects informative samples efficiently, while view entropy and view divergence each improve active-learning performance in ablation studies.

  • Takeaways & Limitations

    The unified strategy acquires labels for promising image regions while reducing labeling effort for semantic segmentation.

Abstract

from arXiv · show

We propose ViewAL, a novel active learning strategy for semantic segmentation that exploits viewpoint consistency in multi-view datasets. Our core idea is that inconsistencies in model predictions across viewpoints provide a very reliable measure of uncertainty and encourage the model to perform well irrespective of the viewpoint under which objects are observed. To incorporate this uncertainty measure, we introduce a new viewpoint entropy formulation, which is the basis of our active learning strategy. In addition, we propose uncertainty computations on a superpixel level, which exploits inherently localized signal in the segmentation task, directly lowering the annotation costs. This combination of viewpoint entropy and the use of superpixels allows to efficiently select samples that are highly informative for improving the network. We demonstrate that our proposed active learning strategy not only yields the best-performing models for the same amount of required labeled data, but also significantly reduces labeling effort. For instance, our method achieves 95% of maximum achievable network performance using only 7%, 17%, and 24% labeled data on SceneNet-RGBD, ScanNet, and Matterport3D, respectively. On these datasets, the best state-of-the-art method achieves the same performance with 14%, 27% and 33% labeled data. Finally, we demonstrate that labeling using superpixels yields the same quality of ground-truth compared to labeling whole images, but requires 25% less time.

1. Introduction

ViewAL addresses costly semantic-segmentation annotation by selecting informative samples using prediction inconsistencies across views and localized superpixel uncertainty. It combines viewpoint entropy, view-based criteria, and superpixel labeling to reduce effort while preserving informative selection.

  • Deep-learning systems require increasingly costly amounts of high-quality ground-truth data, motivating active learning to reduce labeling costs.
  • Combining viewpoint entropy with superpixels enables efficient selection of samples that are highly informative for improving the network.
  • ViewAL estimates uncertainty from inconsistencies in predictions for the same object across multiple viewpoints.The method refers to this measure as viewpoint entropy.
  • Its most-informative-view criterion uses KL divergence between prediction probability distributions across viewpoints.
  • Superpixel-based scoring and label acquisition target localized segmentation uncertainty while reducing labeling effort without sacrificing annotation quality.

2. Related Work

Related active-learning methods select uncertain, diverse, or model-changing samples, with uncertainty approaches particularly common for deep neural networks. Prior work has adapted these strategies to segmentation, but many methods were initially validated on classification or face computational limitations.

  • Active-learning strategies commonly select samples by uncertainty, diversity, or expected model change.
  • Uncertainty sampling queries examples the model is least certain about, whereas diversity methods seek representative instances and model-change methods seek labels that would alter the model most.
  • Uncertainty-based methods can query outliers, while expected model change has had limited success with deep neural networks because of computational complexity.
  • Softmax confidence, margin, entropy, Monte Carlo dropout, and ensembles provide uncertainty estimates for deep neural networks, although ensembles can require substantial memory and computation.
  • Prior active-learning research has adapted classification and probabilistic-model approaches to semantic segmentation, including regional selection with uncertainty and cost estimates.

3. Method

ViewAL iteratively trains a segmentation network, scores unlabeled superpixels using viewpoint entropy and divergence, selects informative regions, and acquires their labels until the budget is exhausted.

  • Method overview: ViewAL repeats network training, uncertainty estimation, superpixel selection, and oracle annotation until the labeling budget is reached or all data are labeled.The labeled and unlabeled datasets are updated after each acquisition round, and the network is retrained from scratch.
  • Uncertainty scoring: View entropy measures uncertainty from the average class distribution across viewpoints, while view divergence measures disagreement between the current view and predictions from other views.View entropy is shared by pixels in correspondence; divergence identifies views whose predictions differ most from other viewpoints.
  • Uncertainty scoring: The method computes pixel-wise class probabilities with MC dropout, then uses pose and depth to cross-project predictions among corresponding observations.The cross-projected distributions provide multiple viewpoint-specific predictions for each pixel.
  • Region selection: Superpixel scores are obtained by averaging the pixel-level view entropy and view divergence values within each region.Superpixels exploit localized segmentation structure and permit region-level annotation rather than labeling entire images.
  • Label acquisition: Only selected superpixels receive oracle labels; unlabeled superpixels in partially labeled images are assigned the ignore label during retraining.This produces images containing many labeled regions and remaining ignored regions.
  • Region selection: Selection first chooses the highest-view-entropy superpixel, then selects the highest-view-divergence superpixel among regions overlapping it by at least 50%.The overlapping set is removed from later consideration after selection, and the process continues until the requested equivalent of K images is selected.

4. Results

ViewAL achieves strong active-learning performance and labeling efficiency across three indoor datasets. Ablations show that viewpoint entropy, superpixel selection, MC dropout, and view divergence each contribute to the final method.

  • Active learning performance: ViewAL achieves 95% of maximum model performance with 7%, 17%, and 24% labeled data on SceneNet-RGBD, ScanNet, and Matterport3D, respectively.The closest method on ScanNet requires 27% labeled data to reach the same performance.
  • Active learning performance: On ScanNet, ViewAL reaches 27.7% mIoU with 17% labeled data, compared with 28.9% mIoU using the whole dataset.This corresponds to around 95% of whole-dataset performance.
  • Labeling effort: Superpixel labeling reduces annotation time by 25%, taking 202 minutes versus 271 minutes for labeling 50 equivalent images.The effort comparison uses a user study with 50 images and 2,000 corresponding superpixels.
  • Labeling effort: Superpixel selections contain fewer ground-truth classes than fixed windows, averaging 1.40 versus 1.83 unique classes per selection.The majority of superpixel selections contain only a single ground-truth label.
  • Ablation studies: View entropy alone outperforms random selection and softmax entropy, while superpixels, MC dropout, and view divergence progressively improve performance.Superpixels focus selection on high-scoring regions; MC dropout improves class-posterior estimates; view divergence selects the most discrepant view.

5. Conclusions

The paper concludes that ViewAL uses cross-view prediction inconsistency and superpixel-based regional selection to improve semantic-segmentation active learning while reducing labeling effort. The approach is evaluated on indoor datasets but is described as agnostic to the underlying multi-view representation.

  • 5. Conclusions: ViewAL measures uncertainty from prediction inconsistencies across views and acquires labels only for promising image regions through superpixels.The conclusion presents view entropy and regional superpixel selection as components of a unified strategy.
  • 5. Conclusions: The method reduces labeling effort by requesting annotations for selected superpixel regions rather than entire images.The conclusion links regional selection with acquiring labels for only the most promising areas.
  • 5. Conclusions: ViewAL is reported on widely used indoor datasets and is agnostic to the respective multi-view representation.The authors describe application to other computer-vision tasks as a promising research avenue, rather than an evaluated result.

A. Dataset Statistics

The experiments use subsets of three public RGBD datasets spanning synthetic indoor trajectories and real indoor scenes. These datasets provide multi-view observations with semantic annotations, while the subset choice limits computational cost.

  • Dataset Statistics: The study evaluates ViewAL on SceneNet-RGBD, ScanNet, and Matterport3D, all of which provide RGBD views with semantic annotations.SceneNet-RGBD is synthetic, whereas ScanNet and Matterport3D contain real indoor scenes.
  • Dataset Statistics: The datasets contain approximately 5M SceneNet-RGBD frames, 2.5M ScanNet views, and 200K Matterport3D views.ScanNet covers 1,513 real indoor scenes, and Matterport3D covers 90 real building-scale scenes.

B. Baseline Active Learning Methods

The paper compares ViewAL with uncertainty-, diversity-, and mixed-strategy active-learning baselines for semantic segmentation. These baselines differ in how they score uncertainty, representativeness, regional structure, or pseudo-labeling.

  • Baseline Active Learning Methods: RAND selects K unlabeled samples randomly during each active-selection iteration.It provides the random-selection baseline.
  • Baseline Active Learning Methods: CONF ranks images using summed per-pixel softmax confidence and selects the K least-confident samples.Confidence is based on the probability of the most probable class.
  • Baseline Active Learning Methods: MAR selects samples with the smallest difference between the top two softmax probabilities, whereas ENT sums per-pixel entropy across each image.Entropy incorporates probabilities from all classes, unlike confidence and margin.
  • Baseline Active Learning Methods: CEAL combines uncertainty measures with pseudo-labeling, adding high-confidence unlabeled samples whose softmax entropy is below a decreasing threshold.The implementation assigns pseudo-labels at pixel level for segmentation.
  • Baseline Active Learning Methods: MCDR estimates uncertainty with Monte Carlo dropout, while RMCDR applies regional scoring and CSET and MREP represent diversity-based selection.MREP combines uncertainty with representativeness; CSET minimizes the maximum distance to selected points.

C. Performance with Imperfect Depth and Pose

The method is evaluated when only RGB frames are available, using reconstructed scene geometry for frame associations. It still outperforms random sampling and is only slightly worse than using true depth and poses.

  • Experimental setup: With only RGB frames available, the evaluation reconstructs depth and camera parameters from scenes using COLMAP.The experiment uses five ScanNet scenes because scene reconstruction is time-consuming.
  • Results: The reconstructed-depth-and-pose variant outperforms the RAND baseline.
  • Results: The reconstructed-geometry variant performs only slightly worse than the variant using true depth and poses.

D. Comparison with baselines allowed to select superpixels

When competing methods are also allowed to select superpixels rather than windows or whole images, most methods benefit from superpixel selection.

  • Comparison: Most baseline methods benefit when they select superpixels instead of windows or entire images.

E. Handling non-static data

The method’s view-entropy and divergence computations require superpixel associations across frames, limiting the current experiments to static scenes. Extending these associations to dynamic scenes is identified as future work.

  • Constraint: View-entropy and divergence computation requires associating superpixels between frames.
  • Constraint: The current association procedure applies only to static scenes where objects do not change positions across frames.
  • Future direction: Future work could address dynamic scenes using optical flow estimates or keypoint descriptor matching.

F. Result Tables

The supplementary results report semantic-segmentation comparisons across three datasets and an ablation study isolating superpixels, MC dropout, and view divergence.

  • Benchmark tables: Tables 2–4 report semantic-segmentation performance in terms of mIoU on SceneNet-RGBD, ScanNet, and Matterport3D.
  • Ablation study: Table 5 reports an ablation study for the proposed method.
  • Ablation study: Adding superpixels substantially improves ViewAL(VE), while MC dropout and view divergence provide further improvements in sequence.
  • Ablation study: The final configuration, ViewAL(VE+Spx+MCDR+VD), adds view divergence to ViewAL(VE+Spx+MCDR).
Loading 1911.11789v2…