Source-linked AI summary
CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentation
Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, Carsten Rother
TL;DR
Semantic segmentation relies on large labeled datasets, but manual annotation is costly. CEREALS combines information measures with learned spatial cost estimates to select regions for annotation. On Cityscapes, it reaches 95% of full-training-set performance with 17% of the click-measured effort.
Problem
Semantic segmentation needs large labeled training sets, but manual annotation is costly while acquiring unlabeled data is usually easy.
Source: §1 · pp. 1–2Method
CEREALS selects informative, low-cost image regions by combining pixelwise information measures with spatial annotation-cost estimates from a learned cost model.
Source: §1 · p. 2, §3 · pp. 5–7Results
95% of full-training-set Cityscapes performance is achieved with 17% of the effort measured by clicks.
Source: §5 · p. 10Takeaways & Limitations
Combining information content and cost estimates is reported as a way to build new training datasets cost-effectively from scratch.
Source: §5 · p. 10Performance scaling to other network architectures remains unexamined because of the high computational demands of that evaluation.
Source: §5 · p. 10
Abstract
from arXiv · showhide
State of the art methods for semantic image segmentation are trained in a supervised fashion using a large corpus of fully labeled training images. However, gathering such a corpus is expensive, due to human annotation effort, in contrast to gathering unlabeled data. We propose an active learning-based strategy, called CEREALS, in which a human only has to hand-label a few, automatically selected, regions within an unlabeled image corpus. This minimizes human annotation effort while maximizing the performance of a semantic image segmentation method. The automatic selection procedure is achieved by: a) using a suitable information measure combined with an estimate about human annotation effort, which is inferred from a learned cost model, and b) exploiting the spatial coherency of an image. The performance of CEREALS is demonstrated on Cityscapes, where we are able to reduce the annotation effort to 17%, while keeping 95% of the mean Intersection over Union (mIoU) of a model that was trained with the fully annotated training set of Cityscapes.
1 Introduction
Semantic segmentation needs large, diverse labeled datasets, but manual annotation is costly. CEREALS uses active learning to select informative, low-cost regions; on Cityscapes, it reports 95% of full-training-set performance with 17% of the labeling effort measured by clicks.
1 Introduction
- CNN-based segmentation requires large amounts of diverse labeled training data to achieve satisfying results.The introduction also notes that performance seems to scale linearly with an exponential increase of training data.Source: §1 · p. 1
- 95% of full-training-set Cityscapes performance is reached with 17% of labeling effort measured by clicks.The comparison is with annotating the full Cityscapes training set.Source: Figure 1 · p. 2
- Manual annotation is costly, whereas acquiring large amounts of unlabeled data is usually easy.The paper identifies large-scale labeled training sets as a bottleneck for applying CNN models to new domains.Source: §1 · p. 2
- The relationship among annotation time, label noise, and model capacity had not been investigated for semantic segmentation, to the authors’ knowledge.The work focuses on cost-effectively creating reliable training data for high-performing CNNs.Source: §1 · p. 2
- CEREALS selects regions with high information content and low estimated annotation costs, using a learned cost-prediction CNN and spatial estimates.The framework seeks a minimal set of informative data while minimizing user interactions, measured by clicks.Source: §1 · p. 2
2 Related Work
Prior work addresses annotation costs through pre-training, weaker or additional data, synthetic data, interactive tools, and active learning. The paper distinguishes its approach by estimating spatial annotation costs and selecting informative image regions for dense segmentation.
2 Related Work
- Pre-training on large, less complex databases such as ImageNet is used when ground-truth data are scarce and can improve the final model’s discrimination ability.The passage describes this as a standard practice.Source: §2 · p. 3
- Weakly supervised methods use faster labels such as class presence or bounding boxes, while semi-supervised methods combine fine-grained labels with unlabeled data.The paper notes that supervised models outperform weakly supervised methods given sufficient data, and that semi-supervised methods have improved performance over labeled data alone.Source: §2 · p. 3
- Synthetic data methods pair generated images with dense annotations, but producing diverse, realistic scenes requires effort.The paper identifies scene diversity as important for satisfactory real-world results.Source: §2 · p. 3
- Interactive segmentation uses sparse user input to extract objects, and prior tools assist annotation, though cited approaches have limitations for dense semantic segmentation.The cited work includes methods with imprecision, no quantitative dense-semantic-segmentation results, or dependence on lidar sensors.Source: §2 · p. 3
- Prior segmentation active learning includes superpixel-based methods, while CEREALS scores fixed-size regions and estimates spatial annotation costs to find informative, cheap regions.The paper notes that informative regions need not match superpixel extents and that superpixels may fail to separate smoothly transitioning semantic regions.Source: §2 · pp. 3–4
3 Method
CEREALS selects image regions for annotation by combining pixel-level information with estimated annotation costs. Its batch-mode process uses region proposals to prioritize high-scoring, non-overlapping regions from the unlabeled pool.
3 Method
- The method builds pixel-level information maps from the segmentation model’s posterior probabilities, using entropy and vote entropy as information measures.Vote entropy measures disagreement among committee members; the committee is formed using the model’s dropout stochasticity rather than separately trained classifiers.Source: §3 · pp. 5–6
- A learned cost model estimates annotation effort as the number of clicks required to label each image, using click data from prior acquisition steps.The cost estimate is represented as a cost map for each image in the unlabeled pool.Source: §3 · pp. 6–7
- CEREALS selects image regions using both information content and estimated annotation costs, targeting a trade-off between annotation effort and model performance.It considers quadratic regions of varying sizes and prioritizes regions with high information and lower cost.Source: Figure 2 · p. 5, §3 · p. 7
- Sliding windows aggregate pixel-level information and estimated costs into region maps, which are scaled and fused using one of three functions.The third fusion function uses α to interpolate between region information and cost maps.Source: §3 · p. 7
- Non-maximum suppression selects high-scoring, fixed-size region proposals without overlap within an acquisition round; the top proposals are then added to the labeled pool.The batch-mode process repeats model training, information estimation, acquisition, and annotation until a target performance or labeling budget is reached, or the pool is exhausted.Source: §3 · pp. 4, 7
4 Results
CEREALS’ experiments compare region-selection strategies and annotation effort, then test combining entropy-based information with cost estimates. With the learned cost model, the method reaches 95% of full-training performance at c95 = 17.07%.
4 Results
- The experiments average five runs and start from a randomly selected seed set of 50 fully annotated images; the study also tests a learned CNN for regressing spatial cost information.The experiments include comparisons across region sizes and annotation effort.Source: §4 · p. 8
- c95 = 17.07% with the learned cost model’s multiplicative fusion of information and estimated costs.The model is trained on oracle feedback from previous acquisition steps.Source: Figure 5 · p. 9, §4 · p. 10
- p95 = 16.74% with entropy sampling in the region-based experiments, which improved as region sizes decreased.The authors suggest smaller regions allow querying higher-density areas.Source: §4 · pp. 8–9
- The reported p95 = 10.01% corresponded to c95 = 33.76%, showing that queried-label share and click effort can differ.The paper measures effort by clicks relative to the total clicks required to annotate Cityscapes.Source: §4 · p. 9
- c95 = 14.68% when entropy sampling uses ground-truth clicks as an upper bound on cost prediction.This result assumes the cost prediction network would always decide correctly.Source: Figure 5 · p. 9, §4 · p. 10
5 Conclusion
CEREALS combines information content and cost estimates for cost-effective semantic segmentation training. On Cityscapes, it reached 95% of full-training-set performance with 17% of the annotation effort, while cross-architecture scaling remains open.
5 Conclusion
- 17% of Cityscapes training-set click effort achieved 95% of full-training-set performance on Cityscapes.Effort is measured by clicks executed for annotating the training set.Source: §5 · p. 10
- CEREALS is a cost-effective active-learning method for semantic segmentation tailored to fully convolutional neural networks.Source: §5 · p. 10
- Combining information content and cost estimates is described as a powerful approach for cost-effectively building training datasets from scratch.Source: §5 · p. 10
- How CEREALS scales to architectures with varying representational power remains untested because such evaluation is computationally demanding.Source: §5 · p. 10
- The authors encourage sharing human annotation-cost ground truth for existing and future manually labeled computer-vision datasets.Source: §5 · p. 10
A.1 Implementation Details
The implementation masks unlabeled output locations during image-based training and specifies training choices for the segmentation and cost models. The segmentation model reports a Cityscapes full-training-set mIoU of 0.605.
A.1 Implementation Details
- Unlabeled output-space locations are masked so they contribute no loss when training either the segmentation model or the cost model.The implementation supports both output-space masking and cropping annotated regions while accounting for their input-space receptive field.Source: §A.1 · p. 18
- Training uses Adam with learning rate 0.0001, alpha 0.99, beta 0.999, batch size 1, and convergence after 10 epochs without application-loss improvement.Source: §A.1 · p. 18
- The full Cityscapes training set yielded mIoU 0.605 for the employed FCN8s-based model, while width multiplier 1.0 reproduced the original model's mIoU 0.65.Results are computed on the full Cityscapes validation dataset.Source: §A.1 · p. 18
- CEREALS can use any segmentation model producing posterior probability distributions, but the cost model would need adaptation or independence from that model.Source: §A.1 · p. 18
- The cost model replaces softmax with linear activation and minimizes mean squared error on ground-truth clicks clipped to [0,10].Click data is downscaled by a factor of 8, and the cost model has no final upsampling layer.Source: §A.1 · p. 18
A.2 Results
The results section reports experiments in Table 1, whose caption defines the annotation-effort measures and comparison basis. Figures 6 and 7 provide overviews of specified acquisition steps.
A.2 Results
- Table 1 reports p95 annotated pixels or c95 clicks needed for 95% of full-training-set mIoU; results are averaged over five repetitions.Source: Table 1 · p. 19, §A.2 · p. 19
- Figure 6 presents a detailed overview of CEREALS's first two acquisition steps.Source: Figure 6 · p. 20
- Figure 7 presents a detailed overview of CEREALS's acquisition steps three and ten.Source: Figure 7 · p. 21