PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Early-Learning Correction for Segmentation from Noisy Annotations

Sheng Liu, Kangning Liu, Weicheng Zhu, Yiqiu Shen, Carlos Fernandez-Granda

arXiv:2110.03740v2cs.CVcs.LG

TL;DR

Pixel-level segmentation annotations are costly, and noisy-label learning has received less attention in segmentation than in classification. ADELE monitors learning by category to correct annotations and promote multiscale consistency; it outperforms standard approaches on human-like noisy medical segmentation and achieves state-of-the-art WSSS results on PASCAL VOC 2012.

  • Problem

    Compared with classification, comparatively fewer studies address segmentation from noisy labels, despite costly pixel-level annotations and annotation variation in applications such as medical imaging.

    Source: §1 · p. 1

  • Method

    ADELE monitors each category's IoU during training to detect memorization, adaptively correct annotations, and regularize prediction consistency across scales.

    Source: §1 · p. 2

  • Results

    ADELE avoids memorization and outperforms standard baselines on thoracic-organ segmentation with human-like synthesized noise, and achieves state-of-the-art WSSS results on PASCAL VOC 2012.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work identifies annotation-noise robustness as a crucial challenge for segmentation applications such as medicine.

    Source: §7 · p. 9

    ADELE's gains depend to some extent on initial annotation quality; highly structured noise may prevent early learning and make label correction unsuccessful.

    Source: §6 · p. 9

Abstract

from arXiv · show

Deep learning in the presence of noisy annotations has been studied extensively in classification, but much less in segmentation tasks. In this work, we study the learning dynamics of deep segmentation networks trained on inaccurately-annotated data. We discover a phenomenon that has been previously reported in the context of classification: the networks tend to first fit the clean pixel-level labels during an "early-learning" phase, before eventually memorizing the false annotations. However, in contrast to classification, memorization in segmentation does not arise simultaneously for all semantic categories. Inspired by these findings, we propose a new method for segmentation from noisy annotations with two key elements. First, we detect the beginning of the memorization phase separately for each category during training. This allows us to adaptively correct the noisy annotations in order to exploit early learning. Second, we incorporate a regularization term that enforces consistency across scales to boost robustness against annotation noise. Our method outperforms standard approaches on a medical-imaging segmentation task where noises are synthesized to mimic human annotation errors. It also provides robustness to realistic noisy annotations present in weakly-supervised semantic segmentation, achieving state-of-the-art results on PASCAL VOC 2012. Code is available at https://github.com/Kangningthu/ADELE

1. Introduction

Pixel-level annotation is costly, and segmentation from noisy labels has received less study than classification. ADELE exploits category-specific learning dynamics to correct annotations and improve robustness, with reported gains in medical segmentation and state-of-the-art WSSS results on PASCAL VOC 2012.

1. Introduction

  • Pixel-level annotations are costly to gather and may require domain expertise, while medical segmentation annotations can vary across readers.The paper identifies noisy annotation learning as an important practical topic.Source: §1 · p. 1
  • Segmentation networks first fit clean annotations and later memorize false ones, but these stages occur at different times across categories because pixel-wise labels are imbalanced.The paper also notes that segmentation noise may be ubiquitous across examples, unlike methods that assume some samples are completely clean.Source: §1 · pp. 1–2
  • ADELE monitors each category's IoU curve to detect the start of memorization, adaptively corrects annotations, and adds multiscale consistency regularization.The correction exploits early learning for individual classes, while the regularization promotes spatial consistency.Source: §1 · p. 2
  • In WSSS, classification models generate pixel-level annotations from weak signals, and ADELE is applied to the segmentation-training stage where those annotations are noisy.The described pipeline generates annotations using variations of Class Activation Maps and post-processing.Source: §1 · p. 2
  • ADELE achieves state of the art on PASCAL VOC 2012 for WSSS and outperforms standard baselines on thoracic-organ segmentation with human-like synthesized annotation errors.The paper also reports consistent improvement when ADELE is combined with several existing pixel-level annotation methods.Source: §1 · p. 2

2. Methodology

Noisy-label segmentation networks first learn to correct some wrongly labeled pixels, then increasingly fit the incorrect annotations, with both phases occurring at different speeds across categories. ADELE uses category-specific training dynamics to time annotation correction and adds consistency regularization across scales.

2.1. Early learning and memorization in segmentation from noisy annotations

  • Early-learning IoU rises and later falls as IoU with incorrect labels grows, with both dynamics occurring at different speeds across semantic categories.The pattern is reported for SegTHOR and also observed for WSSS.Source: §2.1 · pp. 2–3, Figure 3 · p. 3
  • After early learning, models recover some missing or incorrect regions, but later memorize the noisy annotations and lose those corrections.This pattern is described for medical-imaging noise and WSSS annotations.Source: §2.1 · p. 3, Figure 4 · p. 4

2.2. Adaptive label correction based on early-learning

  • ADELE corrects categories at different times because correcting too early can degrade annotation quality and a single correction time can be suboptimal.It detects deceleration in training IoU against noisy annotations, using a fitted curve and the relative derivative change to choose correction timing by category.Source: §2.2 · pp. 3–5
  • ADELE uses confident model predictions to update annotations, with correction restricted to predictions above confidence threshold τ = 0.8.The relative-change threshold is r = 0.9; correction is applied at every subsequent epoch after its trigger.Source: §2.2 · p. 5

2.3. Multiscale consistency

  • Multiscale consistency regularizes predictions from rescaled copies of an input toward their average, and the combined method further improves performance on medical-imaging and WSSS tasks.The experiments use downscaling ×0.7, no scaling, and upscaling ×1.5; the consistency term applies only when the average prediction's maximum entry exceeds ρ = 0.8.Source: §2.3 · pp. 5–6, Figure 6 · p. 6

3. Related work

Prior noisy-label work has focused more on classification than segmentation. ADELE instead studies segmentation learning dynamics and adapts label correction to category-specific early-learning.

3. Related work

  • Classification research established early-learning followed by memorization for noisy labels, while ADELE reports these dynamics vary by semantic category in segmentation.ADELE uses this category variation for class-adaptive correction.Source: §3 · p. 6
  • Prior noisy-segmentation methods use setting-specific information such as specialized architectures, dataset knowledge, or completely clean samples [63],,, [34].ADELE instead studies learning dynamics and exploits early-learning for label correction.Source: §3 · p. 6
  • WSSS methods refine classification-generated pixel annotations, but ADELE adapts to their remaining noise; unlike seeded region-growing, it neither grows labels nor requires a foreground estimate or accurate seed annotations,.The cited WSSS pipeline generates pixel-level annotations from a classification model.Source: §3 · pp. 6–7

4. Segmentation on Medical Images with Annotation Noise

ADELE improves segmentation trained with synthetic annotation noise across corruption levels. The experiments compare class-adaptive correction and multi-scale consistency with a multi-scale-input baseline.

4. Segmentation on Medical Images with Annotation Noise

  • The SegTHOR experiment uses 2D slices of 3D CT scans and simulates annotation noise with random dilation and erosion of every training annotation.The main noisy-annotation setting has mIoU 0.6 against the ground truth; validation and test annotations remain uncorrupted.Source: §4 · p. 7
  • ADELE outperforms the baseline at all three evaluation epochs, while correcting labels for all classes simultaneously is detrimental.The table reports Best Val, Last Epoch, and Max Test mIoU.Source: Table 1 · p. 7
  • ADELE improves performance across a wide range of corruption levels, with substantial gains at moderate noise levels.Noise severity is measured by the mIoU of noisy annotations; lower mIoU means higher noise.Source: Figure 7 · p. 7, §4 · p. 7
  • Multi-scale inputs outperform original-scale-only training, spatial consistency further improves performance, and combining either with label correction substantially improves results.ADELE combines label correction with the proposed regularization and achieves the best performance.Source: §4 · p. 7

5. Noisy Annotations in Weakly-supervised Semantic Segmentation

ADELE is applied to weakly-supervised segmentation pipelines using noisy pixel labels generated from image-level supervision. It improves several WSSS baselines and achieves state-of-the-art results with SEAM and ICD on PASCAL VOC 2012.

5. Noisy Annotations in Weakly-supervised Semantic Segmentation

  • The method is tested in a WSSS pipeline where pixel-level annotations generated from image-level supervision train a segmentation network.The PASCAL VOC 2012 evaluation uses annotations generated by AffinityNet, SEAM, and ICD.Source: §5 · pp. 7–8
  • Class-adaptive label correction performs best when combined with multi-scale consistency regularization.Table 2 reports last-epoch validation mIoU for SegTHOR and PASCAL VOC 2012.Source: Table 2 · p. 8
  • ADELE substantially improves AffinityNet, SEAM, and ICD on PASCAL VOC 2012 validation and test sets; ADELE with SEAM or ICD achieves state-of-the-art performance.The comparison is reported in Table 3.Source: §5 · p. 8, Table 3 · p. 8
  • ADELE improves most SEAM categories, but gains are limited for poorly performing baselines such as chair and bike, whose errors are highly structured.Examples include bike predictions that miss inner structure and chairs misclassified as sofas.Source: §5 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8

6. Limitations

ADELE's effectiveness depends partly on initial annotation quality, and highly structured errors may prevent the early-learning signal needed for label correction.

6. Limitations

  • Poor initial annotations can yield only marginal improvement or even negative impact; highly structured noise may prevent early learning and make correction unsuccessful.The cited examples include coarsely segmented bicycles and chairs consistently mislabeled as sofas.Source: §6 · p. 9

7. Conclusion

The conclusion reports ADELE's results on thoracic-organ segmentation and WSSS, and the accompanying appendix supplies further results, illustrations, implementation details, and ablations.

7. Conclusion

  • ADELE improves robustness on thoracic-organ segmentation with noise resembling human annotation errors and achieves state-of-the-art WSSS results on PASCAL VOC 2012.The authors also call for further study of early learning and memorization beyond classification.Source: §7 · p. 9

Appendix for “Adaptive Early-Learning Correction for Segmentation from Noisy Annotations”

Licence for the dataset

  • The paper notes that PASCAL VOC images follow Flickr terms of use and that SegTHOR data use follows its data-use agreement.Source: Licence for the dataset · p. 12

Appendix for “Adaptive Early-Learning Correction for Segmentation from Noisy Annotations”

  • Additional PASCAL VOC illustrations show early learning and memorization progressing at different speeds across semantic categories.IoU on incorrectly labeled pixels first rises during early learning and later falls as memorization occurs.Source: Appendix for “Adaptive Early-Learning… · p. 12, Figure 10 · p. 13, Figure 11 · p. 14
  • A supplementary figure reports that multi-scale consistency regularization produces more accurate corrected annotations on PASCAL VOC.Source: Figure 12 · p. 15
  • The appendix illustrates category-specific correction timing, generally beginning near the end of early learning, with exceptions including boat and motorbike.The timing is based on the relative slope change of a fitted IoU curve.Source: Figure 13 · p. 15, Figure 14 · p. 16
  • The appendix includes additional segmentation examples and illustrates how structured errors, such as bicycle and chair annotation errors, may hinder label correction.The examples compare SEAM with SEAM+ADELE and show a possible limitation of the correction approach.Source: Figure 15 · p. 17, Figure 16 · p. 18, Figure 17 · p. 19

A. Additional Experimental Results

This section adds supplementary examples and results.

A. Additional Experimental Results

WSSS dataset (PASCAL VOC 2012)

Supplementary PASCAL VOC analyses show that early learning and memorization occur at different speeds across classes, and that multiscale consistency improves corrected annotations. Visual comparisons describe ADELE’s segmentation changes and flag highly structured noise as a boundary for label correction.

WSSS dataset (PASCAL VOC 2012)

  • Early learning occurs for most PASCAL VOC classes at different speeds, with correction beginning near its end for most categories.The passage describes supplementary learning curves and their zoomed-in and curve-fitting illustrations.Source: WSSS dataset (PASCAL VOC 2012) · p. 22
  • Multiscale-consistency regularization leads to more accurate corrected annotations on PASCAL VOC.Source: WSSS dataset (PASCAL VOC 2012) · p. 22
  • ADELE reduces false positives in some examples and produces more complete segmentations in others.The examples include boat, person, sofa, bottle, TV, cat, dog, bird, bus, and horse.Source: WSSS dataset (PASCAL VOC 2012) · p. 22
  • Highly structured annotation noise may prevent early learning, leaving that phase unavailable for label correction.Source: WSSS dataset (PASCAL VOC 2012) · p. 22

More visual examples for SegTHOR

A.2. Additional Results

Additional results report ADELE’s performance against external-saliency methods and across categories and noise levels. Combined with NSROM, ADELE reaches state-of-the-art WSSS results with a ResNet segmentation backbone.

A.2. Additional Results

  • 71.6 validation mIoU and 72.0 test mIoU are reported for ADELE+NSROM, state-of-the-art WSSS results with a ResNet segmentation backbone.ADELE using initial labels from SEAM and ICD also achieves state-of-the-art performance without external saliency models, on par with or slightly better than methods relying on them.Source: §A.2 · p. 22
  • ADELE performs substantially better than SEAM on most PASCAL VOC semantic categories and on average.Table 5 gives category-wise IoU comparisons on the validation set.Source: §A.2 · p. 22, Table 5 · p. 23
  • ADELE improves SegTHOR model performance across different noise levels.Table 6 reports test mIoU for the baseline and ADELE, using the model with the best validation mIoU.Source: §A.2 · p. 23, Table 6 · p. 23

B. Implementation details

Implementation details specify ADELE’s hyperparameters and training procedures for SegTHOR and PASCAL VOC. For VOC, label correction uses predictions from an unaugmented evaluation at each epoch because random training augmentations may not be invertible.

B.1. Hyperparameters

  • The hyperparameters λ, ρ, r, and τ control consistency strength, consistency confidence, curve-fit correction timing, and pixel correction confidence, respectively.Consistency regularization applies when the maximum prediction probability in the averaged output exceeds ρ; pixels are corrected only above τ.Source: §B.1 · p. 23

B.2. Medical segmentation with simulated noise: SegTHOR

  • SegTHOR noise synthesis uses mask dilation and erosion for over-annotation and under-annotation, respectively.Over-annotation assigns surrounding background pixels to an organ; under-annotation assigns edge foreground pixels to background.Source: §B.2 · p. 23
  • The SegTHOR correction algorithm averages network outputs across rescaled inputs and uses category-specific IoU-curve thresholds and confidence thresholds to correct annotations.The pseudocode describes correcting selected pixels to the model’s hard predicted labels and returning corrected annotations.Source: §B.2 · p. 24
  • SegTHOR experiments randomly choose the type and degree of synthetic noise for each example and train UNet for 100 epochs with batch size 5.Training uses SGD with a learning rate of 0.01; testing uses single-scale inputs without augmentation.Source: §B.2 · p. 24
  • For medical-image label correction, predictions are taken during training at every iteration and converted to hard labels.The images receive no augmentation in this setting.Source: §B.2 · p. 24

B.3. Weakly Supervised Semantic Segmentation: PASCAL VOC 2012

  • PASCAL VOC initial pixel-level annotations come from AffinityNet, SEAM, or ICD, and segmentation uses DeepLabv1 with a ResNet38 backbone.The training setup follows cited prior settings and initializes the backbone from an ImageNet-pretrained classification model.Source: §B.3 · p. 24
  • VOC testing follows the SEAM inference pipeline, including multiscale inference and CRF.Source: §B.3 · p. 25
  • VOC label correction uses predictions from unaugmented inputs evaluated at each epoch because random augmentations such as cropping may not be invertible.The model outputs are converted to hard labels for correction.Source: §B.3 · p. 25

C.1. Medical segmentation with simulated noise: SegTHOR

On SegTHOR, iteration-based label correction slightly improves best test mIoU and substantially improves last-epoch mIoU over epoch-based correction. Across SegTHOR and PASCAL VOC, ADELE is robust to τ, while extreme r values undermine correction.

C.1. Medical segmentation with simulated noise: SegTHOR

  • ADELE computes correction outputs either after each training iteration or at the end of each epoch.The SegTHOR ablation compares these two schedules for model-output-based label correction.Source: §C.1 · p. 25
  • Iteration-based label correction achieves slightly better best test mIoU and substantially better last-epoch mIoU than epoch-based correction on SegTHOR.Both options improve results over the baseline; the authors suggest iteration-based correction is more effective at preventing memorization.Source: §C.1 · p. 25, Table 8 · p. 25
  • On SegTHOR, very small r values trigger correction before the network is well trained, while very large r values can prevent correction before training stops; results are robust to τ.The authors report similar r-value behavior on PASCAL VOC, and Figure 21 reports test mIoU results for the τ ablation on SegTHOR.Source: §C.1 · p. 26, Figure 21 · p. 26

C.2. Weakly Supervised Semantic Segmentation: PASCAL VOC

  • On PASCAL VOC, very small r values trigger premature correction and harm generalization, while very large r values can leave performance similar to using no correction; results are robust to τ.Figure 22 reports validation mIoU at the last training epoch for the r and τ ablation.Source: §C.2 · p. 26, Figure 22 · p. 26
Loading 2110.03740v2…