PaperlayerView exact revision on arXiv

Source-linked AI summary

Incremental Learning for Semantic Segmentation of Large-Scale Remote Sensing Data

Onur Tasar, Yuliya Tarabalka, Pierre Alliez

arXiv:1810.12448v1cs.LGcs.CVstat.ML

TL;DR

Remote-sensing segmentation must accommodate incoming data and class labels when the full prior training set may not be available. The paper adapts one network using a frozen memory model and, depending on the data setting, stored prior samples or memory outputs. Across three datasets, the authors report learning new classes without catastrophic forgetting of old classes.

  • Problem

    Remote-sensing data and class annotations arrive over time, while storing all prior training data is not always feasible, making incremental learning without losing old-class performance important.

    Source: §I · p. 2

  • Method

    The approach adapts the network to new and old classes using a frozen prior network as memory, and preserves prior information with stored samples or memory outputs according to the data setting.

    Source: B. Adapting the Network to the New… · p. 4, C. Remembering From the Previous… · p. 5

  • Results

    On three datasets, the proposed approach achieves high performance for new classes without catastrophically forgetting old classes.

    Source: B. Contributions · p. 3, §IV · p. 13

  • Takeaways & Limitations

    The authors conclude that a network can add segmentation capabilities for new classes while retaining dense-labeling abilities for previously trained classes without the entire prior training set.

    Source: §IV · p. 13

    With fixed representation, newly added small-object classes such as cars can be learned poorly.

    Source: D. Experiments on the Benchmark Datasets · p. 11

Abstract

from arXiv · show

In spite of remarkable success of the convolutional neural networks on semantic segmentation, they suffer from catastrophic forgetting: a significant performance drop for the already learned classes when new classes are added on the data, having no annotations for the old classes. We propose an incremental learning methodology, enabling to learn segmenting new classes without hindering dense labeling abilities for the previous classes, although the entire previous data are not accessible. The key points of the proposed approach are adapting the network to learn new as well as old classes on the new training data, and allowing it to remember the previously learned information for the old classes. For adaptation, we keep a frozen copy of the previously trained network, which is used as a memory for the updated network in absence of annotations for the former classes. The updated network minimizes a loss function, which balances the discrepancy between outputs for the previous classes from the memory and updated networks, and the mis-classification rate between outputs for the new classes from the updated network and the new ground-truth. For remembering, we either regularly feed samples from the stored, little fraction of the previous data or use the memory network, depending on whether the new data are collected from completely different geographic areas or from the same city. Our experimental results prove that it is possible to add new classes to the network, while maintaining its performance for the previous classes, despite the whole previous training data are not available.

I. INTRODUCTION

The introduction frames remote-sensing segmentation as a large-scale mapping need complicated by changing data availability and class annotations. It reviews incremental-learning approaches and identifies dense labeling as an unaddressed setting in the cited work.

I. INTRODUCTION

  • High-resolution satellite data make large-scale, pixel-wise classification essential for generating and updating maps across application areas.The paper names agriculture, navigation, environmental management, urban monitoring, and mapping.Source: §I · p. 1
  • Prior segmentation studies often train and validate within one city, while the Inria dataset separates cities but labels only building and non-building.The paper notes that same-city train and validation data come from the same distribution.Source: §I · p. 1
  • Remote-sensing data arrive over time, class maps may differ across sources, and storing all prior training data is not always feasible.The introduction also describes annotations for different classes arriving sequentially for the same imagery.Source: §I · pp. 1–2

A. Related Work

  • Earlier incremental-learning methods grow network architectures, reuse selected old samples, or preserve important parameters to retain prior-task information.Other reviewed approaches generate synthetic old data, reconstruct features, or constrain parameter changes using importance estimates.Source: A. Related Work · p. 2
  • Knowledge-distillation methods have also been used to retain prior-task performance, but the reviewed work does not address incremental dense labeling.The cited approaches span sequential Bayesian updates and distillation-based losses.Source: A. Related Work · p. 3

B. Contributions

The paper studies incremental segmentation in two settings: classes arriving from different geographic locations and labels arriving sequentially for the same area. It evaluates the approach on remote-sensing datasets and reports learning new classes without catastrophic forgetting in one network.

B. Contributions

  • The study addresses both geographically separate data streams and sequential class annotations for the same geographic area.Luxcarta represents the different-location setting; Vaihingen and Potsdam are used for the same-area setting.Source: B. Contributions · p. 3
  • The cross-location setting is tested on data from different geographic areas than the training data, extending evaluation beyond building-only generalization.The paper contrasts its multi-class evaluation with the Inria dataset's building-only study.Source: B. Contributions · p. 3
  • On three datasets, the proposed approach achieves high performance for new classes without catastrophically forgetting old classes, using one network.The authors identify this as the first proposed solution for incremental semantic segmentation of remote-sensing data.Source: B. Contributions · p. 3

A. Network Architecture

The network is a U-net variant with a VGG16-based encoder, decoder, and center convolutional layers. It produces binary maps for class-specific tasks, with shared lower layers and class-specific final layers.

A. Network Architecture

  • The U-net variant uses the first 13 VGG16 convolutional layers as its encoder, alongside a decoder and two center convolutional layers.Skip connections concatenate encoder pooling outputs with the corresponding decoder deconvolution outputs.Source: A. Network Architecture · p. 3
  • Convolutions use 3 × 3 kernels and stride 1, while 2 × 2 pooling and deconvolution operations halve and double spatial dimensions, respectively.Except for the final convolution, convolutional and deconvolutional operations are followed by ReLU.Source: A. Network Architecture · p. 4
  • The network treats each class's binary segmentation as a task, sharing bottom-layer information while dedicating final layers to class outputs.At test time, sigmoid probabilities are thresholded at 0.5 to produce binary segmentation maps.Source: A. Network Architecture · p. 4

B. Adapting the Network to the New Training Data

Adaptation adds current classes while retaining the network’s previous-class predictions despite missing old-class annotations. A frozen memory network supplies those predictions, and the updated network is trained with classification and distillation losses.

B. Adapting the Network to the New Training Data

  • The current training data and the previous and current class sets are denoted Dcurr, Lprev, and Lcurr, with Lprev and Lcurr disjoint.Source: B. Adapting the Network to the New… · p. 4
  • The updated network learns current classes and fine-tunes previous-class segmentation on current data, although current annotations omit previous classes.Its output covers the union of previously learned and current classes.Source: B. Adapting the Network to the New… · p. 4
  • A frozen memory network supplies previous-class probability maps alongside current-class ground truth for training the updated network.The updated network shares the memory network’s structure except for its final classification layer, which gains filters for the current classes; soft probability maps are used to reduce propagated error.Source: B. Adapting the Network to the New… · p. 4
  • The adaptation loss adds classification loss to distillation loss, balancing current-class supervision against matching the memory network’s previous-class predictions.The classification term compares predictions with current ground truth, while the distillation term measures disparity from the memory network’s previous-class probabilities.Source: B. Adapting the Network to the New… · p. 5

C. Remembering From the Previous Training Data

Remembering uses samples from stored previous data when incoming data come from different geographic locations. A remembering loss trains the updated network on previous data, with class-aware patch selection intended to preserve less frequent classes.

C. Remembering From the Previous Training Data

  • For data arriving from different geographic locations, the method regularly reuses a small stored fraction of previous data to reduce overfitting to current data.The approach accounts for class imbalance when selecting patches to store.Source: C. Remembering From the Previous… · p. 5
  • Stored patches are selected using class-frequency weights and patch importance, then supplemented with a random sample to diversify the remembered data.The number selected by importance and at random is left to the user.Source: C. Remembering From the Previous… · p. 5
  • The remembering loss compares the updated network’s predictions with target labels on previous training data.Source: C. Remembering From the Previous… · p. 5
  • During remembering, classification layers for classes absent from the remembered dataset are frozen while the rest of the network is optimized.The user determines how often and on which data to apply the remembering loss.Source: C. Remembering From the Previous… · p. 5

A. Methods Used for Comparison

The comparison includes static and multiple learning, fixed representation, and fine-tuning. These strategies differ in whether they support continual classes, preserve earlier performance, and how much training or test-time computation they require.

A. Methods Used for Comparison

  • Static learning assumes all training images and annotations for the same classes are available at training and does not support continual addition of classes.Source: A. Methods Used for Comparison · p. 5
  • Multiple learning adds a classifier for each new training set, so classifier storage grows linearly and test-time segmentation can become expensive.All trained classifiers must be run to generate maps for all classes.Source: A. Methods Used for Comparison · pp. 5–6
  • Fixed representation freezes the feature network and trains new classification layers, preserving initial-class performance but hindering learning of new classes.The earlier classification layers are restored at test time to generate maps for all classes.Source: A. Methods Used for Comparison · p. 6, Figure 5 · p. 6
  • Fine-tuning updates the whole network using only new training data; it performs well on new classes but suffers catastrophic forgetting.Source: A. Methods Used for Comparison · p. 6

B. Datasets and Evaluation Metrics

The experiments use Luxcarta satellite imagery and the ISPRS Vaihingen and Potsdam benchmarks. Performance is assessed per class with IoU and F1-score.

B. Datasets and Evaluation Metrics

  • Luxcarta contains 8-bit RGB satellite images at 1 m resolution from 22 cities in France and Austria, with annotations for buildings, roads, high vegetation, water, and railways.Road and railway ground truth is represented by polylines; other classes use polygons rasterized into label maps.Source: B. Datasets and Evaluation Metrics · p. 6
  • Vaihingen and Potsdam provide aerial imagery at 9 cm and 5 cm resolution, respectively, with annotations for six classes; clutter is excluded from Vaihingen experiments.The datasets provide training and test tiles, and eroded ground truth is used to assess performance.Source: B. Datasets and Evaluation Metrics · p. 7
  • Per-class performance is evaluated by comparing binary predicted maps with binary ground truth using IoU and F1-score.Source: B. Datasets and Evaluation Metrics · p. 7

C. Experiments on the Luxcarta Dataset

The Luxcarta experiments test incremental learning as classes and city data arrive in stages, comparing it with static and alternative learning approaches. Incremental learning stayed closest to static learning, and retaining some prior data helped preserve old-class performance.

C. Experiments on the Luxcarta Dataset

  • The experiments streamed three training sets from 18 cities and retained 30% of prior patches, split evenly between importance-selected and random samples.On Train2, Lrem reminder updates alternated with adaptation updates; Train3 used reminders from both Train1 and Train2.Source: C. Experiments on the Luxcarta Dataset · p. 7
  • Incremental learning had the closest performance to static learning, while old-class performance could improve over time when prior data were revisited.Without prior data, old-class performance may decrease as the network adapts to new data and the memory network is imprecise on new data.Source: Table III · p. 8, C. Experiments on the Luxcarta Dataset · p. 8
  • Static learning performed best with diverse, complete training data; fixed representation retained old-class performance but struggled on new classes, while fine-tuning learned new classes but forgot old ones.Fine-tuning was particularly effective for water when only one class was added.Source: C. Experiments on the Luxcarta Dataset · p. 8
  • Close-ups show incremental learning detecting high vegetation missed without Lrem and outperforming multiple learning for buildings, while road and railway remain difficult look-alikes.Multiple learning delineated large building borders less well, and incremental learning appeared to detect some roads and railways missed without Lrem.Source: C. Experiments on the Luxcarta Dataset · p. 10, Figure 9 · p. 10
  • Road and railway were the most difficult classes, with lower numeric results than other classes, and their visual similarity sometimes led to confusion.Incremental learning appeared to detect some roads and railways misclassified without Lrem.Source: C. Experiments on the Luxcarta Dataset · p. 10

D. Experiments on the Benchmark Datasets

On the benchmarks, the memory network supported learning new classes while old-class IoU stayed relatively stable, and the new low-vegetation class converged faster than with static learning. Results also varied by class and dataset, including stronger incremental-learning performance for visually similar clutter classes.

D. Experiments on the Benchmark Datasets

  • The benchmark setup provided annotations sequentially; for Potsdam, clutter was included in the initial annotations and the memory network supplied old-class outputs while optimizing only Ladapt.The results report class F1-scores, Vaihingen IoU plots, and close-ups from both benchmarks.Source: D. Experiments on the Benchmark Datasets · pp. 10–11
  • On Potsdam, incremental learning better distinguished clutter from visually similar impervious-surface and low-vegetation pixels than multiple learning.The close-ups show examples of clutter misclassified by multiple learning but correctly detected by the proposed approach.Source: Table V · p. 11, D. Experiments on the Benchmark Datasets · pp. 11, 13, Figure 12 · p. 13
  • IoU fluctuated more on Luxcarta than Vaihingen, and the authors reported lower accuracy for the same classes on Luxcarta, citing its lower resolution and greater visual variation.Luxcarta validation cities were unseen during training, whereas Vaihingen training and test tiles came from the same city.Source: D. Experiments on the Benchmark Datasets · p. 11, Figure 11 · p. 12
  • Fixed representation performed poorly when the newly added class was a small object such as car, although some other classes were learned relatively well.The reported examples include impervious surface and low vegetation.Source: D. Experiments on the Benchmark Datasets · p. 11
  • Low vegetation reached similar accuracy in only a few epochs with the proposed approach, versus roughly 50 epochs with static learning, while previously learned classes’ IoU stayed quite flat.The authors attribute the memory network’s usefulness for old classes to training and test tiles coming from the same city.Source: D. Experiments on the Benchmark Datasets · p. 11, Figure 11 · p. 12
  • Multiple learning performed better for visually distinct classes, especially the low-sample car class on Vaihingen; Potsdam’s nDSM data made car detection easier and narrowed the gap.The authors link the clutter result to joint training via the memory network, unlike multiple learning’s isolated classifiers.Source: D. Experiments on the Benchmark Datasets · pp. 11, 13, Table IV · p. 11, Table V · p. 11

IV. CONCLUDING REMARKS

The proposed incremental-learning method adds new segmentation classes while retaining dense labeling for previously learned classes without requiring all prior training data. Experiments across three datasets report high performance on new classes without forgetting old ones, while domain adaptation for new geographic locations remains future work.

IV. CONCLUDING REMARKS

  • The method learns new classes while retaining dense labeling abilities for formerly trained classes without using all previous training data.Source: §IV · p. 13
  • Across three datasets, the proposed approach achieves high performance for new classes without forgetting old classes.The experiments also report common approaches as extremely inefficient or inapplicable to streaming data, previous-class features alone as inefficient for learning new classes, and catastrophic forgetting without regularization.Source: §IV · p. 13
  • Domain adaptation is planned to help the trained network generalize to data from new geographic locations.Source: §IV · p. 13

Related papers

Loading 1810.12448v1…