PaperlayerView exact revision on arXiv

Source-linked AI summary

MSeg: A Composite Dataset for Multi-domain Semantic Segmentation

John Lambert, Zhuang Liu, Ozan Sener, James Hays, Vladlen Koltun

arXiv:2112.13762v1cs.CV

TL;DR

Multi-domain semantic segmentation is limited by inconsistent taxonomies and annotation practices, making robust single-model performance across environments difficult. MSeg reconciles seven datasets and their pixel-level labels, then evaluates unified models through zero-shot transfer and broader scene-understanding tasks. The resulting models generalize to unseen datasets, rank first on WildDash-v1 and second in the 2020 Robust Vision Challenge, while also producing successful instance and panoptic segmentation results.

  • Problem

    Inconsistent taxonomies and annotation practices hinder training a single semantic segmentation model that works across domains.

  • Method

    MSeg constructs a composite dataset by reconciling seven datasets' taxonomies and relabeling object masks into compatible pixel-level annotations.

  • Results

    MSeg-trained models generalize to unseen datasets, rank first on WildDash-v1 and second in the 2020 Robust Vision Challenge, and support successful instance and panoptic segmentation.

  • Takeaways & Limitations

    A reconciled multi-domain dataset supports unified models with robust cross-dataset performance beyond semantic segmentation.

  • Takeaways & Limitations

    Whether MSeg relabeling improves instance and panoptic segmentation cannot be directly determined because many component datasets lack instance labels.

Abstract

from arXiv · show

We present MSeg, a composite dataset that unifies semantic segmentation datasets from different domains. A naive merge of the constituent datasets yields poor performance due to inconsistent taxonomies and annotation practices. We reconcile the taxonomies and bring the pixel-level annotations into alignment by relabeling more than 220,000 object masks in more than 80,000 images, requiring more than 1.34 years of collective annotator effort. The resulting composite dataset enables training a single semantic segmentation model that functions effectively across domains and generalizes to datasets that were not seen during training. We adopt zero-shot cross-dataset transfer as a benchmark to systematically evaluate a model's robustness and show that MSeg training yields substantially more robust models in comparison to training on individual datasets or naive mixing of datasets without the presented contributions. A model trained on MSeg ranks first on the WildDash-v1 leaderboard for robust semantic segmentation, with no exposure to WildDash data during training. We evaluate our models in the 2020 Robust Vision Challenge (RVC) as an extreme generalization experiment. MSeg training sets include only three of the seven datasets in the RVC; more importantly, the evaluation taxonomy of RVC is different and more detailed. Surprisingly, our model shows competitive performance and ranks second. To evaluate how close we are to the grand aim of robust, efficient, and complete scene understanding, we go beyond semantic segmentation by training instance segmentation and panoptic segmentation models using our dataset. Moreover, we also evaluate various engineering design decisions and metrics, including resolution and computational efficiency. Although our models are far from this grand aim, our comprehensive evaluation is crucial for progress. We share all the models and code with the community.

1 INTRODUCTION

Semantic segmentation has not yet produced a single vision system that works reliably across varied environments. MSeg addresses this by unifying datasets through taxonomy reconciliation and annotation alignment, improving cross-domain generalization.

  • Motivation: A single semantic segmentation model should operate across indoor, driving, urban, and natural environments instead of requiring separate models.Practitioners otherwise need multiple dataset-specific models and a controller to select among them.
  • Problem: Different taxonomies and annotation practices substantially reduce the accuracy and generalization of naively combined multi-domain datasets.Datasets may disagree about category definitions and pixel-level labeling conventions.
  • Approach: MSeg unifies seven datasets by reconciling their taxonomies and relabeling object masks to produce compatible annotations.The unified taxonomy contains 194 categories rather than the more than 300 classes produced by a naive merge.
  • Results: MSeg training yields models that generalize better to datasets unseen during training under zero-shot cross-dataset transfer.The benchmark tests models on datasets disjoint from MSeg training data.
  • Broader evaluation: The study also evaluates instance segmentation, panoptic segmentation, resolution, and computational efficiency to assess broader scene-understanding capability.These evaluations examine tasks beyond semantic segmentation and the trade-off between accuracy and efficiency.

2 RELATED WORK

Prior work mixes datasets or studies domain generalization, but unified semantic segmentation remains hindered by incompatible taxonomies and annotations. MSeg directly addresses this incompatibility and extends evaluation beyond semantic segmentation.

  • Cross-domain semantic segmentation: Existing dataset-mixing efforts largely focus on single-domain driving applications or combine datasets without resolving incompatible annotations.MSeg instead targets multiple domains and performs deeper label unification.
  • Cross-domain semantic segmentation: Cross-dataset transfer is significantly inferior to self-training in prior driving evaluations, a pattern also observed for individual or naively mixed datasets.This motivates evaluating robust transfer after taxonomy and annotation reconciliation.
  • Domain adaptation and generalization: MSeg derives a consistent taxonomy that bridges datasets from multiple domains, addressing a central obstacle to unified semantic segmentation.Compatible category definitions also support applying one classifier across environments.
  • Domain adaptation and generalization: Domain generalization methods train on multiple domains without target-domain data, whereas adaptation uses target-distribution samples during training.The paper operates in the generalization setting and evaluates previously unseen environments.
  • Domain adaptation and generalization: MSeg is complementary to domain-generalization research because its compatible taxonomy and annotations facilitate shared classifiers across environments.The paper frames dataset unification as infrastructure for future domain-generalization work.
  • Visual learning over diverse domains: Unlike multi-task benchmarks that allow training on all datasets, MSeg evaluates a single classifier under zero-shot cross-dataset generalization.Its focus is semantic segmentation with reconciled semantic labels rather than transferring inductive biases across tasks.
  • Beyond semantic segmentation: Because most component training datasets contain instance labels, MSeg also supports instance and panoptic segmentation experiments.The paper reports successful results for both extensions.

3 THE MSEG DATASET

MSeg constructs a unified multi-domain segmentation dataset by reconciling incompatible taxonomies and annotations across diverse source datasets. Its 194-class taxonomy preserves compatible distinctions while using systematic merging, splitting, and mask relabeling to support cross-domain training.

  • Dataset composition: MSeg combines multiple diverse semantic segmentation datasets after a label-unification process, using selected datasets for training and zero-shot cross-dataset validation.Data from test datasets, including their training splits, is never used for MSeg training.
  • Taxonomy construction: Naive dataset mixing yields low accuracy and poor generalization because source taxonomies and annotation practices are inconsistent.These inconsistencies make a unified cross-domain semantic segmentation model difficult to train without reconciliation.
  • Taxonomy construction: Taxonomy decisions aim to preserve as many classes as possible while maintaining a flat structure compatible with standard training methods.The authors use decision rules involving manual inspection, equivalence, unique categories, subset or superset relationships, and unlabeled cases.
  • Taxonomy construction: Superset categories may be shattered into enforceable pre-existing subsets; otherwise related classes are merged when new boundaries cannot be drawn or classification is not reliable.An “-other” category can be added when shattered masks fit none of the existing subsets.
  • Taxonomy construction: 316 classes from naively merged datasets were condensed into a unified taxonomy of 194 classes.The unified classes are derived from classes in the component datasets, with class statistics reported in Figure 3.
  • Relabeling instances: 31 classes required splitting, so Amazon Mechanical Turk annotators relabeled 221,323 masks into finer-grained MSeg categories.The workflow treats relabeling as multiway classification rather than recomputing object boundaries.

4 EXPERIMENTAL RESULTS

MSeg is evaluated as a unified multi-domain segmentation resource using zero-shot transfer, in-domain validation, qualitative comparisons, and controlled ablations. Across these evaluations, MSeg models generalize consistently, achieve strong benchmark performance, and benefit from taxonomy reconciliation and relabeling.

  • Zero-shot quantitative results: Zero-shot transfer evaluates MSeg models on test datasets unseen during training, using comparisons with individual-dataset, naive-mixing, and oracle models.The semantic segmentation model is trained using 1 million crops; WildDash has no training set, so no oracle performance is reported there.
  • Zero-shot quantitative results: 28% higher harmonic-mean mIoU is achieved by MSeg than by the best individually trained baseline, COCO, across test datasets.No individual training dataset performs well across all test datasets, whereas MSeg performs consistently across them.
  • Zero-shot quantitative results: 13.8 points is the ScanNet gap between an oracle model and MSeg, while MSeg remains competitive with oracle models on many other datasets.ScanNet is especially difficult because its scenes are captured while the photographer is in motion, allowing the oracle to learn motion-blur effects.
  • WildDash benchmark: 1st place is achieved on the WildDash-v1 leaderboard by an MSeg model trained without WildDash images, improving 9.3 percentage points over the best comparable prior model.The improvement is reported as a 24% relative gain, and the model outperforms methods that used WildDash validation data during training.
  • Zero-shot qualitative results: MSeg succeeds across ScanNet, KITTI, and WildDash qualitative examples where baselines fail on indoor objects, difficult illumination, trucks, and snow-covered roads.Relabeled COCO counter masks support recognition of ScanNet counters; MSeg also correctly identifies WildDash trucks and snowy road surfaces in the cited examples.
  • Performance on training datasets: 76% higher aggregate harmonic-mean accuracy is achieved by MSeg than by the best individually trained baseline on the MSeg training datasets.The MSeg model matches or outperforms individually trained models across almost all datasets, though most validation results are not directly literature-comparable under the merged taxonomy.
  • Ablation study for relabeling: Unified taxonomy improves performance over a naive class-union merge, and compatible relabeling provides an additional improvement.The controlled evaluation compares the naive merge, MSeg without relabeling, and the full MSeg dataset.

4.6 Performance in the 2020 Robust Vision Challenge

The paper evaluates MSeg models in the RVC as an extreme generalization setting, alongside resolution, efficiency, instance, and panoptic segmentation analyses. MSeg performs competitively despite taxonomy and dataset differences, while resolution affects the trade-off between in-domain accuracy and zero-shot generalization.

  • RVC evaluation: MSeg ranked second in the 2020 Robust Vision Challenge despite using only three of RVC’s seven training datasets and a different evaluation taxonomy.The RVC required a single unified model, and MSeg was evaluated without additional training.
  • RVC evaluation: A matching taxonomy raises MSeg’s Mapillary mIoU to 5% above SwiftNet’s.Nineteen fine-grained Mapillary classes were merged into MSeg’s universal taxonomy, affecting backward label mapping.
  • Resolution: The 480p model shows superior zero-shot generalization, whereas higher resolution helps performance on training datasets.The paper presents distributional distance as a conjectured explanation for this discrepancy.
  • Computational efficiency: Realtime single-scale inference is possible on an NVIDIA RTX 2080 Ti desktop GPU.Runtime is averaged over 100 single-scale forward passes using optimized inference code.
  • Beyond semantic segmentation: 720p instance segmentation is clearly superior to 480p, while 1080p does not improve upon 720p.The evaluation reports box average precision and mask average precision across resolutions.
  • Beyond semantic segmentation: Panoptic quality is similar across resolutions quantitatively, but high-resolution models perform best qualitatively.Panoptic segmentation jointly evaluates instance and semantic segmentation capabilities.

5 CONCLUSION

The conclusion presents MSeg as a composite dataset for multi-domain scene understanding, built through taxonomy reconciliation and large-scale mask relabeling. The resulting models generalize across domains and support practical evaluation beyond semantic segmentation.

  • MSeg reconciles the taxonomies of seven semantic segmentation datasets to construct a composite dataset for multi-domain scene understanding.
  • Large-scale Mechanical Turk mask relabeling handles cases where categories need to be split.
  • MSeg models generalize to unseen datasets, rank first on WildDash, rank second in the RVC, and include resolution, instance, and panoptic evaluations.

15 APPENDIX

The appendix provides additional details and experimental results.

  • The appendix contains additional methodological details.
  • The appendix contains additional experimental results.
  • The appendix supplements the paper’s main presentation with further information.

6 RELABELING INTERFACE AND WORKFLOW

The relabeling workflow uses Mechanical Turk to split masks that require finer category distinctions. Annotators receive binary mask-classification tasks through a batched interface.

  • Tasks: The workflow includes cabinet-merged, curtain, and animal mask-splitting tasks.
  • Batching: Human intelligence tasks are distributed in batches of 100 masks to each worker.
  • Interface: Mechanical Turk annotators perform binary mask classification tasks.

7 ADDITIONAL DETAILS ON MSEG

MSeg’s additional details document the taxonomy mapping and the relabeling counts for COCO Panoptic classes.

  • Figure 2’s partial taxonomy mapping is supplemented by Figures 9 and 10 for the remaining classes.The mapping covers meta-training dataset taxonomies and the MSeg taxonomy.
  • Table 13 records how many COCO Panoptic masks were relabeled for classes a–f.

8 ADDITIONAL QUALITATIVE COMPARISON WITH SWIFTNET

This section presents an additional qualitative comparison of the method with SwiftNet on ADE20K, shown in Figure 11.

  • The authors qualitatively compare their method with SwiftNet on ADE20K.The comparison is presented in Figure 11.
  • Figure 11 provides an additional qualitative comparison on ADE20K.
  • The comparison supplements the main paper’s qualitative study on WildDash-v2.

9 PANOPTIC CLASS MAPPING

MSeg resolves inconsistent stuff–thing designations across source datasets by applying a universal classification rule.

  • Different training datasets use their own stuff and thing classes, and the boundary can be ambiguous.
  • A universal class is treated as a thing only when every mapped source class is a thing.
  • If any source dataset labels a mapped class as stuff, MSeg treats the universal class as stuff across datasets.The rule prevents a class from being treated as a thing when one source designation conflicts.
  • Table 21 summarizes the stuff-versus-thing assignments in MSeg and notes that BDD and SUN RGBD lack instance labels.

10 ADDITIONAL RE-LABELING FOR DENOISING

MSeg includes supplementary evidence on taxonomy mapping and additional relabeling intended to address annotation errors, including errors in IDD.

  • Qualitative inspection found many erroneous annotations for some IDD classes, prompting additional relabeling of rider, bicycle, and motorcycle masks.
  • Table 21 displays source-dataset stuff and thing assignments, with thing classes in red and stuff classes in blue.
  • Figures 9 and 10 visualize additional mappings from source datasets to the unified MSeg taxonomy.Figure 9 indicates that split and merge operations were used for some mappings.
  • Figure 11 compares SwiftNet and MSeg qualitatively on ADE20K RVC test images and discusses taxonomy-driven confusion among building classes.The taxonomy merges skyscrapers, buildings, and houses because workers could not consistently separate them.
  • Figure 12 shows an IDD example originally labeled rider that was relabeled as bicycle.
Loading 2112.13762v1…