PaperlayerView exact revision on arXiv

Source-linked AI summary

DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model

Xiuye Gu, Yin Cui, Jonathan Huang, Abdullah Rashwan, Xuan Yang, Xingyi Zhou, Golnaz Ghiasi, Weicheng Kuo, Huizhong Chen, Liang-Chieh Chen, David A Ross

arXiv:2306.01736v1cs.CVcs.AIcs.LG

TL;DR

DaTaSeg asks whether one model can learn related segmentation tasks across datasets, where prior approaches often specialize by task or dataset. It shares mask proposals and semantic classifiers across datasets, and reports improved performance alongside weakly supervised transfer.

  • Problem

    Prior work generally uses task-specific architectures, while the question is whether diverse segmentation datasets can co-train one model across tasks.

    Source: §1 · p. 1

  • Method

    DaTaSeg shares mask proposals and network weights across datasets, uses text embeddings as class classifiers, and applies task-specific merging with weak box supervision.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    DaTaSeg improves all datasets over separate training, with ADE20k semantic gains of +6.1 and +5.1 mIoU using ResNet50 and ViTDet-B, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model supports weakly supervised transfer to ADE20k panoptic and Objects365 instance segmentation, and direct transfer to other datasets with open-vocabulary segmentation.

    Source: §1 · p. 2

    In weakly supervised ADE20k panoptic segmentation, multiple instances of the same category are sometimes not separated.

    Source: G Limitations · p. 19

Abstract

from arXiv · show

Observing the close relationship among panoptic, semantic and instance segmentation tasks, we propose to train a universal multi-dataset multi-task segmentation model: DaTaSeg.We use a shared representation (mask proposals with class predictions) for all tasks. To tackle task discrepancy, we adopt different merge operations and post-processing for different tasks. We also leverage weak-supervision, allowing our segmentation model to benefit from cheaper bounding box annotations. To share knowledge across datasets, we use text embeddings from the same semantic embedding space as classifiers and share all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg improves performance on all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised knowledge transfer on ADE panoptic and Objects365 instance segmentation. Experiments show DaTaSeg scales with the number of training datasets and enables open-vocabulary segmentation through direct transfer. In addition, we annotate an Objects365 instance segmentation set of 1,000 images and will release it as a public benchmark.

1 Introduction

DaTaSeg addresses whether one model can learn related segmentation tasks across datasets, rather than using task-specific models. It combines a shared representation and classifier strategy with multi-dataset training, reporting gains and transfer across tasks.

1 Introduction

  • DaTaSeg uses mask proposals with class labels as a shared representation, with task-specific merging and post-processing to produce different segmentation outputs.The representation also supports weak bounding-box supervision, which is cheaper to collect than mask annotations.Source: §1 · p. 2
  • The model shares network weights across datasets and uses text embeddings as class classifiers in a shared semantic space.The embeddings support knowledge sharing between categories with similar meanings across datasets.Source: §1 · p. 2
  • Switching the text-embedding classifier enables open-vocabulary segmentation.The paper contrasts this unified approach with dataset-specific model components.Source: §1 · p. 2
  • DaTaSeg improves all datasets over separate training, with ADE20k semantic gains of +6.1 and +5.1 mIoU using ResNet50 and ViTDet-B, respectively.The gains are measured against a model trained only on ADE20k semantic.Source: §1 · p. 2
  • The model enables weakly supervised transfer to ADE20k panoptic and Objects365 instance segmentation, and transfers directly to other datasets.It outperforms open-vocabulary panoptic methods on Cityscapes and performs comparably with open-vocabulary segmentation works on Pascal Context.Source: §1 · p. 2
  • The authors annotated Objects365 validation images with instance masks for release as an evaluation benchmark.The set is a subset of the validation set.Source: §1 · p. 2

2 Related Work

Prior work has unified segmentation architectures or trained across datasets, but generally has not combined multiple tasks and datasets in one model. DaTaSeg targets that combination and uses a shared mask representation with task-specific operations.

2 Related Work

  • Many multi-dataset methods unify vocabularies for a single task, while DaTaSeg targets datasets with different vocabularies and different tasks.The cited prior approaches include vocabulary merging and text-embedding alignment methods.Source: §2 · p. 2
  • The shared mask-proposal representation uses distinct merge rules for panoptic, semantic, and instance segmentation, with training losses adapted to supervision type.The figure caption also specifies Hungarian matching during training.Source: Figure 2 · p. 3
  • Prior unified segmentation models handle multiple tasks but retain separate weights for different datasets; DaTaSeg aims for one model across tasks and datasets.The passage describes mask-transformer approaches and later multi-task train-once designs.Source: §2 · p. 3
  • Existing weakly supervised segmentation methods include task-specific approaches, while DaTaSeg presents cross-dataset sharing across annotation types as a systematic alternative.The passage notes fewer attempts at weakly supervised panoptic segmentation and contrasts customized components with a data-centric approach.Source: §2 · p. 3

3 Method

DaTaSeg uses mask proposals with class predictions as a shared representation across segmentation tasks, then adapts merging and post-processing to each task. Shared network parameters, text-embedding classifiers, and co-training support learning across datasets, including with bounding-box supervision.

3.1 A universal segmentation representation

  • Mask proposals pair binary foreground masks with class predictions and can represent instances, regions, or object parts across datasets with different thing and stuff labels.Proposals can overlap and combine into higher-level outputs; the same representation accommodates categories treated differently across datasets.Source: §3.1 · p. 3

3.2 Merging predictions for specific tasks

  • MERGE combines same-class mask logits by element-wise maximum and averages the corresponding class-prediction logits.The maximum on raw logits is equivalent to merging soft masks after sigmoid; absent classes receive -100 mask logits.Source: §3.2 · p. 4
  • Panoptic segmentation merges stuff categories, semantic segmentation merges thing and stuff categories, and instance segmentation leaves proposals separate.The task-specific merge is used during training before Hungarian matching and at inference, with non-overlapping post-processing for panoptic and semantic outputs.Source: §3.2 · p. 4

3.3 Weakly-supervised instance segmentation

  • For weakly supervised instance segmentation, DaTaSeg applies a projection loss to compare horizontal and vertical projections of predicted masks with matched ground-truth boxes.Although box consistency alone is insufficient to supervise masks, the authors report that multi-dataset, multi-task transfer makes additional handcrafted losses unnecessary.Source: §3.3 · pp. 4–5

3.4 Network architecture with knowledge sharing

  • The network shares parameters across datasets and uses frozen text-encoder embeddings of category names as classifiers in a shared semantic space.The architecture generates mask and class predictions from learnable segment queries through a transformer decoder and embedding heads.Source: §3.4 · p. 5

3.5 Co-training strategy

  • At each co-training iteration, DaTaSeg samples one dataset and draws an entire batch from it, using per-dataset sampling ratios to account for dataset sizes.This strategy supports different dataset settings and distinct task losses.Source: §3.5 · p. 6

4 Experiments

Experiments evaluate DaTaSeg on ADE20k semantic, COCO panoptic, and Objects365 detection, using mask or box supervision and standard mIoU, PQ, and AP metrics. Across comparisons, the shared model improves results, supports weakly supervised transfer, and transfers to unseen segmentation datasets.

4.1 DaTaSeg improves over dataset-specific models

  • +6.1 mIoU with ResNet50 and +5.1 mIoU with ViTDet-B on ADE20k semantic; DaTaSeg also outperforms separately trained models on all datasets.The comparison matches the number of images seen per dataset and reports the main results in Table 1.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 DaTaSeg enables weakly-supervised transfer

  • +2.9 PQ and +5.4 PQ on ADE20k panoptic, and +2.0 AP and +1.4 AP on Objects365 instance segmentation, are reported with weak supervision.ADE panoptic uses semantic supervision and Objects365 instance segmentation uses bounding-box supervision; the ADE fully supervised references are 37.7 PQ and 42.3 PQ.Source: §4.2 · p. 7

4.3 DaTaSeg scales with size of backbones

  • With ViTDet-L, DaTaSeg reaches 53.5 mIoU on ADE20k semantic and 54.0 PQ on COCO panoptic, with gains consistent across the tested backbones.The design is described as independent of detailed network architecture provided the model outputs mask proposals with class predictions.Source: §4.3 · pp. 7–8

4.4 DaTaSeg scales with number of datasets

  • Performance generally improves as more training datasets are added; a COCO panoptic–Objects365 detection model reaches 18.3 mIoU on unseen ADE semantic, comparable to LSeg+ at 18.0.The dataset-combination study uses ResNet50 under the same settings.Source: Table 2 · p. 7, §4.4 · p. 8

4.5 DaTaSeg enables open-vocabulary segmentation

  • Without finetuning, DaTaSeg outperforms ODISE on Cityscapes panoptic and performs comparably on the semantic benchmarks in the system-level comparison.The comparison differs in training data and backbones; Cityscapes also has a domain gap from DaTaSeg’s training sets.Source: §4.5 · p. 8, Table 3 · p. 8

4.6 DaTaSeg scales with number of queries

  • Performance generally rises with more segment queries, except on ADE20k panoptic, while 50 queries already give reasonably good results.The reported query ablation uses a ResNet50 backbone; the authors note the low query count may benefit memory-limited applications.Source: Table 4 · p. 8, §4.6 · p. 9

4.7 Adding dataset-specific modules hurts performance

  • Dataset-specific modules hurt performance on all datasets except ADE20k semantic, with larger drops on the weakly supervised tasks.The comparison adds dataset-specific modules to DaTaSeg while keeping other settings the same.Source: Table 5 · p. 8, §4.7 · p. 9

4.8 Qualitative analysis

  • Qualitative results show generally good localization but more error-prone classification, including successes on transparent objects, occlusion, and crowd scenes.The examples use ViTDet-L and include fully and weakly supervised settings.Source: §4.8 · p. 9

5 Conclusion

DaTaSeg targets one universal model spanning multiple datasets and semantic, instance, and panoptic segmentation. It reports gains over separately trained models, transfer and open-vocabulary capabilities, and a step toward broader use of multi-task, multi-dataset training.

5 Conclusion

  • DaTaSeg surpasses separately trained models, with especially significant gains on smaller datasets such as ADE20k semantic.The conclusion also describes weakly supervised transfer to tasks not explicitly trained for, direct transfer to more datasets, and open-vocabulary segmentation.Source: §5 · p. 10
  • The model targets a single universal framework for multiple datasets and semantic, instance, and panoptic segmentation.Its approach combines a universal segmentation representation with a shared semantic embedding space for classification.Source: §5 · p. 10
  • DaTaSeg enables weakly supervised transfer to tasks not explicitly trained for, and directly transfers to additional datasets while supporting open-vocabulary segmentation.Source: §5 · p. 10
  • The authors see multi-task, multi-dataset training becoming the norm rather than the outlier, and position this work as a step in that direction.Source: §5 · p. 10

A Broader Impacts

DaTaSeg is presented as a way to meet varied segmentation needs with one model and to use data from multiple sources, especially where masks are difficult to collect. On PASCAL VOC, it outperforms the compared zero-shot methods, but the comparison uses different training sources.

A Broader Impacts

  • One segmentation model could serve memory-limited applications that otherwise use separate models for semantic, instance, and other segmentation needs.Source: A Broader Impacts · p. 13
  • Using data from multiple sources particularly benefits smaller datasets, and transfer from other sources can support weakly supervised segmentation when segmentation data are hard to collect.The authors give rare wild animals as an example: co-training on a small set of those animals and a larger dataset of common animals.Source: A Broader Impacts · p. 13
  • Because DaTaSeg learns from its training data, it may inherit that data's biases and fairness issues, though less-biased datasets may compensate for issues in others.Source: A Broader Impacts · p. 13

B Comparison with zero-shot segmentation on PASCAL VOC

  • DaTaSeg with a ResNet-50 backbone outperforms the compared zero-shot methods on all categories of PASCAL VOC 2012.DaTaSeg never saw PASCAL VOC during training, whereas the zero-shot methods trained on 15 VOC classes and held out five; the authors caution that training sources differ, so this is not an apples-to-apples comparison.Source: B Comparison with zero-shot… · p. 14, Table 6 · p. 14

C More about our newly labeled Objects365 instance segmentation validation dataset

The authors annotated 1,000 Objects365 V2 validation images with instance masks and describe the annotation process, dataset characteristics, and intended benchmark use.

C More about our newly labeled Objects365 instance segmentation validation dataset

  • The annotated set samples 1,000 Objects365 V2 validation images, using the dataset's complete bounding-box annotations to guide mask labeling.Raters paint masks on the original, uncropped image with the ground-truth box and category shown; they may skip boundaries too blurry or dark to annotate.Source: C More about our newly labeled… · p. 14
  • The work produced 12,836 valid masks from 13,372 questions, taking 800.59 rater hours and averaging 3.74 minutes per valid mask.Crowd instances were skipped in both training and validation, following common instance-segmentation practice.Source: C More about our newly labeled… · p. 15
  • Mask counts are relatively long-tailed across categories, many images have numerous instance annotations, and relative mask sizes are close to LVIS statistics.Source: C More about our newly labeled… · p. 15
  • The visualized annotations are described as high quality, including careful handling of small objects, thin structures, complicated shapes, and occlusions.The dataset stores binary masks; raters avoid occluded regions that are not part of the object being annotated.Source: C More about our newly labeled… · p. 15, Figure 6 · p. 15
  • During evaluation, objects skipped by raters are ignored when the model predicts a mask for them, rather than counted as false positives.Source: C More about our newly labeled… · p. 15
  • The Objects365 instance segmentation dataset is intended for release as a benchmark, including evaluation of weakly supervised instance segmentation learned from bounding boxes.Source: C More about our newly labeled… · p. 15

D More details on “adding dataset-specific modules hurts performance”

The supplementary analysis tests dataset-specific queries and classification components motivated by cross-dataset inconsistencies. Across the reported comparisons, removing these modules improves performance, while dataset-specific components also complicate direct transfer.

D More details on “adding dataset-specific modules hurts performance”

  • Dataset-specific queries enter the transformer decoder as auxiliary inputs, attend to image features, and interact with segment queries, but make no mask or class predictions.They are used to provide dataset-specific information at the beginning of the decoder.Source: D More details on “adding… · p. 16
  • The design keeps localization parameters shared but adds dataset-specific classification heads and background classifiers because dataset inconsistency is seen primarily as a classification issue.Background definitions vary across datasets.Source: D More details on “adding… · p. 16
  • The dataset-specific modules are lightweight, but direct transfer leaves unclear which dataset's parameters to use and hurts open-vocabulary capability.Source: D More details on “adding… · p. 17, Figure 8 · p. 17
  • Removing dataset-specific queries gives the best results on COCO panoptic and Objects365, while using 30 queries hurts COCO panoptic and ADE semantic.The caption attributes the harm from many queries to weakened knowledge sharing across datasets and tasks.Source: D More details on “adding… · p. 17, Table 7 · p. 17
  • Removing dataset-specific classification modules improves performance on all datasets and tasks, especially weakly supervised tasks.The authors suspect weakly supervised tasks rely more on knowledge sharing.Source: D More details on “adding… · p. 17, Table 8 · p. 18
  • Removing dataset-specific modules improves performance across datasets and backbones, with the largest gains on Objects365 instance and COCO panoptic; with the modules, COCO panoptic does not beat its separately trained baseline.The comparison includes ResNet50, ViTDet-B, and ViTDet-L backbones.Source: D More details on “adding… · p. 17, Table 9 · p. 18

E Architecture change from Mask2Former

DaTaSeg modifies Mask2Former to meet TPU computation constraints and speed decoder training. The changes include replacing deformable attention with an FPN, simplifying losses and evaluation sizing, and reusing final-layer matching indices.

E Architecture change from Mask2Former

  • TPUs require fixed-shape static computation graphs, so DaTaSeg replaces Mask2Former’s deformable-attention pixel decoder with a plain FPN.The cited Mask2Former comparison reports 51.9 PQ for deformable attention and 50.7 PQ for FPN on COCO panoptic.Source: E Architecture change from Mask2Former · p. 18
  • DaTaSeg omits sampled point loss and uses vanilla mask loss; the cited Mask2Former comparison reports 51.9 PQ for point loss and 50.3 PQ for mask loss.Evaluation also uses fixed input sizing: resize, then pad.Source: E Architecture change from Mask2Former · p. 18
  • For faster decoder training, each layer uses matching indices computed from the final decoder layer but retains its own prediction head.Initial-query mask predictions are not used as attention masks because they are not based on the input image.Source: E Architecture change from Mask2Former · p. 18

F Additional implementation details

The implementation preprocesses fully supervised masks into the shared format, applies dataset-specific training settings, and uses task-specific postprocessing thresholds. The reported weights table identifies matching-cost and total-loss weights for the training datasets.

F Additional implementation details

  • Panoptic and semantic groundtruth is converted to class-labeled binary masks, while box-supervised instance segmentation needs no such preprocessing.Thing instances and stuff categories are grouped separately; masks are downsampled by four to save training memory.Source: F Additional implementation details · p. 18
  • Training uses a 1:4:4 sampling ratio for ADE semantic, COCO panoptic, and Objects365 detection, with dataset-specific learning-rate multipliers of 3, 5, and 2.Inputs are randomly scaled and padded or cropped to 1024 × 1024; ADE uses a 0.5–2.0 scale range, while other datasets use 0.1–2.0.Source: F Additional implementation details · pp. 18–19
  • The training settings vary by backbone: ResNet50 trains for 500k iterations, ViTDet-B for 600k, and ViTDet-L for 540.5k.The settings also specify batch sizes, learning rates, TPU resources, and a step schedule for ResNet50.Source: F Additional implementation details · p. 19
  • Postprocessing filters panoptic proposals at confidence 0.85 for COCO and 0.8 for ADE, and removes final panoptic or instance segments smaller than four pixels.Instance predictions are capped at 100 per image; final scores multiply classification and localization scores. Table 10 lists matching-cost and total-loss weights for all training datasets.Source: F Additional implementation details · p. 19, Table 10 · p. 19

G Limitations

DaTaSeg does not introduce an efficiency-focused framework, although the approach is described as compatible with different network architectures that output its shared representation. In weakly supervised ADE20k panoptic segmentation, same-category instances are sometimes not separated.

G Limitations

  • The framework does not introduce efficiency improvements; sampled-point mask loss is one efficiency option the authors leave undeployed.They state the framework is orthogonal to network architecture when it can output the universal segmentation representation.Source: G Limitations · p. 19
  • In weakly supervised ADE20k panoptic segmentation, multiple instances of the same category are sometimes not separated.The authors suggest MixUp or Copy-Paste may help mitigate this issue and leave improvement for future work.Source: G Limitations · p. 19

H Additional qualitative results

Qualitative examples show DaTaSeg predictions can differ from groundtruth because labels or scenes are ambiguous and annotations omit objects. Direct transfers to unseen datasets show high-quality localization and classification, including challenging objects and large vocabularies.

H Additional qualitative results

  • COCO panoptic examples include ambiguous label choices and objects that DaTaSeg predicts but groundtruth omits.Examples include pavement versus dirt and window versus tree labels, along with predicted mirror-stuff, shelf, light, bottle, and additional books absent from groundtruth.Source: H Additional qualitative results · p. 19, Figure 9 · p. 20
  • Direct transfers to PC-59 and PC-459 show small-object and fine-structure segmentation, with the PC-459 results demonstrating open-vocabulary segmentation.The transfer examples are described as high quality in localization and classification, including large-vocabulary segmentation and difficult cases such as occlusions and complicated scenes.Source: H Additional qualitative results · pp. 19–20, Figure 10 · p. 21, Figure 11 · p. 22
  • DaTaSeg produces high-quality semantic segmentation on COCO despite training on COCO panoptic.The figure compares input images, predictions, and groundtruth.Source: Figure 12 · p. 23
  • Direct transfer to Cityscapes panoptic shows qualitative results on a street-view dataset described as different from all training datasets.The figure compares predictions with groundtruth.Source: Figure 13 · p. 24
Loading 2306.01736v1…