PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple multi-dataset detection

Xingyi Zhou, Vladlen Koltun, Philipp Krähenbühl

arXiv:2102.13086v2cs.CV

TL;DR

Object detection is fragmented across datasets with limited domains and vocabularies, motivating unified general-purpose systems. The paper shares a detector architecture across datasets, automatically learns a common taxonomy, and obtains strong cross-dataset performance with deployment to new domains without retraining.

  • Problem

    Object detection datasets have limited image domains and label vocabularies, leaving models locked into corresponding domains rather than providing broad recognition.

  • Method

    The method shares a backbone across dataset-specific detector outputs, uses dataset-specific objectives, and automatically learns mappings into a unified label space using visual cues.

  • Results

    Multi-dataset detectors perform well across test datasets, outperform single-dataset models, and generalize to new domains without retraining.

  • Takeaways & Limitations

    A unified detector can be deployed in new domains without additional knowledge or modification, while avoiding manual taxonomy reconciliation.

  • Takeaways & Limitations

    The label-space optimization does not take task performance into consideration when constructing the joint taxonomy.

Abstract

from arXiv · show

How do we build a general and broad object detection system? We use all labels of all concepts ever annotated. These labels span diverse datasets with potentially inconsistent taxonomies. In this paper, we present a simple method for training a unified detector on multiple large-scale datasets. We use dataset-specific training protocols and losses, but share a common detection architecture with dataset-specific outputs. We show how to automatically integrate these dataset-specific outputs into a common semantic taxonomy. In contrast to prior work, our approach does not require manual taxonomy reconciliation. Experiments show our learned taxonomy outperforms a expert-designed taxonomy in all datasets. Our multi-dataset detector performs as well as dataset-specific models on each training domain, and can generalize to new unseen dataset without fine-tuning on them. Code is available at https://github.com/xingyizhou/UniDet.

1. Introduction

Object detection is fragmented across datasets with limited domains and vocabularies. The paper trains one detector across disparate datasets and automatically unifies their label spaces, improving generalization to unseen domains.

  • Single detection datasets are limited in image domains and label vocabularies, motivating unified recognition across diverse datasets.
  • A shared detector uses separate dataset-specific outputs and supervision to train across disparate datasets.
  • The resulting detector produces duplicate outputs for classes appearing in multiple datasets.
  • Visual cues automatically define merge costs and jointly optimize a unified taxonomy, dataset mappings, and detector through 0-1 integer programming.
  • The system is evaluated on COCO, Objects365, and OpenImages at unprecedented scale.
  • A unified detector matches dataset-specific models on each training dataset and generalizes to new domains without retraining.

2. Related Work

Prior multi-dataset recognition systems combine diverse supervision but often rely on manual taxonomy reconciliation, shared training recipes, or smaller label spaces. The paper positions automatic large-scale taxonomy unification as complementary to these approaches.

  • MSeg manually unified seven semantic-segmentation taxonomies and used Mechanical Turk to resolve inconsistent annotations.
  • Universal object detectors improve robustness from diverse supervision but may use small datasets or identical training recipes across datasets.
  • A prior universal detector manually merged COCO, Pascal VOC, and SUN-RGBD taxonomies and used dataset-specific pseudo-labels.
  • The paper’s label-space learning removes manual labor while unifying COCO, Objects365, and OpenImages across 900+ classes.
  • The approach complements LVIS’s federated extension of COCO annotations and zero-shot methods for novel categories.

3. Preliminaries

Object detection predicts bounding-box locations and class-wise scores, but combining datasets is difficult because their sizes, class distributions, and effective training losses differ. A single loss therefore does not generalize across datasets.

  • Object detection predicts a location b_i ∈ R^4 and class-wise scores d_i ∈ R^|L| for each object in an image.
  • Standard COCO training minimizes a box-level loss over sampled images and corresponding annotated bounding boxes.
  • A natural multi-dataset strategy concatenates annotations and merges repeated labels into one larger dataset and label space.
  • Objects365 has 5× more images than COCO and OpenImages is 18× larger, while their class-frequency distributions are also highly imbalanced.
  • Large-scale datasets require different sampling and rare-class treatments, so training each detector may require a dataset-specific loss.
  • No single loss generalizes to all datasets, motivating a different multi-dataset training formulation.

4. Training a multi-dataset detector

The method trains a partitioned detector with shared backbone features but dataset-specific outputs and objectives, then automatically merges those outputs into a unified label space. It uses detector-output correlations and constrained optimization to preserve detection quality while avoiding manual taxonomy design.

  • Partitioned detector: A partitioned detector shares a backbone across datasets while retaining dataset-specific classification outputs and training losses.This setup mimics training separate dataset-specific detectors in parallel while using one shared network.
  • Unified label space: The unified label space maps joint labels to at most one dataset-specific label per dataset and does not impose hierarchical relations between concepts.Different label granularities remain as separate labels in the learned space.
  • Output unification: The method averages outputs for common classes to form joint detection scores, then reprojects them into dataset-specific outputs.The recovered outputs are intended to avoid degradation in the joint classifier's performance.
  • Automatic taxonomy learning: Label mappings are learned automatically from correlations in pretrained detector firings, using visual data rather than hand-designed or language-based mappings.These correlations serve as a proxy for perceptual similarity between concepts.
  • Optimization: The combinatorial search grows exponentially with the number of datasets, but low-cost merge enumeration and integer linear programming solve tested label spaces within seconds, including spaces larger than 600 labels and up to six datasets.For two datasets the objective is weighted bipartite matching; for more datasets it becomes weighted graph matching and is NP-hard.
  • Loss functions: The mapping objective can use either Average Precision on validation data or distortion between partitioned and unified detector outputs.These alternatives respectively optimize the evaluation metric or minimize the change caused by unification.

5. Experiments

The experiments show that dataset-specific sampling and losses make multi-dataset training effective, while learned label unification improves taxonomy quality and enables unified inference across domains.

  • 5.1. Multi-dataset detection: Uniform dataset sampling, class-aware sampling, and dataset-specific losses progressively improve multi-dataset detector training.The partitioned detector combines uniform inter-dataset sampling, class-aware sampling within Objects365 and OpenImages, and a hierarchy-aware loss for OpenImages.
  • 5.1. Multi-dataset detection: At an 8× schedule, the partitioned detector surpasses the single-dataset model on COCO and matches the OpenImages and Objects365 models.Longer schedules allow the partitioned detector to match or outperform dataset-specific models under comparable training conditions.
  • 5.2. Unified multi-dataset detection: The learned label space has 701 classes, agrees with the human expert on over two-thirds of labels, and captures visual distinctions that linguistic matching misses.It groups concepts such as “Cow” and “Cattle” while separating visually different “oven” concepts that share the same word.
  • 5.2. Unified multi-dataset detection: The learned label space consistently outperforms the human expert baseline by an average 0.3 mAP margin across repeated training runs.The comparison uses three runs with different random seeds and reports mean and standard deviation.
  • 5.2. Unified multi-dataset detection: Unified detectors avoid requiring the target domain at test time, whereas partitioned detectors and ensembles require the test dataset source to be known.Retraining under the unified taxonomy regains accuracy lost by direct offline unification.
  • 5.3. Cross-dataset evaluation: Across cross-dataset tests, multi-dataset detectors generally perform well, outperform a four-model single-dataset ensemble, and exceed the VOC-trained upper bound without VOC training images.The unified model outperforms the partitioned detector overall and operates on a unified taxonomy.

6. Conclusion

The paper presents a simple multi-dataset detector and an automatically constructed unified taxonomy. It also identifies visual-only label learning and absent hierarchy modeling as limitations.

  • The method trains one object detector across multiple datasets and automatically constructs a unified taxonomy.
  • The resulting detector can be deployed in new domains without additional knowledge.
  • Limitations: The label space learning algorithm currently uses only visual cues, while language cues could further improve performance.
  • Limitations: The formulation does not model label hierarchies, so COCO person and OpenImages boy remain independent classes.

A. Dataset details

The experiments use official releases and specified train/validation protocols across several detection and segmentation datasets. Dataset-specific choices define the evaluation splits and reported metrics.

  • COCO uses the standard 2017 train/validation split, while OpenImages uses Challenge-2019 and Objects365 its default version.
  • Mapillary uses the Robust Vision Challenge official release, and ScanNet uses the first 80% of scenes for training and last 20% for validation.
  • KITTI uses the RVC challenge instance-segmentation version containing 200 images.
  • WildDash is evaluated on its public version with standard mAP, excluding the official website's negative-label metric.
  • CrowdHuman evaluation uses visible bounding-box annotations.

B. Computation of label space learning algorithm and pruning

The label-space algorithm prunes candidate merges before solving an integer linear program. Thresholded greedy enumeration keeps computation manageable despite the potential exponential number of merges.

  • The optimization problem scales linearly with the number of potential merges |T|, which may grow exponentially with the number of datasets.
  • An aggressive threshold τ keeps the number of candidate merges |T′| manageable.
  • Candidate merges are greedily enumerated from two-class sets to three-class sets and larger sets.
  • The greedy algorithm runs in O(|T′| maxi |Li_hat|), while cost computation takes seconds for distortion loss and about 10 minutes for AP loss.
  • The integer programming solver finds the optimal solution within one second for both loss functions.
  • Algorithm 1 computes merge costs, constructs candidate sets, and solves the ILP to produce the unified label space L and transformations T.

C. Adding new datasets to a label space

The learned label space can be expanded after training to incorporate new datasets and labels. New classes are matched to existing classes using detector-based merge costs, while fine-grained distinctions can remain separate.

  • A label-space expansion algorithm allows adding datasets and labels after the unified detector is trained.
  • For a new dataset, the unified detector evaluates AP between each new class and each existing unified class, then chooses the lowest-cost merge.
  • A new class is merged only when its AP change is below the implementation threshold of 5 AP.
  • The optimization handles differing label granularities by keeping fine-grained Objects365 bird species separate from COCO birds when they should not merge.

E. Instance segmentation

The unified framework is evaluated for instance segmentation across eight diverse datasets, with WildDash and KITTI held out for testing. It performs consistently well on training datasets and generalizes best to the new test datasets.

  • Evaluation setup: The instance-segmentation evaluation uses eight datasets, with WildDash and KITTI reserved as testing-only domains.The setup follows the Robust vision challenge and excludes the two smaller, similar datasets from training.
  • Results: No single dataset-specific model performs well across all test domains.
  • Results: The unified model performs consistently well on training datasets and generalizes best to KITTI and WildDash.
Loading 2102.13086v2…