PaperlayerView exact revision on arXiv

Source-linked AI summary

OW-DETR: Open-world Detection Transformer

Akshita Gupta, Sanath Narayan, K J Joseph, Salman Khan, Fahad Shahbaz Khan, Mubarak Shah

arXiv:2112.01513v3cs.CV

TL;DR

OWOD requires detecting known and unknown objects while incrementally learning newly identified classes, but unknown proposals, unknown–background separation, and diverse contextual modeling remain difficult. OW-DETR addresses these issues with a transformer-based detector combining attention-driven pseudo-labeling, novelty classification, and objectness scoring. It outperforms ORE on MS-COCO unknown recall and achieves state-of-the-art incremental detection performance on PASCAL VOC.

  • Problem

    OWOD must detect known and unknown objects and incrementally learn new classes despite challenges in proposing unknowns, separating them from background, and modeling diverse context.

  • Method

    OW-DETR uses a transformer-based architecture with multiscale context encoding, attention-driven pseudo-labeling, novelty classification, and objectness scoring.

  • Results

    OW-DETR consistently outperforms ORE across MS-COCO OWOD settings and achieves state-of-the-art incremental object detection performance on PASCAL VOC.

  • Takeaways & Limitations

    The dedicated components support detecting unknown objects and transferring foreground characteristics from known classes to unknown classes within the evaluated OWOD settings.

  • Takeaways & Limitations

    The approach is motivated by the need to model diverse unknown objects and relies on assumptions that fewer inductive biases may benefit their detection.

Abstract

from arXiv · show

Open-world object detection (OWOD) is a challenging computer vision problem, where the task is to detect a known set of object categories while simultaneously identifying unknown objects. Additionally, the model must incrementally learn new classes that become known in the next training episodes. Distinct from standard object detection, the OWOD setting poses significant challenges for generating quality candidate proposals on potentially unknown objects, separating the unknown objects from the background and detecting diverse unknown objects. Here, we introduce a novel end-to-end transformer-based framework, OW-DETR, for open-world object detection. The proposed OW-DETR comprises three dedicated components namely, attention-driven pseudo-labeling, novelty classification and objectness scoring to explicitly address the aforementioned OWOD challenges. Our OW-DETR explicitly encodes multi-scale contextual information, possesses less inductive bias, enables knowledge transfer from known classes to the unknown class and can better discriminate between unknown objects and background. Comprehensive experiments are performed on two benchmarks: MS-COCO and PASCAL VOC. The extensive ablations reveal the merits of our proposed contributions. Further, our model outperforms the recently introduced OWOD approach, ORE, with absolute gains ranging from 1.8% to 3.3% in terms of unknown recall on MS-COCO. In the case of incremental object detection, OW-DETR outperforms the state-of-the-art for all settings on PASCAL VOC. Our code is available at https://github.com/akshitac8/OW-DETR.

1. Introduction

OWOD requires detectors to recognize known and unknown objects while incrementally learning newly labeled classes, creating challenges in proposal generation, unknown–background separation, and contextual modeling. OW-DETR addresses these challenges with a transformer-based architecture and dedicated pseudo-labeling, novelty-classification, and objectness components.

  • Open-world setting: OWOD combines known-object detection, unknown-object identification, and incremental learning of newly labeled classes across training episodes.Unknown detections can be labeled by an oracle and incorporated without retraining from scratch on previously known classes.
  • Challenges: Detectors must generate candidate boxes for unknown objects, separate unknowns from background, and model diverse object sizes and contextual relations.These requirements differ from conventional closed-world detection, where only seen classes appear at inference.
  • Prior approach: ORE uses class-agnostic RPN proposals with high objectness and no overlap with known ground truth to auto-label pseudo-unknowns during training.The approach was an early OWOD formulation but depends on a two-stage Faster R-CNN pipeline.
  • Prior approach: ORE is limited by weakly supervised validation for unknown-category energy estimation, a single prototype for diverse unknowns, and convolutional modeling without explicit long-range dependencies.These issues can hinder known–unknown separation and contextual representation in images containing diverse objects.
  • Proposed approach: OW-DETR introduces a transformer-based detector with multiscale self-attention, deformable receptive fields, attention-driven pseudo-labeling, novelty classification, and objectness scoring.The design uses fewer open-world assumptions and reduced inductive bias while separating pseudo-unknowns from knowns and foreground from background.
  • Results: OW-DETR is evaluated on MS-COCO and PASCAL VOC, outperforming ORE by 1.8% to 3.3% in unknown recall on MS-COCO.The authors also report superiority over the state of the art for incremental object detection on PASCAL VOC.

2. Open-world Detection Transformer

OW-DETR adapts Deformable DETR for open-world detection by combining attention-driven pseudo-labeling, novelty classification, and objectness scoring. It uses these components to identify candidate unknowns, distinguish them from known classes and background, and support incremental learning.

  • Open-world Detection Transformer: OWOD requires detecting known objects, identifying unknown objects, and incrementally learning newly labeled classes without retraining from scratch.Unknown detections can be forwarded to an oracle for labeling before the model updates its known-class set.
  • Novelty Classification: OW-DETR uses novelty classification to classify queries into known or unknown classes and objectness scoring to separate foreground objects from background.The foreground set includes ground-truth known objects and pseudo-unknowns, while dedicated losses train novelty classification, objectness, and box regression jointly.
  • Attention-driven Pseudo-labeling: Attention-driven pseudo-labeling selects high-attention unmatched query boxes as candidate unknown objects for training.Objectness scores are computed from activation magnitude within each query proposal’s region of interest, and the top-ku queries among M−K unmatched queries are pseudo-labeled unknown.
  • Multi-scale Context Encoding: OW-DETR adapts single-stage Deformable DETR to encode long-term multi-scale context with fewer inductive biases for unknown-object detection.Deformable attention samples a small fixed number of points across multi-scale feature maps, providing richer context over a larger receptive field.
  • Training and Inference: The model is trained end-to-end with the joint loss L = L_n + L_r + αL_o, combining novelty classification, box regression, and objectness scoring.Focal loss is used for novelty classification and objectness, while box regression uses standard ℓ1 loss; α weights objectness scoring.

3. Experiments

Experiments evaluate OW-DETR on MS-COCO for OWOD and on PASCAL VOC for incremental object detection. OW-DETR improves unknown-object retrieval and known-class performance over relevant baselines, while ablations and qualitative examples support the roles of its components.

  • MS-COCO OWOD: MS-COCO is divided into four nonoverlapping tasks, with previously encountered classes known and later classes unknown during each task.In Task 4, all 80 classes are known, so U-Recall is not computed.
  • PASCAL VOC iOD: OW-DETR achieves favorable performance against existing approaches across all three PASCAL VOC incremental object detection settings.The settings introduce 10, 5 or 1 classes to detectors trained on the remaining 10, 15 or 19 classes, respectively.
  • MS-COCO OWOD: 7.5, 6.2 and 5.7 U-Recall are achieved by OW-DETR on MS-COCO Tasks 1, 2 and 3, respectively, versus 4.9, 2.9 and 3.9 for ORE−EBUI.OW-DETR also achieves absolute known-class mAP gains up to 3.6% over ORE across four tasks.
  • Qualitative Analysis: Attention maps activate more strongly around foreground regions, and qualitative detections recover several unknown objects but miss some challenging instances.Examples include detected racket, umbrella and fire hydrant, with sink and table missed in other images.
  • Ablation Study: Adding novelty classification with attention-selected pseudo-unknowns enables unknown detection, while adding objectness further raises U-Recall to 7.5, 6.2 and 5.7 on Tasks 1, 2 and 3.The baseline cannot detect unknown objects; the final framework uses novelty classification and objectness scoring to improve unknown retrieval.

4. Relation to Prior Art

OW-DETR addresses OWOD by combining a transformer framework with attention-driven pseudo-labeling, novelty classification and objectness scoring. This distinguishes it from standard, incremental, open-set, and earlier open-world detection approaches.

  • Relation to Prior Art: Standard object detection assumes the label space encountered throughout a model’s life matches the training label space.Incremental detection instead introduces newer classes during training and addresses catastrophic forgetting.
  • Relation to Prior Art: Open-set detection rejects unknown test objects, whereas OWOD detects known and unknown objects while incrementally learning newly known classes.OW-DETR is positioned within the latter problem formulation.
  • Relation to Prior Art: OW-DETR uses transformer-based multi-scale contextual modeling with fewer inductive biases and transfers objectness knowledge from known to novel classes.Its dedicated components are attention-driven pseudo-labeling, novelty classification and objectness scoring.

5. Conclusions

OW-DETR is a transformer-based approach for open-world object detection, evaluated on MS-COCO and PASCAL VOC. It consistently outperforms ORE on MS-COCO OWOD tasks and achieves state-of-the-art incremental detection performance on PASCAL VOC.

  • Conclusions: OW-DETR combines attention-driven pseudo-labeling, novelty classification and objectness scoring to detect unknown objects in open-world settings.These components are designed specifically for open-world detection.
  • Conclusions: OW-DETR consistently outperforms ORE across all MS-COCO OWOD task settings.The evaluation covers MS-COCO and PASCAL VOC benchmarks.
  • Conclusions: OW-DETR achieves state-of-the-art performance for incremental object detection on PASCAL VOC.This conclusion is reported for the evaluated PASCAL VOC setting.

A.1. Evaluation using WI and A-OSE Metrics

The WI and A-OSE analysis evaluates confusion between known and unknown objects, alongside U-Recall for unknown-instance retrieval. OW-DETR improves all three metrics across tasks over Faster R-CNN-based ORE.

  • Metrics: Wilderness impact measures known-class precision degradation when unknown instances are included, while A-OSE counts unknown instances classified as known classes.Both metrics quantify confusion caused by unknown objects.
  • Metrics: U-Recall measures the model’s ability to retrieve unknown object instances.It directly relates to the unknown class and complements WI and A-OSE.
  • Results: OW-DETR improves WI, A-OSE and U-Recall across tasks over the Faster R-CNN-based ORE comparison.The comparison excludes ORE’s EBUI because it uses held-out validation data with weak unknown supervision.

A.2. Proposed MS-COCO Split for Open-world

The proposed MS-COCO split addresses data leakage in the original OWOD task design by grouping classes across super-categories within tasks, creating a harder evaluation setting.

  • Motivation: The original ORE split can leak information across tasks by distributing related classes from the same super-category across different tasks.Examples include vehicle and animal classes introduced across Tasks 1 and 2.
  • Proposed split: The proposed split introduces classes across super-categories and places all classes from each super-category in the same task.This mitigates cross-task leakage by avoiding the spread of a super-category across tasks.
  • Evaluation setting: The stricter split is more challenging for open-world object detection than the original ORE split.The split contains nearly 20 classes per task.

A.3. Fully- vs. Self-supervised Pretraining

The framework uses self-supervised DINO pretraining for its ResNet-50 backbone to avoid potential violations of the open-world setting associated with labeled pretraining.

  • Self-supervised pretraining: OW-DETR uses a ResNet-50 backbone pretrained on ImageNet1K with self-supervised DINO without labels.This choice avoids using class annotations during pretraining.
  • Fully supervised pretraining: Fully supervised pretraining may overlap with novel classes and violate the open-world evaluation assumptions.The paper additionally evaluates a fully supervised pretrained backbone for comparison.

B. Additional Qualitative Results

Qualitative and tabulated evidence shows OW-DETR detecting unknown objects more effectively than ORE and converting appropriate unknown predictions into known-class detections after incremental learning.

  • OWOD comparison: OW-DETR correctly identifies unknown refrigerators and road signs that ORE misses or misclassifies in MS-COCO examples.The qualitative comparison attributes these improvements to attention-driven pseudo-labeling, novelty classification, and the objectness branch.
  • OWOD comparison: The MS-COCO comparisons evaluate wilderness impact, absolute open set error, and unknown class recall.Unknown recall measures the ability to retrieve unknown object instances.
  • OWOD comparison: OW-DETR improves unknown recall over ORE across tasks on the stricter MS-COCO split.The proposed split mitigates data leakage and is more challenging than the original ORE split.
  • Evolution of predictions: After incremental training, parking meters, giraffes, zebras, and traffic lights transition from unknown predictions to known-class detections.Some Task 1 confusions arise from visual similarity between unknown objects and known classes such as horse.
  • Overall findings: The combined quantitative and qualitative results indicate more accurate unknown-object detection in the open-world setting.The reported evaluation uses the proposed OW-DETR contributions across the examined comparisons.

C. Societal Impact and Limitations

The paper highlights privacy and legal cases requiring objects or identities to be withheld or forgotten, while acknowledging that performance remains limited by the difficulty of OWOD.

  • Societal impact: Privacy or legal concerns may require an incremental model not to detect particular objects or fine-grained categories.The paper frames these requirements as relevant to open-world learning but outside its scope.
  • Limitations: Solutions for selectively unlearning or forgetting object types, attributes, or identities are beyond the scope of this work.The paper identifies these problems as highly relevant and significant.
  • Limitations: Despite improvements over ORE in Recall and mAP, the reported performances remain on the lower side.The paper attributes this limitation to the challenging nature of open-world object detection.

D. Additional Implementation Details

OW-DETR uses multi-scale 256-channel feature maps as input to a deformable transformer encoder and follows a specified multi-stage training setup.

  • Backbone feature maps are projected to 256 channels and provided as multi-scale input to the deformable transformer encoder.Training uses PyTorch on eight NVIDIA Tesla V100 GPUs.
  • Each task trains OW-DETR for 50 epochs, followed by 20 epochs of finetuning during incremental learning.
Loading 2112.01513v3…