PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Open World Object Detection

K J Joseph, Salman Khan, Fahad Shahbaz Khan, Vineeth N Balasubramanian

arXiv:2103.02603v2cs.CVcs.AIcs.LG

TL;DR

The paper defines Open World Object Detection for detectors that must recognize unseen objects as unknown and later learn their categories without forgetting known classes. It formalizes this setting and proposes ORE, combining contrastive clustering with energy-based unknown identification; experiments analyze its open-world behavior and incremental detection performance.

  • Problem

    Open World Object Detection addresses detection when test images may contain classes unavailable during training, requiring unknown identification and later incremental learning.

  • Method

    ORE combines contrastive clustering and energy-based unknown identification to detect and learn unknown object categories in the proposed setting.

  • Results

    ORE achieves lower WI and A-OSE than Faster RCNN, while balanced finetuning restores known-class mAP from 4.076% to 51.09% after Task 2.

  • Takeaways & Limitations

    Identifying and characterizing unknown instances reduces confusion in incremental object detection, while component studies find ALU, EBUI, and contrastive clustering jointly strongest for unknown handling.

  • Takeaways & Limitations

    The method assumes a bound on the maximum number of classes to expect and uses exemplar memory whose size trades off known-class accuracy against unknown handling.

Abstract

from arXiv · show

Humans have a natural instinct to identify unknown object instances in their environments. The intrinsic curiosity about these unknown instances aids in learning about them, when the corresponding knowledge is eventually available. This motivates us to propose a novel computer vision problem called: `Open World Object Detection', where a model is tasked to: 1) identify objects that have not been introduced to it as `unknown', without explicit supervision to do so, and 2) incrementally learn these identified unknown categories without forgetting previously learned classes, when the corresponding labels are progressively received. We formulate the problem, introduce a strong evaluation protocol and provide a novel solution, which we call ORE: Open World Object Detector, based on contrastive clustering and energy based unknown identification. Our experimental evaluation and ablation studies analyze the efficacy of ORE in achieving Open World objectives. As an interesting by-product, we find that identifying and characterizing unknown instances helps to reduce confusion in an incremental object detection setting, where we achieve state-of-the-art performance, with no extra methodological effort. We hope that our work will attract further research into this newly identified, yet crucial research direction.

1. Introduction

Open World Object Detection addresses the gap between closed-world training and deployment environments containing unseen objects. It requires detectors to identify unknown instances and incrementally learn newly labeled classes while preserving known-class recognition.

  • Motivation: Existing object detectors assume every class encountered at inference is available during training.Relaxing this assumption introduces unknown-object identification and incremental learning of newly labeled classes.
  • Contribution: Open World Object Detection is presented as a novel problem distinct from related Open Set and Open World classification settings.The paper argues that its distinct challenges matter for practical object detectors.
  • Problem setting: Open World Object Detection unifies identifying unknown objects and subsequently learning their categories as labels progressively arrive.The setting is motivated by the ability to identify what is not yet known and then learn it when information becomes available.
  • Challenge: Unknown instances are learned as background during detector training, yet existing detectors can classify them as known classes with high confidence.This makes open-set classification methods nontrivial to adapt directly to object detection.
  • Practical motivation: The proposed setting models dynamic real-world deployments in which complete knowledge of future object classes is unavailable.The intended behavior is to label unseen objects as unknown, recognize known objects, and incorporate new labels into the existing knowledge base.

2. Related Work

Related work progresses from Open Set classification to Open World classification and Open Set detection. The paper emphasizes that object detection remains difficult because unknown objects are often absorbed into background during training and later misclassified as known classes.

  • Open Set Classification: Open Set classification treats the training knowledge as incomplete and addresses test-time encounters with unknown classes.Prior methods include one-vs-rest classifiers and probabilistic models that reduce confidence for samples distant from known training examples.
  • Open World Classification: Open World classification extends static recognition by allowing known and unknown classes to coexist and by adapting when labels for unknowns arrive.The cited approach recalibrates class probabilities within a nearest-class-mean framework.
  • Open Set Detection: Open Set detection studies show that object detectors can classify unknown classes as known ones with high confidence despite background training signals.This literature also develops measures for evaluating open-set detection behavior.

3. Open World Object Detection

Open World Object Detection formalizes detection with a time-varying known-class set and an open-ended unknown-class set. A model must detect known classes, label unseen instances as unknown, and forward unknown instances for human selection of new classes.

  • Formal definition: At time t, the known classes are K_t = {1, 2, ..., C}, while unknown classes form an open-ended set U = {C + 1, ...}.Training data D_t contains images and labeled object instances whose class labels belong to K_t.
  • Data representation: Each labeled object instance records a class label and bounding-box center coordinates, width, and height.The formal label representation is y_k = [l_k, x_k, y_k, w_k, h_k], with l_k ∈ K_t.
  • Detection and learning: The model M_C detects previously encountered classes and assigns label 0 to test instances belonging to unseen classes.Unknown instances can then be forwarded to a human user who identifies new classes of interest.

4. ORE: Open World Object Detector

ORE addresses open-world detection by identifying unknown instances without explicit supervision and incrementally learning their labeled categories without forgetting earlier classes. It combines contrastive clustering, RPN-based auto-labeling, and energy-based identification within a Faster R-CNN detector.

  • ORE jointly identifies unknown instances without explicit supervision and incrementally learns their labeled categories without retraining from scratch or forgetting earlier instances.
  • ORE Architecture: Faster R-CNN supplies the base detector, while its RPN and classification head are adapted for unknown auto-labeling and identification.Features from the RoI head are contrastively clustered as part of the unified architecture.
  • Contrastive Clustering: Contrastive clustering separates same-class features from dissimilar-class features in the detector’s latent space.The loss keeps similar instances close and pushes dissimilar instances apart.
  • Contrastive Clustering: ORE maintains class prototypes using class-specific feature queues and periodically updates them with momentum-weighted new prototypes.Prototype updates allow cluster centers to evolve gradually as the network’s features change.
  • Auto-labelling Unknowns with RPN: The class-agnostic RPN pseudo-labels high-objectness proposals that do not overlap ground-truth objects as potential unknowns.This avoids manually re-annotating unknown instances in already labeled datasets.
  • Energy Based Unknown Identifier: An energy-based classification head transforms Faster R-CNN logits into free energy and models known and unknown energy distributions with shifted Weibull distributions.The method uses the separation between known and unknown energy values to identify unseen samples.

5. Experiments and Results

The paper evaluates Open World Object Detection through a task-based protocol and compares ORE with standard and incremental detectors. ORE models unknown instances explicitly while retaining known-class detection and supporting progressive class learning.

  • The evaluation protocol groups classes into sequential tasks, treating earlier classes as known and future classes as unknown while new labels arrive progressively.
  • Wilderness Impact and A-OSE measure confusion of unknown instances with known classes, while mAP measures detection of known classes.WI is evaluated at recall R = 0.8, and incremental learning uses mAP at IoU threshold 0.5.
  • ORE extends Faster R-CNN with contrastive clustering and energy-based unknown identification, alongside mechanisms for handling incrementally changing class sets.
  • ORE achieves significantly lower WI and A-OSE than Faster R-CNN while preserving known-class performance and handling progressively labeled unknown classes.When Task 2 classes are added, baseline mAP falls from 56.16% to 4.076%, balanced finetuning restores 51.09%, and ORE addresses both known detection and unknown handling.
  • ORE also performs favorably against state-of-the-art incremental object detectors without changing its methodology.

6. Discussions and Analysis

The analyses examine ORE’s components, exemplar-memory trade-offs, convergence, and qualitative behavior. Results support the joint contribution of unknown identification and contrastive clustering, while revealing limits from memory size and misclassification.

  • 6.1 Ablating ORE Components: Combining auto-labeling of unknowns with energy-based identification performs better than either component alone, and adding contrastive clustering gives the best WI and A-OSE performance.
  • 6.1 Ablating ORE Components: Removing energy-based unknown identification increases WI and A-OSE, while unknown identification does not cause a severe known-class mAP drop.
  • 6.2 Sensitivity Analysis on Exemplar Memory Size: Increasing exemplar memory beyond a moderate size worsens unknown handling, whereas at least 10 instances per class substantially improve previously known-class accuracy.The paper selects N_ex = 50 as a balance between known-class performance and unknown handling.
  • 6.3 Comparison with an Open Set Detector: ORE’s qualitative predictions classify most unseen objects as unknown but misclassify one giraffe as a horse, illustrating a failure case.
  • 6.3 Comparison with an Open Set Detector: Unknown instances also form clusters, and the contrastive clustering loss gradually decreases during training, indicating convergence.

7. Conclusion

The paper introduces Open World Object Detection to address unknown objects and progressive learning in dynamic environments. ORE combines energy-based unknown detection with contrastive clustering as a foundation for this setting.

  • Open World Object Detection requires detectors to label unseen objects as unknown and progressively learn them when new labels become available.
  • ORE’s key novelties are an energy-based classifier for unknown detection and a contrastive clustering approach for open world learning.
  • The paper presents Open World Object Detection as an open research direction beyond closed-set benchmarks.

Supplementary Material

The supplementary material provides additional analyses, implementation details, discussion, and qualitative evidence for the proposed Open World Object Detection approach.

  • The supplement adds experimental analysis, implementation details, discussion, and results that develop further insights into ORE.
  • Supplementary topics include feature-store queue size, momentum, clustering margin, energy temperature, contrastive clustering, failure cases, related work, and qualitative results.

A. Varying the Queue Size of FStore

Varying the FStore queue size produces relatively similar performance because class prototypes are periodically updated with newly observed features. The experiments use Q = 20 throughout.

  • A. Varying the Queue Size of FStore: Different FStore queue sizes yield relatively similar performance when learning Task 1.After a prototype is defined, periodic updates with newly observed features make the number of stored features less significant.
  • A. Varying the Queue Size of FStore: ORE uses Q = 20 for all experiments.
  • A. Varying the Queue Size of FStore: Varying the number of features used to compute class prototypes does not have a huge impact on performance.

B. Sensitivity Analysis on η

The ablations examine prototype-update momentum, contrastive-clustering margin, and energy temperature. Gradual prototype evolution and greater latent-space separation improve performance, while temperature has an optimal range between 1 and 2.

  • B. Sensitivity Analysis on η: Higher η improves performance, indicating that gradually evolving class prototypes benefits contrastive clustering.Larger η means newly computed prototypes have a smaller effect on current prototypes.
  • C. Varying the Margin (∆) in Lcont: Increasing margin ∆ improves performance on known and unknown classes, supporting greater separation in the latent space.∆ sets the minimum distance between an input feature vector and dissimilar class prototypes.
  • D. Varying the Temperature (T) in Eqn. 4: Temperature T = 1 is used throughout, while T = 2 gives slight improvement in unknown detection and larger values hurt performance.The reported optimal temperature lies between T = 1 and T = 2.
  • Methodological rationale: Contrastive clustering separates unknowns from known instances and separates classes to alleviate forgetting.The 2048-dim RoI-head feature vector is contrastively clustered, with the contrastive loss added to Faster R-CNN and trained end-to-end.

F. Further Implementation Details

The implementation uses queued feature storage, periodic prototype updates, and standard SGD training within an incremental object-detection framework. ORE adds modest runtime and memory costs, while one threshold-based unknown-identification variant performs worse than ORE.

  • F. Further Implementation Details: Each task is trained for 8 epochs, approximately 50k iterations, using SGD with learning rate 0.01 on 8 Nvidia V100 GPUs.The effective batch size is 8.
  • F. Further Implementation Details: ORE uses queue size 20, begins clustering after 1k iterations, and updates prototypes every 3k iterations with η = 0.99.
  • F. Further Implementation Details: The class-incremental object-detection setting requires adaptation to newly observed classes without retraining from scratch on old classes.Prevailing approaches use knowledge distillation to regularize learning and avoid forgetting old-class information.
  • F. Further Implementation Details: ORE adds 0.1349 sec/iter during training and inference and 0.009 sec/iter over standard Faster R-CNN, while exemplar memory for Nex = 50 uses approximately 34 MB.The storage expense for maintaining FStore is negligible.
  • F. Further Implementation Details: A max(softmax(logits)) < t criterion with t = {0.3, 0.5, 0.7} achieves A-OSE 11815 ± 352.13, WI 0.0436 ± 0.009, and mAP 55.22 ± 0.02, inferior to ORE.

J. Qualitative Results

Qualitative results show ORE identifying unknown instances and incrementally learning them after labels arrive. The examples also expose failures under occlusion, crowding, difficult viewpoints, and cluttered or small-object conditions.

  • J. Qualitative Results: Across Figs. 8–13, ORE identifies unknown instances and detects them after new classes are incrementally added.Sub-figure (a) shows identified unknowns before adding classes, while sub-figure (b) shows detections afterward.
  • K. Discussion Regarding Failure Cases: Occlusion and crowding confuse ORE, while difficult viewpoints and small unknown objects beside larger known objects also cause misclassifications or missed detections.Reported examples include external-storage, walkman, bag, and giraffe→horse.
  • J. Qualitative Results: ORE identifies a kite as unknown after Task 1 and later detects both kite and aeroplane after learning kite in Task 3.
  • J. Qualitative Results: After Task 2, ORE identifies unseen apple and orange instances as unknown; after Task 3, it labels them correctly while still detecting another unknown.
  • J. Qualitative Results: ORE initially identifies clock as unknown and later learns it in Task 4, demonstrating incremental learning of an identified unknown.
  • J. Qualitative Results: Additional examples show toothbrush, suitcase, book, laptop, and mouse being learned or detected after their task labels arrive.Figure 11 also reports external-storage and walkman as unknowns that are not detected after Task 4.
  • J. Qualitative Results: In a cluttered scene, ORE detects clock and reduces false positives for car and bicycle after Task 4, but fails to label the red suitcase.
Loading 2103.02603v2…