Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wild

1. Introduction

• Object-level OOD detection is underexplored, and detectors can confidently label unseen objects as known classes; annotating many OOD objects in complex scenes is costly.A scene may contain OOD regions alongside in-distribution objects.Source: §1 · p. 1
• STUD distills unknown objects from videos in the wild to regularize the detector’s decision boundary, using videos that contain both known and unknown objects.The authors describe this as exploiting video information for OOD identification in object detectors.Source: §1 · pp. 1–2
• Spatial-temporal distillation selects candidate unknowns using an OOD measurement, combines their features by dissimilarity, and aggregates objects across frames for diversity.The framework also trains an uncertainty regularization branch that contrastively shapes the uncertainty surface between in-distribution and distilled unknown objects.Source: §1 · p. 2
• STUD distills diverse unknowns without costly human OOD annotations and is reported as more advantageous than pixel-space synthesis or negative proposals.The paper presents video-based unknown distillation as its first use of rich video information for OOD identification in object detection.Source: §1 · p. 2
• 10.88% in FPR95 on BDD100K: STUD outperforms the best baseline while preserving object-detection accuracy on in-distribution data.The framework was evaluated on BDD100K and Youtube-VIS [66].Source: §1 · p. 2

2. Problem Setup

• Object-level OOD detection asks a detector trained on known categories to identify test-time objects outside those categories and refrain from confident predictions.The setup frames detection as binary classification between in-distribution and out-of-distribution objects.Source: §2 · p. 2
• The detector’s training set consists of in-distribution images, bounding boxes, and semantic labels drawn from an unknown joint distribution.The setup denotes the image and label spaces as X = R^d and Y = {1, 2, ..., K}.Source: §2 · p. 2
• Neural networks model bounding-box regression pθ(b|x) and object classification pθ(y|x, b).For a predicted test object, the OOD task predicts whether it is in-distribution or OOD, with OOD objects outside the support of Y.Source: §2 · p. 2

3. Unknown-Aware Object Detection

• STUD trains an object detector and uncertainty branch jointly, using distilled unknown objects to regularize the model for OOD detection.The two components address unknown-object distillation and model regularization.Source: §3 · p. 2, §3.1 · p. 3

3.1. Spatial-Temporal Unknown Distillation

• Sampling multiple nearby video frames expands the reference-object pool, allowing temporal aggregation to discover diverse unknown counterparts.The method samples T frames within [t0 − R, t0 + R]; T = 1 is the pair-frame setting.Source: §3.1 · p. 4
• Energy filtering selects reference-frame proposals with mild scores; dissimilarity-weighted feature combinations then form unknown counterparts for key-frame ID objects.Higher dissimilarity indicates more dissimilar features, and proposals are selected within a specified energy-score percentile range.Source: §3.1 · pp. 3–4

3.2. Unknown-Aware Training Objective

• The uncertainty loss contrasts ID-object and distilled-unknown energy scores; after training, their distributions separate as the loss converges.Figure 4(a) shows the loss during training, and Figure 4(b) shows the post-training negative-energy distributions.Source: §3.2 · p. 4, Figure 4 · p. 5
• At inference, STUD uses the logistic-regression uncertainty output and a threshold to distinguish ID from OOD objects; the threshold can target high ID accuracy.The threshold is typically chosen so a high fraction of ID data, such as 95%, is correctly classified; the components also benefit one another through energy-based filtering and regularization.Source: §3.2 · p. 5

4.1. Benchmark Construction

• BDD100K and Youtube-VIS serve as ID datasets, with MS-COCO and nuImages as OOD datasets; the data are deduplicated to avoid semantic overlap.OOD performance is measured with FPR95 and AUROC, and ID detection with mAP.Source: §4.1 · p. 5

4.2. Comparison with Baselines

• 15.77% on BDD100K with COCO as OOD and 17.66% on Youtube-VIS with nuImages as OOD: STUD improves FPR95 over GAN-based outlier synthesis.STUD also outperforms competitive baselines on both datasets while preserving ID object-detection accuracy.Source: §4.2 · p. 6, Table 1 · p. 6

4.3. Ablation Studies

• STUD outperforms alternatives that use independent objects, pixel-space synthesis, or Gaussian noise as distilled unknowns.Independent-object variants may lack diversity or be too simple; GAN synthesis is unstable, while mixup harms object-detection performance.Source: §4.3 · pp. 6–7, Table 2 · p. 6
• Mild-energy proposals perform best for unknown distillation, providing unknowns at a suitable difficulty level for contrastive uncertainty regularization.The filtering ablation uses T = 3 and R = 9.Source: §4.3 · p. 7, Table 3 · p. 7
• AUROC falls from 85.67% to 80.35% on COCO when R is infinite, while T = 3 improves AUROC by 5.24% over T = 1 on COCO.OOD detection benefits from reference frames mildly close to the key frame and from aggregating multiple frames, but a larger T hurts performance, possibly due to redundant features.Source: §4.3 · p. 7, Figure 5 · p. 7
• A mild uncertainty-regularization weight is desirable, and STUD outperforms baseline OOD methods on AUROC in most tested cases.The ablation evaluates COCO and nuImages as OOD datasets.Source: §4.3 · p. 7, Table 4 · p. 7
• 11.35% AUROC improvement on COCO as OOD: STUD’s logistic uncertainty loss outperforms the squared hinge alternative.Treating distilled unknowns as an extra class does not outperform either; the learnable weight θu adjusts the logistic slope for a sharper binary decision boundary.Source: §4.3 · p. 8, Table 5 · p. 8

4.4. Qualitative analysis

• On OOD images from MS-COCO and nuImages, STUD identifies OOD objects and reduces false positives relative to vanilla Faster R-CNN.False-positive confidence is also lower for STUD in the illustrated examples.Source: §4.4 · p. 8, Figure 6 · p. 8

Related work

• Classification OOD methods include post hoc scores, generative approaches, and outlier exposure using natural or GAN-synthesized images [65], but real outlier data is difficult to obtain for object detection.Generative methods are described as generally less competitive than discriminative methods and harder to optimize [19] [28] [52] [63].Source: §5 · p. 8
• Object-detection OOD work includes virtual outlier synthesis [8], negative-object samples [23], localization uncertainty [14], and approximate Bayesian methods [4] [40] [41].The approximate Bayesian methods require multiple inference passes, which are computationally expensive on larger datasets and models.Source: §5 · p. 8
• OOD generalization, zero-shot detection, and incremental detection differ from OOD detection; Wang et al. [62] address OOD generalization without temporal aggregation.The cited open-world methods address catastrophic forgetting [24] or use auxiliary information such as class attributes.Source: §5 · pp. 8–9
• Video anomaly detection targets anomalous events, which need not differ semantically from ID training data; prior approaches do not synthesize unknowns from video or use energy-based regularization.Source: §5 · p. 9

• STUD distills diverse unknown objects during training by exploiting videos’ rich spatial-temporal information.Source: §6 · p. 9
• The distilled unknowns meaningfully improve the decision boundary between ID and OOD data.Source: §6 · p. 9
• STUD achieves state-of-the-art OOD detection performance while preserving ID-task performance.Source: §6 · p. 9

A. Experimental details

• The OOD evaluation uses test data from MS-COCO and nuImages with labels disjoint from the respective ID datasets; Table 6 summarizes the tasks.Source: A. Experimental details · p. 12, Table 6 · p. 12
• For YouTube-VIS, the last 597 training videos form the ID evaluation set and the remaining 2,388 are used for training.Source: A. Experimental details · p. 12

B. In-distribution classes

• BDD100K has 8 in-distribution classes and YouTube-VIS has 40, with examples including car and pedestrian in BDD100K and airplane and bear in YouTube-VIS.Source: B. In-distribution classes · p. 12

C. Software and hardware

• Experiments used Python 3.8.5, PyTorch 1.7.0, and NVIDIA GeForce RTX 2080Ti GPUs.Source: C. Software and hardware · p. 12

D. Baselines

• MSP, ODIN, Generalized ODIN, Mahalanobis distance, CSI, energy score, and Gram matrices were applied to the detectors’ classification branch.Source: D. Baselines · p. 12
• For ODIN and Mahalanobis distance, noise magnitude was set to 0 because region cropping and ROIAlign prevent end-to-end differentiation.Source: D. Baselines · p. 12

E. Ablation study on a different backbone architecture

• The alternative-backbone evaluation uses Faster R-CNN with RegNetX-4.0GF and compares STUD with the same OOD baselines as the main paper.Source: E. Ablation study on a different… · p. 12
• 3%: CSI’s mAP decreases on YouTube-VIS, while STUD achieves stronger OOD detection and preserves or slightly increases ID mAP with RegNetX-4.0GF.Source: Table 7 · p. 13, E. Ablation study on a different… · p. 13
• Table 7 compares competitive OOD detection methods using a model trained on ID data only, with results reported as percentages.Source: Table 7 · p. 13

F. Additional visualization examples

• Figures 7–8 show MS-COCO and nuImages OOD images with BDD100K as the in-distribution dataset.The panels compare vanilla Faster-RCNN (top) with STUD (bottom).Source: Figure 7 · p. 14, Figure 8 · p. 15
• Blue marks objects detected and classified as in-distribution; green marks OOD objects detected by STUD, which reduce false positives among detected objects.The color key applies across the additional visualizations in Figures 7–10.Source: Figure 7 · p. 14, Figure 8 · p. 15, Figure 9 · p. 16
• Figures 9–10 show MS-COCO and nuImages OOD images with Youtube-VIS as the in-distribution dataset.The panels compare vanilla Faster-RCNN (top) with STUD (bottom).Source: Figure 9 · p. 16, Figure 10 · p. 17