PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Object Detection

Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, Ajay Divakaran

arXiv:1804.04340v2cs.CV

TL;DR

Zero-shot object detection aims to localize classes absent from training, a challenge compounded by the expense of annotating large vocabularies. The paper combines visual-semantic embeddings with background-aware detectors and auxiliary semantic-space sampling, reporting dataset-dependent results including improved MSCOCO recall with dense sampling.

  • Problem

    Detecting unseen object classes remains underexplored, in part because bounding-box annotation across thousands of categories is expensive.

    Source: §1 · p. 2

  • Method

    The paper adapts visual-semantic embeddings for ZSD, proposes fixed-background and latent-assignment detectors, and densely samples semantic labels using auxiliary category data.

    Source: §1 · p. 3, §5 · p. 13

  • Results

    Results were dataset-dependent: dense sampling raised MSCOCO recall from 22.14% to 27.19%, while latent assignment outperformed alternatives on VisualGenome at 5.40% recall.

    Source: §4.3 · p. 11

  • Takeaways & Limitations

    Dense sampling can improve unseen-class prediction when relevant regions of the embedding space are sparsely sampled.

    Source: §4.3 · p. 11

    Further work is needed to generalize bounding-box regression and hard-negative mining to new objects.

    Source: §5 · p. 13

Abstract

from arXiv · show

We introduce and tackle the problem of zero-shot object detection (ZSD), which aims to detect object classes which are not observed during training. We work with a challenging set of object classes, not restricting ourselves to similar and/or fine-grained categories as in prior works on zero-shot classification. We present a principled approach by first adapting visual-semantic embeddings for ZSD. We then discuss the problems associated with selecting a background class and motivate two background-aware approaches for learning robust detectors. One of these models uses a fixed background class and the other is based on iterative latent assignments. We also outline the challenge associated with using a limited number of training classes and propose a solution based on dense sampling of the semantic label space using auxiliary data with a large number of categories. We propose novel splits of two standard detection datasets - MSCOCO and VisualGenome, and present extensive empirical results in both the traditional and generalized zero-shot settings to highlight the benefits of the proposed methods. We provide useful insights into the algorithm and conclude by posing some open questions to encourage further research.

1 Introduction

This section introduces zero-shot object detection (ZSD) for diverse unseen object categories and explains why localization and ambiguous background regions make detection challenging. It proposes semantic-embedding, background-aware, and auxiliary-class-sampling methods, evaluated in traditional and generalized zero-shot settings.

1 Introduction

  • Detection adds precise multi-object localization and invariance demands beyond classification, while scaling supervised annotations to thousands of categories is prohibitively expensive.Detection also relies less on contextual cues and can be degraded by contextual correlations.Source: §1 · p. 2
  • ZSD targets diverse unseen object categories, extending zero-shot classification from semantic class transfer to predicting bounding boxes for multiple objects.In the described setting, training examples come from seen classes and testing includes semantically related classes not observed during training.Source: §1 · pp. 1–3
  • The paper adapts a multimodal semantic-embedding architecture as a ZSD baseline and addresses ambiguous background regions with fixed-background and latent-assignment open-vocabulary methods.Background regions may include both background “stuff” and objects from unannotated or unseen classes.Source: §1 · p. 3
  • Because detection benchmarks cover only a few hundred classes and sparsely sample the label space, the authors use additional data to densify training classes and conduct traditional and generalized evaluations and ablations.The paper reports qualitative and quantitative results to examine the methods’ benefits, shortcomings, and failure cases.Source: §1 · p. 3

2 Related Work

The related work motivates shared semantic representations for detecting unseen categories and distinguishes this proposal-based setting from prior classification, detection, multimodal, and zero-shot detection approaches.

Word embeddings

  • This work uses word embeddings as a shared vector space for images and class labels, enabling detection of objects from unseen categories.The embeddings derive from word co-occurrences in large text corpora.Source: §2 · p. 3

Zero-shot image classification

  • Prior zero-shot classification used attributes, multimodal compatibility models, latent variables, or deep visual-semantic embeddings; this work detects novel categories from labeled boxes and unsupervised word embeddings.The authors refer readers to Fu et al. [13] for a comprehensive survey.Source: §2 · p. 4

Object detection

  • Unlike detectors that scale to 9000 categories [45], this work assumes no training labels for test classes and builds on proposal-based detection approaches.Earlier detection methods classified object proposals with image-classification CNNs, while later methods used a single deep-network pass without proposals.Source: §2 · p. 4

Multi-modal learning

  • Prior multimodal work spans cross-modal retrieval [9], shared image-text-sound representations [4], visual entity localization [58], and cross-domain vision-language descriptors [18]; this work detects unseen classes without attributes, unlike.Li et al. learn object-specific attributes to classify, segment, and predict novel objects; this work targets large object sets in unconstrained settings.Source: §2 · p. 4

Comparison with recent works on ZSD

  • Zhu et al. [61] generate proposals for unseen objects, while Rahman et al. [42] combine max-margin and semantic-clustering losses; this work uses MSCOCO and VisualGenome and addresses unseen-class background corruption.Rahman et al. train and evaluate on ILSVRC-2017, a more constrained dataset with about one object per image on average, and do not consider background corruption by unseen classes.Source: §2 · p. 4

3 Approach

The approach adapts visual-semantic embeddings for zero-shot detection, adds background-aware training to address background diversity, and uses auxiliary data to densely sample the semantic label space.

3.1 Baseline Zero-Shot Detection (ZSD)

  • Baseline ZSD projects region features into shared semantic space, learns cosine-similarity max-margin alignment, adds reconstruction regularization following Kodirov et al. [23], and labels boxes by nearest class embedding.It trains only on seen-class boxes because background proposals may contain unseen objects.Source: §3.1 · pp. 5–6

3.2 Background-Aware Zero-Shot Detection

  • Fixed-background SB adds one background embedding, but diverse backgrounds and unseen objects may collapse into it; LAB instead iteratively assigns background boxes to multiple classes in O and retrains.LAB encodes that background boxes do not belong to seen classes S and may belong to classes in the larger background set O.Source: §3.2 · pp. 6–7

3.3 Densely Sampled Embedding Space (DSES)

  • DSES augments training with external boxes from classes in C − U to densely sample the embedding space, addressing sparse seen-class coverage and improving baseline generalization.The extra data is from diverse external sources and is distinct from seen and unseen classes.Source: §3.3 · p. 7

4 Experiments

Experiments evaluate zero-shot detection on MSCOCO and VisualGenome using new seen/unseen splits, and test background-aware training, dense semantic-space sampling, and generalized zero-shot detection. Results show different approaches work best by dataset, while ablations and qualitative examples examine detection quality and limitations.

4 Experiments

  • The splits use word-vector clustering to assign 80% of classes per cluster to training and 20% to test, yielding 48/17 classes for MSCOCO and 478/130 for VG.MSCOCO training images containing unseen classes were removed to avoid treating unseen objects as background; VG images could not be filtered similarly because doing so would eliminate most training data.Source: §4 · p. 8

4.1 Implementation Details

  • Proposals are assigned seen-class or background labels by IoU, with partial background-box sampling; LAB iteratively labels background boxes, and OI augmentation adds 330 MSCOCO and 238 VG classes.Source: §4.1 · pp. 8–9

4.2 Evaluation Protocol

  • Recall@K is used because incomplete VG annotations can make mAP treat valid detections as false positives; mAP is also reported for fully annotated MSCOCO.A detection is a true positive when its IoU with a ground-truth box exceeds threshold t and no higher-confidence prediction has already been assigned to that ground-truth box.Source: §4.2 · p. 10

4.3 Quantitative Results

  • On MSCOCO, SB reaches 24.39% recall versus 22.14% for baseline, while VG SB falls to 4.09% versus 5.19% for baseline when background boxes map to one vector.The authors attribute the difference to removing unseen-class images from MSCOCO training, unlike VG, where background boxes may include seen or unseen objects.Source: §4.3 · p. 10
  • On VG, LAB reaches 5.40% recall versus 5.19% for baseline, 4.09% for SB, and 4.75% for DSES; on MSCOCO, DSES reaches 27.19% versus 22.14% for baseline.The authors focus on LAB for VG and SB for MSCOCO, citing their appropriateness for each dataset.Source: §4.3 · p. 11

4.4 Generalized Zero-Shot Detection (GZSD)

  • At nt = 0.2 on MSCOCO, DSES records Recall@100 of 15.02% for seen and 15.32% for unseen classes (HM 15.17%), versus LAB’s 14.54% and 10.57% (HM 12.24%).The generalized setting evaluates seen and unseen classes together; the method uses a novelty-detection threshold to choose between the most probable training and test classes.Source: §4.4 · p. 12

4.5 Ablation Studies

  • On MSCOCO, SB’s recall falls by an average 1.14 percentage points from K = All to K = 100, compared with 3.37 points for baseline; background-aware models also outperform baselines at lower K and higher IoU.The authors interpret these trends as evidence that background-aware models produce higher-quality detections.Source: §4.5 · p. 13

4.6 Qualitative Results

  • The model detects “stuff” classes such as “vegetation” and “floor”, but captures no true-positive “zebra” detections, reflecting difficulty inferring fine-grained differences from semantically related seen classes.Performance depends on class appearance and relationships to seen classes. Versus baseline, LAB achieves similar or better performance on 104 of 130 VG classes, while SB and DSES do so on 12 and 13 of 17 MSCOCO classes, respectively.Source: §4.3 · pp. 11–12, §4.6 · p. 13

5 Discussion and Conclusion

The paper adapts visual-semantic embeddings for zero-shot object detection, introducing two background-aware approaches and auxiliary-data sampling to improve semantic-space coverage. It also identifies future work on lexical hierarchies and adapting detection components to new objects.

5 Discussion and Conclusion

  • The work proposes fixed-background and latent-assignment detectors, denser semantic-label sampling with auxiliary data, and novel MSCOCO and VisualGenome splits, validated with quantitative and qualitative results.The latent-assignment approach iteratively assigns background boxes to classes.Source: §5 · p. 13
  • Future work should incorporate lexical ontology relationships, which few prior works address [18], and generalize bounding-box regression and hard-negative mining to new objects.The passage notes that hierarchical predictions such as “cat” as “animal” should incur less loss than predictions such as “cat” as “vehicle.”Source: §5 · p. 13

Related papers

Loading 1804.04340v2…