Source-linked AI summary
Zero-Shot Object Detection
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, Ajay Divakaran
TL;DR
Zero-shot object detection asks models to localize object classes absent from training, a harder setting than classification because detection requires precise localization and background discrimination. The paper adapts visual-semantic embeddings, introduces static and latent background-aware training plus dense label-space sampling, and evaluates them on MSCOCO and VisualGenome. Results show dataset-dependent gains and trade-offs, while the authors identify ontology-aware detection and improved regression and hard-negative mining as open limitations.
Problem
Zero-shot object detection must localize diverse object classes not observed during training, while detection also requires precise bounding boxes and appropriate handling of ambiguous background regions.
Method
The paper aligns bounding-box features with semantic class embeddings and adds fixed-background, latent-assignment background modeling and dense semantic-label sampling with auxiliary data.
Results
The proposed methods produce dataset-dependent results: DSES raises MSCOCO recall from 22.14% to 27.19%, while LAB reaches 5.40% recall on VisualGenome versus 5.19% for baseline.
Takeaways & Limitations
Background handling and semantic-space coverage materially affect zero-shot detection performance, with the most effective approach depending on dataset composition and seen-class coverage.
Takeaways & Limitations
Further work is needed to incorporate lexical ontology information and generalize bounding-box regression and hard-negative mining to new objects.
Abstract
from arXiv · showhide
We introduce and tackle the problem of zero-shot object detection (ZSD), which aims to detect object classes which are not observed during training. We work with a challenging set of object classes, not restricting ourselves to similar and/or fine-grained categories as in prior works on zero-shot classification. We present a principled approach by first adapting visual-semantic embeddings for ZSD. We then discuss the problems associated with selecting a background class and motivate two background-aware approaches for learning robust detectors. One of these models uses a fixed background class and the other is based on iterative latent assignments. We also outline the challenge associated with using a limited number of training classes and propose a solution based on dense sampling of the semantic label space using auxiliary data with a large number of categories. We propose novel splits of two standard detection datasets - MSCOCO and VisualGenome, and present extensive empirical results in both the traditional and generalized zero-shot settings to highlight the benefits of the proposed methods. We provide useful insights into the algorithm and conclude by posing some open questions to encourage further research.
1 Introduction
Zero-shot object detection transfers knowledge from seen to unseen classes while requiring precise localization, making it more difficult than zero-shot classification. The paper introduces ZSD, addresses background and sparse-class challenges, and evaluates proposed methods on standard datasets.
- Problem: Zero-shot detection trains on labeled examples from some classes but must localize instances of classes absent during training.The approach transfers visual models through semantic relationships between seen and unseen classes.
- Problem: Detection is harder than classification because it must predict bounding boxes for multiple objects and handle appearance, occlusion, viewpoint, and aspect-ratio variation.Contextual correlations that may help classification can degrade precise localization.
- Motivation: Scaling supervised detection to thousands of categories is constrained by the expense of obtaining bounding-box annotations.Existing approaches have achieved strong results on datasets with tens to a few hundred categories, while larger-scale detection remains insufficiently studied.
- Challenges: ZSD introduces an additional background-class problem because background regions may contain unseen objects rather than only conventional background content.The paper motivates background-aware detectors to address this ambiguity.
- Contributions: The paper proposes a baseline ZSD method, two background-aware detectors, dense semantic-label sampling with auxiliary data, and new MSCOCO and VisualGenome splits.It reports quantitative and ablation studies in traditional and generalized zero-shot settings.
2 Related Work
Prior zero-shot classification methods use semantic information to recognize unseen categories, while object detection methods have mainly focused on supervised localization. This paper adapts multimodal embeddings and related zero-shot learning ideas to novel object detection.
- Word embeddings: Word embeddings represent words as continuous vectors that encode semantic similarity and provide a shared space for images and class labels.The paper uses this shared space to enable detection of objects from unseen categories.
- Zero-shot image classification: Zero-shot classification has used attributes and multimodal embeddings to learn compatibility between image representations and class-label embeddings.These approaches motivate transferring semantic information to unseen object categories.
- Object detection: The paper follows prior zero-shot classification methodology by combining labeled object bounding boxes with unsupervised word embeddings for novel object detection.This extends semantic transfer from classification to localized detection.
3 Approach
The approach aligns bounding-box features with semantic class embeddings, then extends the baseline with background-aware training and denser sampling of the label space. Background boxes are either handled statically or assigned latent labels iteratively.
- Baseline ZSD: The baseline defines seen, unseen, and other classes, trains only on seen-class boxes, and transfers detection to unseen classes through semantic embeddings.Bounding-box features and class labels are represented in a common semantic space.
- Baseline ZSD: A projection maps deep bounding-box features into the semantic space, where cosine similarity and a max-margin loss align boxes with their true class embeddings.The training objective requires the matching score for the true class to exceed scores for other classes.
- Baseline ZSD: The model regularizes semantic alignment with a reconstruction loss and predicts each test label by selecting the class with the highest similarity score.The reconstruction loss uses squared L2 distance between reconstructed and original deep features.
- Background-aware detectors: Background-aware detection is difficult because background boxes may represent conventional background regions or instances of unseen object classes.The static-background approach extends the baseline with a fixed background class, while the paper also develops a latent-assignment alternative.
- Background-aware detectors: The latent-assignment method alternates between assigning sampled background boxes to classes outside the seen and unseen sets and retraining the detector with those assignments.This EM-like procedure progressively incorporates newly labeled background data.
- Dense semantic sampling: Sparse sampling of seen classes leaves parts of the semantic embedding space without training examples, especially near some unseen classes.The approach addresses this practical problem by densely sampling labels using additional data.
4 Experiments
Experiments evaluate zero-shot detection on MSCOCO and VisualGenome using controlled splits, background-aware models, dense semantic sampling, and generalized detection settings. Results show that model choice depends on dataset structure, while background modeling and dense sampling can improve detection quality in appropriate settings.
- Datasets and protocol: The experiments evaluate ZSD on MSCOCO and VisualGenome using clustered seen/unseen class splits and proposal-based training and evaluation.Classes are divided by clustering word-vector embeddings, and Recall@K is used because missing annotations make mAP sensitive to false positives.
- Quantitative results: On MSCOCO, SB improves recall from 22.14% for baseline to 24.39%, whereas background boxes in VisualGenome can include seen or unseen objects.Removing images containing unseen classes from MSCOCO training prevents this contamination and makes the static-background representation more robust.
- Quantitative results: Dense semantic sampling raises MSCOCO recall from 22.14% to 27.19% but does not improve VisualGenome, which already has many seen classes.Training classes expand from 48 to 378 on MSCOCO, while VisualGenome expands from 478 to 716, limiting the benefit of the same sampling strategy.
- Quantitative results: On VisualGenome, LAB reaches 5.40% recall, compared with 5.19% for baseline, 4.09% for SB, and 4.75% for DSES.LAB consistently improves over SB across IoU thresholds, supporting latent assignment of background objects across the embedding space.
- Class-wise analysis: Class performance depends on both visual appearance and semantic relationships to seen classes, as illustrated by strong gains for cat with DSES and failures for zebra.The zebra detector transfers giraffe-related knowledge but misses distinguishing visual attributes such as stripes and body structure; LAB performs similarly or better than baseline on 104 of 130 VisualGenome classes.
5 Discussion and Conclusion
The paper combines visual-semantic embeddings, background-aware detectors, auxiliary-data sampling, and new MSCOCO and VisualGenome splits for zero-shot detection. It also identifies lexical ontology, bounding-box regression, and hard-negative mining as areas needing further work.
- Discussion and Conclusion: The paper validates visual-semantic embeddings and two background-aware approaches for zero-shot detection using quantitative and qualitative results.The approaches use either a fixed background class or iterative latent assignment, alongside denser semantic-label sampling with auxiliary data.
- Discussion and Conclusion: Novel MSCOCO and VisualGenome splits support extensive evaluation of the proposed zero-shot detection methods.
- Discussion and Conclusion: Future work should incorporate lexical ontology relationships and generalize bounding-box regression and hard-negative mining for new objects.The paper specifically mentions “is a” and “is part of” relationships and hierarchy-aware loss as relevant directions.