Source-linked AI summary
Zero-Shot Grounding of Objects from Natural Language Queries
Arka Sadhu, Kan Chen, Ram Nevatia
TL;DR
Phrase grounding traditionally restricts queried nouns to categories seen during training, creating a gap for localizing novel objects and a proposal-generation challenge. The paper introduces zero-shot grounding and ZSGNet, a single-stage detector-grounding model trained directly on grounding data, with four evaluation conditions and new datasets. ZSGNet outperforms the baseline by 4−8% across unseen splits and performs significantly better than existing baselines in zero-shot settings.
Problem
Existing phrase-grounding systems restrict queried nouns to object types encountered during training, while detector-based proposal generation limits coverage of novel categories.
Method
ZSGNet combines detection and grounding in a single-stage network that uses dense proposals, predicts classification and regression outputs, and trains end-to-end on grounding data.
Results
ZSGNet achieves 4−8% higher performance than QRG across four unseen splits and performs significantly better than existing baselines in zero-shot settings.
Takeaways & Limitations
The paper establishes zero-shot grounding as an evaluated extension of phrase grounding through four conditions and datasets sampled from Visual Genome and Flickr30k Entities.
Takeaways & Limitations
Detector-based grounding can classify unannotated instances of a novel category as background, and ReferIt lacks entity information needed for detector fine-tuning.
Abstract
from arXiv · showhide
A phrase grounding system localizes a particular object in an image referred to by a natural language query. In previous work, the phrases were restricted to have nouns that were encountered in training, we extend the task to Zero-Shot Grounding(ZSG) which can include novel, "unseen" nouns. Current phrase grounding systems use an explicit object detection network in a 2-stage framework where one stage generates sparse proposals and the other stage evaluates them. In the ZSG setting, generating appropriate proposals itself becomes an obstacle as the proposal generator is trained on the entities common in the detection and grounding datasets. We propose a new single-stage model called ZSGNet which combines the detector network and the grounding system and predicts classification scores and regression parameters. Evaluation of ZSG system brings additional subtleties due to the influence of the relationship between the query and learned categories; we define four distinct conditions that incorporate different levels of difficulty. We also introduce new datasets, sub-sampled from Flickr30k Entities and Visual Genome, that enable evaluations for the four conditions. Our experiments show that ZSGNet achieves state-of-the-art performance on Flickr30k and ReferIt under the usual "seen" settings and performs significantly better than baseline in the zero-shot setting.
1. Introduction
The paper extends phrase grounding to zero-shot grounding, where queried nouns may be unseen during training, and addresses proposal-generation limits with an end-to-end single-stage model. It also defines four evaluation conditions and introduces datasets for testing them.
- Problem: Zero-shot grounding allows queries whose nouns were not encountered during training, extending phrase grounding beyond previously seen object types.Existing systems accept novel phrases but restrict their nouns to categories seen in training images.
- Problem: Existing two-stage systems rely on detectors trained on fixed object categories, limiting proposals for novel objects and requiring the system to find suitable regions first.For zero-shot grounding, the model must classify a novel region as foreground and regress its box more accurately.
- Approach: ZSGNet replaces proposal generation followed by classification with a single-stage network using dense proposals and combined language-visual features.It predicts classification scores and regression parameters, trains end-to-end on grounding data, and avoids an externally trained detector.
- Evaluation: The paper distinguishes four evaluation conditions based on query novelty, object-category novelty, semantic similarity to seen objects, and whether similar objects appear in the test image.These conditions separate novel queries, novel referred categories, similar-category cases without matching objects, and cases where a similar trained object is also present.
- Evaluation: New datasets subsampled from Visual Genome and Flickr30k Entities support evaluation across the four zero-shot grounding conditions.The datasets are designed to create examples for each defined case.
- Contributions: The paper introduces zero-shot grounding, proposes ZSGNet, creates evaluation datasets, and evaluates the model on these datasets.The stated contributions include publicly released code and datasets.
2. Related Work
Prior phrase-grounding work relies heavily on annotated datasets, multimodal representations, attention, and pretrained object detectors. This paper differs by combining detection and grounding so it can learn directly from grounding data without detector preprocessing, while retaining efficient single-stage inference.
- Phrase grounding: Phrase-grounding research has developed extensive datasets and methods including reconstruction, global context, multimodal feature learning, attention, and semi-supervised approaches.The cited datasets include Flickr30k, ReferIt, RefCoCo variants, Visual Genome, and GuessWhat.
- Detector-based systems: Common systems use Faster R-CNN or Mask R-CNN to obtain bounding boxes and ROI-pooled features before grounding.This preprocessing restricts systems to object categories represented by the detector.
- This work: The paper combines detection and grounding and learns directly from the grounding dataset without a preprocessing detector.This design targets categories not covered by the detector's fixed label set.
- Single-stage inference: Single-stage detection networks such as SSD, YOLO, FPN, and RetinaNet are associated with efficient or real-time inference.The paper integrates a single-stage detection network into grounding to support zero-shot grounding and efficient inference.
- Related zero-shot work: Related zero-shot work includes open-vocabulary retrieval and zero-shot detection, but the cited zero-shot detection setting requires the background-class set before training.The paper positions its task among these related formulations while distinguishing its grounding setting.
3. Design Considerations for ZSG
ZSG requires grounding systems to handle unseen nouns and object categories, exposing limitations in sparse, detector-dependent two-stage pipelines. ZSGNet addresses these issues with dense, query-conditioned proposals and an end-to-end single-stage architecture.
- ZSG Cases: Zero-shot grounding includes cases involving unseen query nouns, unseen object categories, and semantically similar objects absent or present in the test image.Semantic similarity is defined using clustered word embeddings.
- Limitations in Phrase Grounding Systems: Traditional grounding selects or regresses sparse proposals generated by an object detector trained on fixed categories.This limits generalization to novel categories and can classify unannotated novel objects as background.
- Model Design: ZSGNet replaces the two-stage pipeline with dense, image-size-dependent candidate boxes that combine visual and language features before prediction.The model predicts classification scores and box-regression parameters without a pre-trained object detector.
- Model Design: The architecture encodes the query with a Bi-LSTM, combines it with multi-resolution visual feature maps and location information, and applies fully convolutional prediction.Each candidate box receives a score and regression parameters.
- Model Design: Training classifies anchors as foreground or background and regresses foreground anchors toward tighter boxes using focal and SmoothL1 losses.Anchors match ground truth at IoU ≥0.5, and the final loss is L = Lpred + λLreg with λ = 1.
4. Experiments
The experiments construct zero-shot grounding splits from Flickr30k Entities and Visual Genome, then evaluate ZSGNet on existing and unseen grounding settings. ZSGNet outperforms baselines across unseen splits, with performance varying by semantic distance and training configuration.
- Dataset construction: The experiments create zero-shot datasets by sub-sampling Flickr30k Entities and Visual Genome for the defined evaluation cases.Flickr-Split-0 and Flickr-Split-1 come from Flickr30k, while VG-Split-2 and VG-Split-3 come from Visual Genome.
- Dataset construction: The datasets include separate training, validation, and test splits, with balanced and unbalanced Visual Genome test sets.Flickr-Split-0 excludes query words from training, Flickr-Split-1 reserves the “other” category for validation and test, and VG-Splits report both balanced and unbalanced sets.
- Existing grounding datasets: ZSGNet improves over prior methods on existing Flickr30k and ReferIt evaluations, including an approximately 9% gain over other methods on ReferIt.The reported ReferIt advantage is attributed to learning entities directly from the grounding dataset rather than using unavailable external entity information.
- Zero-shot grounding datasets: ZSGNet achieves 4−8% higher performance than QRG across all four unseen splits despite QRG using an object detector pretrained on Pascal VOC.Accuracy is higher on Flickr-Split-0,1 than on the Visual Genome split, while balanced and unbalanced test accuracy remains the same.
- Zero-shot grounding datasets: Accuracy declines smoothly as the semantic distance between an unseen object and its closest seen object increases.Semantic distance is computed with the L2-norm between GloVe embeddings, and examples are grouped into five unit-length intervals.
- Ablation study: Binary Cross Entropy and Focal loss provide a significant 7% boost over the base loss configuration, while image resizing adds another 4% increase.The ablation uses the ReferIt validation set and the base ZSGNet model.
- Visualization: Visualization shows that GloVe embeddings support generalization to unseen words, while visually similar seen objects can still attract incorrect grounding.The model handles “cigar” and “kayak” despite unseen query words, but confuses “planter” with “plants” when the objects are visually similar.
5. Conclusion
The paper introduces Zero-Shot Grounding for localizing novel objects from query phrases, proposes ZSGNet, and evaluates it against existing baselines.
- Zero-Shot Grounding localizes novel objects referred to by query phrases.
- The paper proposes ZSGNet to address limitations of previous phrase-grounding systems.
- New datasets sampled from existing datasets enable evaluation across four zero-shot grounding cases.
- ZSGNet performs significantly better than existing baselines in the zero-shot setting.
Appendix
The supplementary document provides dataset-sampling details, dataset distributions, and additional analyses of proposal recall and model ablations.
- The supplement details dataset sampling from Flickr30k Entities and Visual Genome and reports their distributions.
- It also provides proposal-recall results and image-blind and language-blind model ablations.
A. Dataset Construction
The datasets are constructed from Flickr30k Entities and Visual Genome to evaluate distinct zero-shot grounding conditions, with cleaning, clustering, splitting, and balancing procedures.
- Dataset sources: Flickr30k Entities supplies noun-phrase bounding boxes and entity labels, while Visual Genome supplies data for additional zero-shot cases.
- Case 0: Case 0 lemmatizes query words, keeps the top 1,000 as always seen, and splits remaining words into disjoint seen and unseen lists.
- Case 0: Flickr-Split-0 tests on images containing unseen query words while training excludes those annotations and uses non-overlapping images.
- Visual Genome construction: Visual Genome processing cleans phrase annotations, resolves overlapping phrase boxes, and recommends IoU 0.3 because the annotations remain spatially imprecise.
- Cases 2 and 3: Case 2 contains test images with unseen objects but no same-image seen object, whereas Case 3 includes such a seen object.
- Balancing: The resulting training data are cluster-imbalanced, so the construction applies threshold-based balancing to compensate.
A.4. Choice of Datasets
Dataset choice depends on the requirements of each zero-shot case and the annotation properties of the available datasets.
- Flickr30k Entities is used for Case 0 and Case 1, while Visual Genome is used for Cases 2 and 3.
- Flickr30k Entities lacks synsets needed for synonym disambiguation, whereas Visual Genome lacks broad categories such as vehicles.
- Flickr30k Entities is chosen for Case 0 because it provides more precise bounding boxes.
B. Dataset Distributions
The datasets are characterized by category and cluster distributions across zero-shot cases, with “other” occupying a larger share in some validation and test splits. Balanced cluster distributions indicate that the balancing strategy succeeded.
- The “other” category occupies a larger share of the Case0 validation and test sets because it has more diverse vocabulary and covers more excluded words.
- Case1 validation and test sets contain only the “other” category, so entity-wise distributions are not meaningful there.
- Case2 and Case3 distributions are reported over K-Means clusters for both unbalanced and balanced datasets.
- The balanced sets have uniform cluster distributions, indicating that the balancing strategy was successful.
C. Proposals from Pre-Trained Detector(s)
The proposal analysis examines whether missing regions explain grounding errors when prior systems rely on pretrained detectors. Proposal recall supports using dense proposals for zero-shot grounding, especially where unseen categories reduce detector coverage.
- Proposal recall is measured at 300 proposals for Faster R-CNN region proposal networks pretrained on Pascal VOC and fine-tuned on target data.
- Fine-tuning on the target dataset significantly improves proposal recall, and QRG performance on Flickr30k Case0 and Case1 follows the same trend.
- Visual Genome has substantially lower proposal recall because it contains more classes and treats unseen classes as negatives during training.
- These recall patterns motivate dense proposals for zero-shot grounding, avoiding dependence on proposals from a pretrained detector.
D. Image Blind and Language Blind Ablations
The ablations show that removing either language or image information undermines grounding, although image-blind predictions exploit center bias in Flickr30k and ReferIt. This shortcut is much weaker on Visual Genome.
- Language-blind models see only the image and produce very low accuracy because multiple correct solutions can exist for one image.
- Image-blind models see only the query and learn to predict a central box, yielding 25−30% accuracy on Flickr30k and ReferIt.
- Image-blind accuracy falls to 9−10% on Visual Genome, which has denser object annotations.