Source-linked AI summary
Natural Language Object Retrieval
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, Trevor Darrell
TL;DR
Natural language object retrieval seeks to localize an object from a natural-language query while accounting for spatial information and global scene context. The paper proposes SCRC, a recurrent candidate-box scoring model that integrates these signals and transfers image-captioning knowledge; it reports improved performance over baselines and reduced over-fitting with small retrieval datasets.
Problem
Natural language object retrieval must localize queried objects beyond fixed categories, because descriptions can express attributes, spatial configurations, interactions, and scene context.
Method
SCRC recurrently scores candidate regions using query text, local descriptors, spatial configurations, and global context, with parameters pretrained on image captioning data.
Results
SCRC significantly outperforms baseline methods, and its full model achieves the highest top-1 precision in the reported ReferIt evaluation scenario.
Takeaways & Limitations
Image-level captioning datasets can alleviate the scarcity of large object-level annotation datasets through pretraining and adaptation.
Takeaways & Limitations
Training assumes annotated image-bounding-box-description tuples for objects, with each object associated with one or more descriptions.
Abstract
from arXiv · showhide
In this paper, we address the task of natural language object retrieval, to localize a target object within a given image based on a natural language query of the object. Natural language object retrieval differs from text-based image retrieval task as it involves spatial information about objects within the scene and global scene context. To address this issue, we propose a novel Spatial Context Recurrent ConvNet (SCRC) model as scoring function on candidate boxes for object retrieval, integrating spatial configurations and global scene-level contextual information into the network. Our model processes query text, local image descriptors, spatial configurations and global context features through a recurrent network, outputs the probability of the query text conditioned on each candidate box as a score for the box, and can transfer visual-linguistic knowledge from image captioning domain to our task. Experimental results demonstrate that our method effectively utilizes both local and global information, outperforming previous baseline methods significantly on different datasets and scenarios, and can exploit large scale vision and language datasets for knowledge transfer.
1. Introduction
Natural language object retrieval localizes an object from a natural-language description, extending beyond fixed-category detection by requiring spatial and scene-level reasoning. The proposed SCRC model scores candidate regions using text, local appearance, spatial configuration, and global context, while transferring knowledge from image captioning.
- Task motivation: Natural language object retrieval localizes an object in an image from a natural-language query describing it.The task supports descriptions involving categories, attributes, spatial configurations, and interactions.
- Task motivation: Unlike text-based retrieval on cropped regions, the task requires spatial configurations and global scene context in addition to object appearance.For example, identifying an object may require recognizing its category, attribute, action, and position relative to the scene.
- Proposed approach: SCRC learns a scoring function that combines the text query, candidate regions, spatial configurations, and global context to rank candidate regions.Its recurrent architecture uses embedded text and visual features as inputs.
- Proposed approach: The recurrent scoring model is trained end-to-end, allowing visual feature extraction and text-sequence embedding to adapt jointly.The paper reports that this approach significantly outperforms a previous bag-of-words method.
- Knowledge transfer: Pretraining on image captioning and subsequent adaptation to object retrieval improves performance and avoids over-fitting, especially with small retrieval datasets.This transfers visual-linguistic knowledge from a related image-text task.
2. Related work
The paper situates natural language object retrieval within visual-linguistic tasks that jointly model text and image content. It builds on recurrent image-captioning and retrieval architectures while adding spatial configuration and global context for region-level localization.
- Related visual-linguistic tasks: Natural language object retrieval, grounding, image captioning, and image retrieval are presented as related tasks for jointly modeling text sequences and image content.This shared structure motivates transferring knowledge across domains.
- Natural language object retrieval: Prior retrieval work used bag-of-words representations, ImageNet-derived embeddings, or text-based image-search methods to match queries with candidate regions.These approaches localize objects by comparing query text with region representations.
- Grounding objects from descriptions: Grounding methods align sentence fragments or text snippets with image regions using pretrained detectors, dependency trees, bidirectional RNNs, or canonical correlation analysis.The cited approaches address localization from image descriptions through region-text alignment.
- Image captioning: SCRC uses LRCN as a base recurrent architecture and incorporates spatial configurations and global context into the recurrent model.LRCN uses a two-layer LSTM with embedded word sequences and image features as inputs.
- Image retrieval: Text-based image retrieval learns ranking functions with recurrent networks, metric learning, correlation analysis, and related methods.The paper distinguishes region-level object retrieval from selecting an entire image.
3. Our model
SCRC retrieves objects by scoring candidate boxes with query text, local appearance, spatial configuration, and global scene context. It transfers image-captioning knowledge through pretraining before end-to-end adaptation to object retrieval.
- 3.1. Spatial Context Recurrent ConvNet: The model combines language and visual information with three LSTM units, local and global CNNs, word embeddings, and a word prediction layer.The language LSTM processes the query sequence, while local and global recurrent units incorporate visual features.
- 3.1. Spatial Context Recurrent ConvNet: SCRC scores candidate bounding boxes using local image descriptors, spatial configurations, and global contextual features.At test time, the highest-scoring candidate boxes are retrieved.
- 3.1. Spatial Context Recurrent ConvNet: The spatial configuration represents each candidate box with eight coordinates and dimensions normalized to the centered image range −1 to 1.The representation contains xmin, ymin, xmax, ymax, xcenter, ycenter, wbox, and hbox.
- 3.1. Spatial Context Recurrent ConvNet: The word prediction layer estimates the next-word distribution conditioned on the local region, whole image, spatial configuration, and preceding query words.Softmax converts the prediction vector into a probability distribution.
- 3.2. Knowledge transfer from image captioning: Pretraining sets Wlocal = 0, making SCRC equivalent to LRCN image captioning and initializing language, prediction, and recurrent parameters from captioning data.The pretrained parameters are transferred and then fine-tuned for natural language object retrieval.
- 3.3. Training for object retrieval: Object retrieval training maximizes the probability of annotated descriptions conditioned on cropped object regions, whole images, and spatial configurations.Ground-truth image-box-description tuples are constructed, and the complete network is fine-tuned end-to-end with SGD.
4. Experiments
Experiments evaluate SCRC across ReferIt, Kitchen, and Flickr30K Entities under annotated-region, proposal-based, and image-level retrieval settings. SCRC achieves the strongest reported results, while context, spatial information, and image-captioning pretraining contribute to performance.
- ReferIt results: The full SCRC model achieves the highest top-1 precision on annotated ReferIt regions and the highest recall on 100 EdgeBox proposals.The proposal setting reports R@1, R@10, and Oracle; the full model has the highest recall in Table 2.
- ReferIt results: Adding spatial configuration, scene-level context, and image-captioning pretraining improves ReferIt performance, with spatial configuration producing the largest reported boost.The comparison isolates these additions in Table 1, while Table 2 confirms the full model’s strongest proposal-based recall.
- Baseline comparisons: SCRC significantly outperforms the bag-of-words CAFFE-7K and directly applied LRCN baselines because it retains spatial configuration and global context while remaining end-to-end trainable.CAFFE-7K loses information through category-name projection, whereas LRCN lacks spatial and global-context encoding for object retrieval.
- Kitchen results: On Kitchen, image-captioning pretraining followed by retrieval fine-tuning gives the best performance in both same-dataset and ImageNET-7K distractor scenarios.The full SCRC model cannot be tested there because Kitchen lacks spatial-configuration and scene-level-context information.
5. Conclusion
SCRC combines local descriptors, spatial configurations, and global scene context in an end-to-end scoring function for natural language object retrieval. The paper also demonstrates knowledge transfer from image captioning and illustrates retrieval across varied objects, regions, and descriptions.
- SCRC scores candidate boxes using local image descriptors, spatial configurations, and global scene-level context.
- The recurrent scoring function is end-to-end trainable and significantly outperforms baseline methods.
- Pretraining and adaptation transfer knowledge from image captioning to natural language object retrieval.
- Examples cover queries for people, object parts, regions, and scene areas, including “man squatting,” “bike wheels,” “river,” and “grass upper right.”
- The figures show correct and failed retrievals, multiple-object scenes, “stuff” regions, and generated descriptions with highlighted bounding boxes.Failure cases include ambiguity and incorrect dataset annotations.