PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Language Object Retrieval

Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, Trevor Darrell

arXiv:1511.04164v3cs.CVcs.CL

TL;DR

Natural-language queries can identify objects through categories, attributes, spatial configurations, and interactions, but retrieval requires spatial and scene context beyond text-based image retrieval. SCRC scores candidate boxes using local, spatial, and global cues, and significantly outperforms baseline methods.

  • Problem

    Natural-language object queries can describe categories, attributes, spatial configurations, and interactions, while retrieval also requires spatial and global scene context.

    Source: §1 · pp. 1–2

  • Method

    SCRC uses a recurrent network to score candidate boxes from local image descriptors, spatial configurations, and global scene-level context.

    Source: §5 · p. 8

  • Results

    The recurrent scoring model and incorporation of spatial configuration and global context significantly outperform baseline methods for natural-language object retrieval.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Pretraining on image-captioning data and adapting to retrieval can leverage image-level annotations to alleviate the scarcity of large object-level annotation datasets.

    Source: §5 · p. 8

    The training setup assumes each training image has annotated objects with bounding boxes and natural-language descriptions.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

In this paper, we address the task of natural language object retrieval, to localize a target object within a given image based on a natural language query of the object. Natural language object retrieval differs from text-based image retrieval task as it involves spatial information about objects within the scene and global scene context. To address this issue, we propose a novel Spatial Context Recurrent ConvNet (SCRC) model as scoring function on candidate boxes for object retrieval, integrating spatial configurations and global scene-level contextual information into the network. Our model processes query text, local image descriptors, spatial configurations and global context features through a recurrent network, outputs the probability of the query text conditioned on each candidate box as a score for the box, and can transfer visual-linguistic knowledge from image captioning domain to our task. Experimental results demonstrate that our method effectively utilizes both local and global information, outperforming previous baseline methods significantly on different datasets and scenarios, and can exploit large scale vision and language datasets for knowledge transfer.

1. Introduction

Natural language object retrieval localizes an object from a description that may specify its category, attributes, actions, and spatial relations. The proposed SCRC scores candidate regions using the query, local features, spatial configurations, and global context, and can transfer knowledge from image captioning.

1. Introduction

  • Natural language object retrieval localizes a described object in an image, extending fixed-category detection to queries that can express categories, attributes, spatial relations, and interactions.The introduction gives robotics commands such as asking a robot to pick up a TV remote on a shelf as an application.Source: §1 · pp. 1–2
  • Text-based image retrieval on cropped regions performs poorly because retrieval also requires object spatial configurations and whole-scene context, and shifts from whole images to bounding boxes.For example, identifying a man in a blue jacket sitting on the right in front of a house requires both region-level evidence and scene-level context.Source: §1 · p. 2
  • SCRC scores candidate locations from the text query, local descriptors, spatial configurations, and global scene context, then retrieves the highest-scoring candidate.Its two-layer LSTM processes embedded text and visual features in successive layers; candidate locations can come from object proposal methods.Source: Figure 1 · p. 1, §1 · p. 2
  • The recurrent scoring model is learned end-to-end and significantly outperforms a previous bag-of-words method, while captioning-domain pretraining followed by fine-tuning transfers visual-linguistic knowledge.This pretraining and adaptation improves performance and avoids overfitting, especially when the object-retrieval training dataset is small.Source: §1 · p. 2

2. Related work

The related work situates natural language object retrieval among tasks that jointly model text and images, motivating transfer from image captioning. It also connects the paper to prior work on object retrieval, grounding, captioning, and image retrieval.

Overview

  • The authors pretrain on image-captioning datasets to initialize word embeddings and visual-feature-based word-sequence prediction, then incorporate spatial configurations and global context into retrieval.The paper frames retrieval, grounding, captioning, and image retrieval as related directions in jointly modeling text and image content.Source: §2 · pp. 2–3

Natural language object retrieval

  • Prior object retrieval includes bag-of-words candidate descriptions, text-to-region feature matching, joint text–visual embeddings, and a concurrent recurrent localization model [23].[10] compares category-name word bags generated from candidate regions with the query; other approaches generate visual features from text or learn joint embeddings.Source: §2 · p. 2

Grounding Objects from Image Descriptions

  • Grounding work aligns sentence fragments or referential phrases with image regions using ranking losses [17], bidirectional RNNs [16], CCA [25], structure prediction, and attention [27].[21] also reasons about textual object co-reference for 3D scene parsing.Source: §2 · p. 2

Image Captioning

  • Recent recurrent image-captioning methods [31] [30] [24] [4] include LRCN [4], whose two-layer LSTM takes embedded words and image features at each time step; this work uses LRCN as its base architecture.The retrieval model incorporates spatial configurations and global context into the recurrent model.Source: §2 · pp. 2–3

Image Retrieval

  • Text-based image retrieval ranks images using recurrent networks [24] [4], metric learning, correlation analysis [20], and other methods [19]; LRCN scores retrieval with p(Squery|I).The probability is the query text sequence conditioned on the image, as generated by the image-captioning model.Source: §2 · p. 3

3. Our model

SCRC retrieves described objects by scoring candidate boxes using the query text, local appearance, box geometry, and whole-image context. It transfers image-captioning knowledge through pretraining, then fine-tunes on annotated image–box–description tuples.

3.1. Spatial Context Recurrent ConvNet

  • SCRC scores each candidate box by conditioning the query on its local descriptor, spatial configuration, and whole-image context.The query likelihood is used to rank candidate boxes for retrieval.Source: §3.1 · pp. 3–4

3.2. Knowledge transfer from image captioning

  • Caption-task pretraining restricts Wlocal = 0, maximizing ground-truth caption probability to initialize language components and transfer knowledge to object retrieval.With Wlocal = 0, the model is equivalent to LRCN [4] for captioning; the learned parameters are transferred to retrieval.Source: §3.2 · p. 4

3.3. Training for object retrieval

  • Training uses image–box–description tuples from annotated objects, treating each description as a sequence-valued label and maximizing its conditional probability.The pretrained network is fine-tuned end-to-end with SGD at a smaller learning rate.Source: §3.3 · pp. 4–5

4. Experiments

Experiments evaluate SCRC on ReferIt, Kitchen, and Flickr30K Entities. The reported results favor full SCRC on ReferIt, pretrained SCRC on Kitchen, and SCRC over CCA on Flickr30K Entities.

4.1. Object retrieval evaluation on ReferIt dataset

  • SCRC achieved the highest top-1 precision on annotated ReferIt boxes and highest recall on 100 EdgeBox proposals, significantly outperforming CAFFE-7K and LRCN.For the proposal scenario, Oracle (R@100) is the percentage of at least one of all 100 proposals being correct, reported as an upper bound for object retrieval systems.Source: §4.1 · p. 6
  • On ReferIt, adding spatial configuration and scene-level context improved SCRC, with spatial configuration yielding the largest boost; captioning pretraining also beat direct retrieval training.The authors attribute spatial gains both to explicit relations such as left/right and learned location priors, and attribute transfer gains to visual-linguistic knowledge from image captioning.Source: §4.1 · p. 6
  • For ReferIt queries describing objects through scene context, SCRC retrieved among multiple same-category objects by combining spatial configurations and scene-level context.The example involves distinguishing one of three people using a description based on the environment.Source: §4.1 · p. 6
  • SCRC also retrieved ReferIt stuff regions such as grass, roads, and sky, in addition to object regions.The paper distinguishes stuff, which lacks regular shape, from bounded, well-defined objects.Source: §4.1 · p. 7
  • Although designed for retrieval, SCRC generated object descriptions from an image, bounding box, and spatial configuration using beam search.The described decoding objective is to maximize p(S|Ibox, Iim, xspatial).Source: §4.1 · p. 7

4.2. Object retrieval evaluation on Kitchen dataset

  • On Kitchen, pretrained and fine-tuned SCRC achieved the best top-1 precision in both distractor settings, outperforming CAFFE-7K and LRCN [4].The dataset has 606 images; the authors report significantly higher accuracy for pretrained SCRC than for direct training, and omit spatial and scene-context inputs because the task is image-level.Source: Table 3 · p. 7, §4.2 · pp. 7–8

4.3. Object retrieval evaluation on Flickr30K Entities dataset

  • On Flickr30K Entities, SCRC achieved higher recall than the Canonical Correlation Analysis (CCA) method [25] using the same top-100 EdgeBox proposals.The dataset contains 31,783 images and 275,775 annotated boxes, with training on referential expressions.Source: Table 4 · p. 8, §4.3 · p. 8

5. Conclusion

SCRC performs natural-language object retrieval by scoring candidate boxes with local, spatial, and global context, and benefits from image-captioning knowledge transfer. The work also shows that image-level annotations can help address the shortage of object-level training data.

Conclusion

  • SCRC combines local image descriptors, spatial configurations, and global scene context in an end-to-end recurrent scoring function that significantly outperforms baseline methods.The paper also reports that incorporating spatial configuration and global context significantly improves retrieval performance.Source: §5 · p. 8
  • Pretraining and adaptation transfer image-captioning knowledge to retrieval, while image-level datasets can alleviate the shortage of object-level annotations.The passage notes that image-level annotations are often easier to collect than object-level descriptions.Source: §5 · p. 8
  • Follow-up work reports successful results from encoding the phrase rather than scoring it [27] and predicting image segmentations instead of bounding boxes [14].Source: §5 · p. 8

Related papers

Loading 1511.04164v3…