PaperlayerView exact revision on arXiv

Source-linked AI summary

Composing Text and Image for Image Retrieval - An Empirical Odyssey

Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, James Hays

arXiv:1812.07119v1cs.CV

TL;DR

The paper studies image retrieval from an image-plus-text query, focusing on how to compose the two modalities into a meaningful feature. It proposes TIRG, a gated residual composition method, and reports improved performance across three retrieval benchmarks and state-of-the-art compositional classification on MIT-States.

  • Problem

    Image retrieval needs a meaningful cross-modal feature composition for queries combining a reference image with text describing desired modifications.

  • Method

    TIRG combines image and text features by using text to modify the image feature through a gated residual connection while preserving the target-image feature space.

  • Results

    The proposed composition method outperforms existing methods on Fashion-200K, MIT-States, and CSS, while achieving state-of-the-art compositional classification on MIT-States.

  • Takeaways & Limitations

    The study systematically evaluates image-text feature composition and shows that the proposed method supports both image retrieval and compositional image classification.

  • Takeaways & Limitations

    The approach uses manually added inductive bias because large paired image-text datasets are difficult to obtain, although this may reduce sample complexity.

Abstract

from arXiv · show

In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image. For example, we may present an image of the Eiffel tower, and ask the system to find images which are visually similar but are modified in small ways, such as being taken at nighttime instead of during the day. To tackle this task, we learn a similarity metric between a target image and a source image plus source text, an embedding and composing function such that target image feature is close to the source image plus text composition feature. We propose a new way to combine image and text using such function that is designed for the retrieval task. We show this outperforms existing approaches on 3 different datasets, namely Fashion-200k, MIT-States and a new synthetic dataset we create based on CLEVR. We also show that our approach can be used to classify input queries, in addition to image retrieval.

1. Introduction

The paper studies image retrieval from an image query plus text describing a desired modification. It compares feature-composition methods and proposes TIRG, reporting improved retrieval and classification results across benchmark datasets.

  • Problem: Image retrieval queries combine a reference image with text describing the desired modification.This setup supports using an already found image as a reference while expressing the difference in text.
  • Problem: The central question is how to represent and compose image and text features so matching target images are retrieved.The paper frames this as learning a meaningful cross-modal feature composition for the query.
  • Contribution: The paper systematically studies feature composition for image retrieval and proposes a new method.It compares several existing approaches because image/text composition for this retrieval setting had not previously been studied, to the authors’ knowledge.
  • Method: TIRG uses a gated residual connection so text modifies the query-image feature while the result remains in the target-image feature space.The method is called Text Image Residual Gating, or TIRG.
  • Evaluation: The study evaluates Fashion-200k, MIT-States, and CSS, a synthetic CLEVR-based dataset for controlled image retrieval experiments.CSS enables controlled experiments using color, shape, and size changes.
  • Results: The proposed feature-combination method outperforms existing methods on all three retrieval benchmarks and achieves state-of-the-art compositional classification on MIT-States.The paper also reports improved results over previous state-of-the-art results on Fashion-200K and MIT-States.

2. Related work

Related work spans image and product retrieval, interactive search, vision-language feature composition, and compositional learning. The paper positions its contribution as benchmarking image-text composition mechanisms specifically for image retrieval.

  • Image retrieval and product search: Image retrieval supports applications including product search, face recognition, and image geolocalization, with cross-modal variants using text, sketches, or other views.This paper considers image-to-image retrieval augmented with modification text.
  • Image retrieval and product search: Prior product-retrieval work incorporates user feedback through relevance, relative, or absolute attributes, including systems that manipulate image attributes.The paper instead incorporates text into the query, which it describes as potentially more flexible than predefined attribute values.
  • Interactive retrieval: Dialog-based interactive image retrieval studies interaction between users and agents, whereas this paper benchmarks image-text composition mechanisms.The task is similar, but the research focus differs.
  • Vision question answering: Vision-language methods combine encoded image and text features using techniques such as parameterized layers, FiLM, and other composition mechanisms.The paper notes that many VQA methods learn a brand-new feature, which can be difficult for image retrieval.
  • Compositional learning: Compositional learning builds complex concepts by combining simpler concepts or attributes and connects to visual-attribute and zero-shot learning.Related work includes composition classifiers and embedding-based composition approaches.

3. Method

The method learns a shared embedding for text-plus-image queries and target images, combining image and text features with TIRG’s gated residual modification. Training contrasts each modified query representation with target and negative-image representations.

  • The framework learns an embedding space where matching text-plus-image queries and target images are close.
  • Reference images are encoded with a ResNet-17 CNN, text with an LSTM, and the resulting features are combined into the query representation.The image feature has 512 channels, while the LSTM hidden state has size 512.
  • The study compares TIRG with image-only, text-only, concatenation, Show and Tell, Attribute as Operator, parameter hashing, Relationship, and FiLM combinations.
  • TIRG modifies image features through gated and residual connections while keeping the result in the target-image feature space.The gating connection helps retain the query-image feature when the text modification is insignificant.
  • The method modifies different network layers by dataset: a fully connected layer for Fashion200k and MIT-States, and the last convolutional feature map for CSS.This choice targets more global and abstract changes in the first datasets and low-level spatial changes in CSS.
  • The training objective pulls modified-query and target-image features together while separating them from non-similar images using a classification loss.The similarity kernel is implemented as either a dot product or negative l2 distance.
  • The manually added TIRG decomposition may reduce sample complexity when large paired image-text datasets are difficult to obtain.
  • Using K = B is more discriminative and faster to fit but more vulnerable to overfitting; the paper uses it for Fashion200k and K = 2 elsewhere.

4. Experiments

Experiments evaluate feature-composition methods on three image-retrieval datasets using repeated recall-at-rank measurements. The implementation uses pretrained ResNet-17 image features, an LSTM text encoder, and a shared training pipeline for comparisons.

  • Experiments use Fashion200k, MIT-States, and the synthetic CSS dataset, with recall at rank k as the main retrieval metric.Recall at rank k measures the percentage of test queries with at least one target or correctly labeled image among the top K results.
  • Each experiment is repeated 5 times, with mean and standard deviation reported for retrieval performance.
  • The implementation uses a pretrained ImageNet ResNet-17 image encoder with output feature size 512 and a randomly initialized LSTM text encoder with hidden size 512.
  • All compared image-text combination methods use the same training pipeline, while TIRG supplies the feature-modification composition module.Training is run for 150k iterations with a starting learning rate of 0.01, and the authors state that code and CSS will be released publicly.

4.1. Fashion200k

Fashion200k evaluates modification-based retrieval on roughly 200k fashion-product images, using one-word description changes as textual query modifications. The paper reports that its methods outperform the prior approach, with TIRG performing best among the tested compositions.

  • Dataset: Fashion200k contains approximately 200k fashion-product images paired with compact attribute-like descriptions.Queries pair products whose descriptions differ by one word, using that word as the modification text.
  • Dataset: Queries use a source product image, a target product image, and the single differing description word as the modification.
  • Results: All evaluated methods outperform the prior approach on Fashion200k.The paper attributes this to modifying the image feature space rather than embedding image and text into the same space.
  • Results: TIRG achieves the best performance among the tested ways of computing the composed feature.

4.2. MIT-States

MIT-States tests retrieval and compositional classification with object nouns and visual states, including unseen objects and unseen state–noun combinations. Retrieval benefits from combining image and text, while TIRG and Relationship are comparable and the proposed method also outperforms prior classification methods.

  • Dataset: MIT-States contains roughly 60k images labeled with object nouns and states, with each noun associated with only about nine adjectives on average.
  • Image retrieval: The retrieval split trains on 80 nouns and tests on unseen objects while using state or adjective text to specify the target modification.
  • Image retrieval: Combining image and text gives better retrieval than either modality alone, although text-only outperforms image-only on this dataset.Similar objects with different states can look drastically different, increasing the importance of modification text.
  • Image retrieval: TIRG and Relationship are comparable and outperform the other composition methods on MIT-States retrieval.
  • Classification with compositionally novel labels: The proposed method outperforms prior methods on classification of unseen state–noun combinations.Its composition represents both adjective and noun as feature vectors and combines them with the proposed mechanism.

4.3. CSS dataset

CSS is a synthetic CLEVR-based benchmark designed for controlled image–text retrieval experiments with attribute and object modifications across 2D and 3D scenes. TIRG outperforms other composition methods, while cross-dimensional retrieval is harder because the image feature spaces differ.

  • Dataset Description: CSS generates 3-by-3 grid scenes whose objects vary in color, shape, and size, rendered as both simple 2D blobs and 3D images.
  • Dataset Description: The dataset creates addition, removal, and attribute-changing modification texts, including multi-word specifications for object properties and positions.
  • Dataset Description: CSS contains 16K training queries and 16K test queries generated by applying random modifications to 1K reference scenes.One reference image can produce multiple targets, and one modification can apply to multiple reference images.
  • Dataset Description: The benchmark supports controlled multi-word queries using either 2D sketch-like images or more realistic 3D images.
  • Results: TIRG outperforms other composition methods for CSS retrieval in both 3D-query and 2D-query settings.The comparison reports R@1 retrieval performance for the two query-image types.
  • Results: Retrieving a 3D target from a 2D query is harder because the corresponding feature spaces are substantially different.The experiments use different feature encoders for the 2D and 3D inputs.
  • Results: Reconstructions from TIRG composition features look visually better and closer to the top retrieved image, but reconstruction quality need not predict retrieval quality.The composition feature is learned to preserve discriminative information and may therefore be lossy.

4.4. Ablation Studies

The ablations examine which feature-modification components, feature locations, and loss settings matter for retrieval performance. Residual and gated connections are important, while convolutional-layer modification mainly benefits spatially localized CSS changes; larger K improves Fashion-200k fitting but can destabilize training elsewhere.

  • Feature modification: Removing either residual or gating features reduces retrieval performance.These ablations can degenerate toward concatenation-based fusion.
  • Feature location: Applying modification to the last fully connected feature remains competitive across all three datasets.Modification of the last convolutional feature map improves performance only on CSS.
  • Feature location: Convolutional-feature modification helps CSS because its changes are more spatially localized than those in Fashion-200k and MIT-States.The comparison links feature-location effectiveness to the spatial extent of dataset modifications.
  • Loss setting: Using K = B fits Fashion-200k better than K = 2, while performance is comparable on the other two datasets but training is less stable.The difference concerns the metric-learning loss, not the feature-composition methods.

5. Conclusion

The paper studies image–text composition for image retrieval, evaluates existing methods, and proposes a new method. The proposed approach improves performance on three benchmark datasets, while future work targets real-world retrieval systems.

  • Conclusion: The work explores composing image and text in the context of image retrieval.
  • Conclusion: The authors experimentally evaluate existing methods and propose a new one.
  • Conclusion: The proposed method improves performance on three benchmark datasets.
  • Conclusion: Future work aims to scale the method to real image retrieval systems in the wild.
Loading 1812.07119v1…