PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models

Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, Stephen Gould

arXiv:2108.04024v1cs.CVcs.CLcs.IR

TL;DR

Composed image retrieval lacks sufficiently complex, open-domain data for studying fine-grained visual-language reasoning, and existing datasets contain false negatives. The paper introduces CIRR and CIRPLANT, which uses a pre-trained vision-and-language transformer to modify image features from text; CIRPLANT outperforms existing methods on open-domain images while matching state-of-the-art accuracy on narrow datasets.

  • Problem

    Existing composed-retrieval methods are evaluated on visually simple, narrow-domain datasets with false negatives, limiting study of fine-grained reasoning in rich image-language contexts.

  • Method

    The paper collects the open-domain CIRR dataset and proposes CIRPLANT, which uses a pre-trained vision-and-language transformer to modify reference-image features conditioned on text for nearest-neighbor retrieval.

  • Results

    CIRPLANT outperforms existing methods on open-domain images while matching state-of-the-art accuracy on existing narrow datasets.

  • Takeaways & Limitations

    CIRR supports research on subtle visual-language reasoning in open-domain composed image retrieval and enables evaluation on fully labeled subsets.

  • Takeaways & Limitations

    CIRPLANT can fail when retrieval requires preserving fine-grained visual cues, such as the same dog breed across reference and target images.

Abstract

from arXiv · show

We extend the task of composed image retrieval, where an input query consists of an image and short textual description of how to modify the image. Existing methods have only been applied to non-complex images within narrow domains, such as fashion products, thereby limiting the scope of study on in-depth visual reasoning in rich image and language contexts. To address this issue, we collect the Compose Image Retrieval on Real-life images (CIRR) dataset, which consists of over 36,000 pairs of crowd-sourced, open-domain images with human-generated modifying text. To extend current methods to the open-domain, we propose CIRPLANT, a transformer based model that leverages rich pre-trained vision-and-language (V&L) knowledge for modifying visual features conditioned on natural language. Retrieval is then done by nearest neighbor lookup on the modified features. We demonstrate that with a relatively simple architecture, CIRPLANT outperforms existing methods on open-domain images, while matching state-of-the-art accuracy on the existing narrow datasets, such as fashion. Together with the release of CIRR, we believe this work will inspire further research on composed image retrieval.

1. Introduction

Composed image retrieval uses an image-language query to identify a target image, requiring models to resolve which visual details to preserve or modify. CIRR addresses limits of prior narrow, weakly labeled datasets with open-domain images, rich annotations, visually similar candidates, and fully labeled subsets.

  • Composed image retrieval finds a corpus image matching a query formed from a reference image and modifying text.The two modalities jointly specify user intent: the image captures scene gist, while language describes finer details.
  • Real-life images require fine-grained visual reasoning to identify relevant details and ignore backgrounds or other irrelevant objects.The challenge includes inferring implicit agreements across visual and language contexts.
  • CIRR is an open-domain dataset of real images with rich, high-quality annotations for important aspects of the reference image and textual description.Its collection is based on open-domain real images from NLVR2.
  • CIRR emphasizes visually similar images and provides fully labeled subsets for studying fine-grained reasoning without the false-negative problem of unlabeled valid matches.Fully labeled evaluation addresses the issue that unlabeled images are treated as negatives.
  • CIRPLANT extends composed image retrieval to open-domain images by leveraging large-scale vision-and-language pre-training while preserving a metric-learning retrieval pipeline.Queries are jointly embedded and later compared with candidate images rather than exhaustively pairing each query with every candidate during model prediction.

2. Related Work

Prior image retrieval work uses image, text, attribute, sketch, or composed image-language queries, while compositional learning develops joint representations across modalities. Existing composed-retrieval datasets are visually simple and label-limited; this work introduces an open-domain alternative designed for fine-grained reasoning and fewer false negatives.

  • Image retrieval: Image retrieval queries may use images, attributes, natural language, sketches, or an image paired with text describing desired differences.Composed retrieval combines a reference image with short modifying text, extending beyond single-modality queries.
  • Image retrieval: Composed image retrieval in this work targets open-domain images and distinguishing visually similar candidates through fine-grained visual-language reasoning.This focus differs from earlier applications centered on fashion products or synthetic objects.
  • Compositional learning: Compositional learning aims to produce joint embeddings that capture salient visual and textual information and their interactions.For composed retrieval, prior work includes residual gating to control image-feature variation using text.
  • Vision-and-language pre-training: Vision-and-language pre-training uses transformer models trained on large-scale image-text triplets to provide representations for downstream tasks.The approach builds on representations learned from larger image-text corpora than task-specific datasets typically provide.
  • Datasets for composed image retrieval: Existing composed-retrieval datasets are often repurposed, visually simple, and paired with automatically generated descriptions based on labels.Examples include synthetic geometric shapes in CSS and attribute-tagged fashion images in Fashion200k.
  • Datasets for composed image retrieval: The proposed dataset uses carefully collected, visually and semantically related open-domain pairs, mitigates false negatives, and includes auxiliary annotations for query ambiguities.Its collection strategy reduces the cost of exhaustively labeling every candidate against every query.

3. The Proposed Model

The model maps a reference image and modifying text into an embedding space shared with target images, using a vision-and-language transformer as the composition module. It uses global ResNet image features, an image-token output representation, and soft triplet-based metric learning for retrieval.

  • Vision-and-language pre-training: Vision-and-language transformers aggregate dependencies among text and image tokens through stacked multi-head self-attention layers.The final token representations can be adapted for downstream tasks.
  • Task formulation: The task maps a reference image-text pair into the same embedding space as its target image, modifying the reference representation according to the text.Retrieval then compares the composed representation with target-image embeddings.
  • Composition module: CIRPLANT uses OSCAR, a vision-and-language pre-trained transformer, as the image-text composition module.The model leverages a pre-trained V&L representation for composed retrieval.
  • Image representation: Images are represented by a single normalized feature from an ImageNet-pre-trained ResNet and a learned fully connected layer rather than regional features.The same representation is used for candidate target images; the authors hypothesize global features simplify text-conditioned reference modification.
  • Output representation: The composed image-text feature is taken from the image-feature token instead of the final [CLS] token.This design uses residual connections between input and output tokens to retain the reference-image information while composing it with text.
  • Metric learning: Soft triplet-based metric learning pulls the composed feature toward the positive target and pushes it away from sampled negative pairs using ℓ2-norm distance.Training averages the loss over randomly sampled negatives.

4. The CIRR Dataset

CIRR addresses the visual complexity and false-negative problems in existing composed image-retrieval datasets through carefully constructed, fully labeled image subsets and targeted annotations. Its open-domain collection emphasizes visually similar images and diverse, compositional modifying text.

  • Dataset motivation: Existing datasets lack visual complexity and extensive candidate labeling, producing many false negatives and limiting subtle visual-textual reasoning.All non-reference, non-target corpus images are treated as negatives despite potentially being valid matches.
  • Dataset overview: CIRR contains over 36,000 annotated query-target pairs designed to distinguish targets from sets of similar images.The modifying text is collected specifically to address false negatives and create challenging examples requiring visual and textual cues.
  • Image source: The dataset uses real-world images from NLVR2, selected for reasonable complexity, similar-image collections, and manually screened content quality.NLVR2 provides collections organized around 1,000 synsets, including examples such as acorn and seawall.
  • Image subset construction: CIRR constructs six-image subsets with semantic and visual similarity, filters near-identical images, and selects diverse candidates using ResNet152 cosine similarity.Images with cosine similarity κi ≥0.94 to the initial image are removed, while candidates within 0.002 of the last addition are skipped.
  • Pairing and annotation: Within each subset, nine image pairs support consecutive modifications and multiple outcomes from the same reference image.Crowd workers write target-specific modification sentences while viewing the remaining subset images to prevent false negatives.
  • Annotation analysis: CIRR annotations are more diverse and longer than Fashion-IQ annotations, with over 40% classified as compositional.The collection uses instructions that discourage subjective descriptions, text mentions, plain comparisons, and target-only descriptions.

5. Experiments

Experiments evaluate CIRPLANT and existing methods on open-domain CIRR and narrow-domain Fashion-IQ, using retrieval metrics designed to expose both corpus-level and fine-grained performance. Results show that CIRR challenges existing methods, while CIRPLANT benefits from transformer composition and vision-and-language pre-training.

  • Datasets and compared methods: Experiments compare baselines and state-of-the-art methods on open-domain CIRR and fashion-product Fashion-IQ.CIRR evaluates open-domain composed image retrieval, while Fashion-IQ contains human-annotated Dress, Shirt, and Toptee images.
  • Dataset analysis: CIRR annotations cover diverse semantic modifications, including addition, negation, comparison, spatial relations, background, and viewpoint changes.The reported coverage includes 15.2% addition, 11.9% negation, 31.7% compare-and-change, 61.4% spatial relations and background, and 12.7% viewpoint.
  • Evaluation: RecallSubset evaluates whether the target ranks within the top-K images sampled from the reference image’s image set, reducing corpus-level false-negative effects.The metric is defined over S\{IR}, where S is the shared image set containing reference and target images.
  • Results on CIRR: CIRR’s relatively low retrieval performance indicates that existing methods developed on narrow-domain datasets struggle with the open-domain benchmark.The RecallSubset results are close to theoretical random guess for state-of-the-art models, indicating difficulty with fine-grained modifications between similar images.
  • Results on CIRR: CIRPLANT without initialization surpasses other state-of-the-art methods on CIRR RecallSubset, while OSCAR initialization produces a significant further advantage.The authors associate the stronger RecallSubset performance with transformer-based word-token attention and the OSCAR result with transferred V&L knowledge.
  • Results on Fashion-IQ: On Fashion-IQ, CIRPLANT with OSCAR initialization outperforms most methods, including generic multimodal learning methods and TIRG.This result supports the reported benefit of combining a transformer architecture with V&L pre-training beyond CIRR.

6. Conclusion

The work expands composed image retrieval to complex, open-domain images through the CIRR dataset and introduces CIRPLANT, a transformer-based model leveraging pretrained vision-and-language knowledge.

  • CIRR addresses existing dataset shortcomings by emphasizing visually similar open-domain images for subtle visiolinguistic reasoning.
  • CIRR is designed to support future studies of subtle visiolinguistic concepts and iterative retrieval with dialogue.
  • CIRPLANT uses a transformer-based architecture and pretrained V&L knowledge for composed image retrieval.

A. Implementation Details

The experiments evaluate established composed image retrieval methods and CIRPLANT under specified encoder, optimization, and training configurations on CIRR and Fashion-IQ.

  • CIRR experiments use default configurations for state-of-the-art methods, including TIRG, MAAF, and their listed baselines.
  • TIRG baselines use ImageNet-pretrained ResNet18 image encoders, randomly initialized LSTM text encoders, and soft-triplet loss.
  • TIRG and MAAF implementations use hidden size 512 and vanilla stochastic gradient descent optimization.
  • Fashion-IQ evaluation keeps CIRPLANT hyperparameters unchanged and samples each minibatch from a single subtype.

B. Additional Metrics

Additional evaluation details define mAP reporting, auxiliary annotation collection, and the four questions used to clarify ambiguities in image pairs.

  • mAP@K comparisons on CIRR are reported in Table 5, with corresponding Recall comparisons in Table 3.
  • Because each query has one true-positive, Precision@K and Recall@K are equivalent, so P@K is omitted.
  • Auxiliary annotations provide explicit training signals for ambiguities, although the work does not use them.
  • Workers answer questions about preserved object characteristics, irrelevant changes, viewpoint changes, and background or lighting changes.
  • Table 6 reports auxiliary-annotation statistics determined by visual content and the main annotation.

D. Collection Details on CIRR

CIRR is constructed from visually similar image subsets, paired annotations, quality controls, and optional dialogue structures that support retrieval and future conversational studies.

  • Image subsets: Each six-image subset is designed to contain visually similar images with appreciable differences.
  • Image pairs: Nine image pairs are drawn from each subset, including consecutive modifications and multiple targets from one reference image.
  • Image subsets: Subsets are formed by ranking corpus images by cosine similarity to a randomly selected image using ImageNet-pretrained ResNet features.
  • Dialogue structures: Closed-loop dialogues, multiple outcomes, and longer paths across subsets are illustrated as structures enabled by the pair construction.
  • Dialogue structures: 71.1% of subsets have closed-loop dialogue paths, while overlaps and rejected poor-quality pairs create variable or incomplete paths.
  • Annotation collection: Main annotations require sentences leading only to the true target, while auxiliary annotations address ambiguities and poor-quality pairs may be removed.
  • Quality control: Quality control combines worker pre-selection, manual review of approximately 30% of annotations, and automated checks.
  • Human performance: Human retrieval performance is measured by averaging results from three AMT workers for each test-split pair.

E. Additional Analysis on CIRR

CIRR emphasizes open-domain, visually similar image retrieval and supports fine-grained analysis through richer annotations and fully labeled subsets. Its collection procedure avoids false-negatives within each image subset, while examples expose errors from both TIRG and CIRPLANT.

  • Image synsets: CIRR contains 124 NLVR2 synsets, averaging 136.6±73.1 images per synset.The five most common synsets include bookcase, bookshop, dobreman, timber wolf, and pug.
  • Image synsets: CIRR pairs images by visual similarity rather than separating similar synsets, because synsets may not capture fine-grained visual comparisons.The paper gives dobreman and French bulldog as an example of similar concepts that are not distinguished during pairing.
  • Dataset comparison: CIRR is comparable in size to existing composed image retrieval datasets while providing richer auxiliary annotations for open-domain images.These annotations support interpretations of implicit human agreements across visual and language modalities.
  • False-negative analysis: CIRR’s collection procedure prevents false-negatives within each image subset, unlike Fashion-IQ’s setting where valid predictions can remain unlabeled.Fully labeled subsets enable evaluation without treating unlabeled valid images as negatives.
  • Error analysis: Auxiliary annotations provide explicit interpretations of retrieval errors and can support fine-grained analysis or future training signals.The paper illustrates negative retrieval results from both TIRG and CIRPLANT on CIRR.

G. Dataset File Description

The dataset files document pair-level structure, collection conventions, annotation guidance, and example visualizations. Supporting materials also describe dataset scale, synset distributions, false-negative examples, human evaluation, and annotation labels.

  • Dataset structure: Dataset notes state that some pair counts are generated on the fly and that pairs and images can overlap between subtypes.The notes also indicate that each pair has two sentences.
  • Dataset comparison: CIRR is comparable in size to existing composed image retrieval datasets while containing richer annotations for open-domain images.Table 7 compares the number of pairs and annotation richness across datasets.
  • Annotation documentation: Annotation documentation records worker-discouraged annotation types and defines soft target labels for identical, indistinguishable, and overly different images.The labels are assigned weights of 1.0, 0.5, and -1.0, respectively.
  • False-negative examples: Figure 10 illustrates Fashion-IQ false-negatives because only one candidate is labeled as the target despite other valid predictions.Each query set combines a reference image with two modification sentences.
  • Collection interface: Figure 12 presents main annotations, four-question auxiliary annotations, and human RecallSubset@1 evaluation on the test split.The main annotation requires unique sentences within each subset.
Loading 2108.04024v1…