PaperlayerView exact revision on arXiv

Source-linked AI summary

ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity

Ginger Delmas, Rafael Sampaio de Rezende, Gabriela Csurka, Diane Larlus

arXiv:2203.08101v2cs.CVcs.IR

TL;DR

Image search with free-form text modifiers must combine a rich but ambiguous reference image with text specifying desired changes. ARTEMIS addresses this by using separate text-guided attention modules for explicit matching and implicit similarity, and it consistently achieves state-of-the-art performance across several benchmarks.

  • Problem

    Single-modality queries are limited, motivating retrieval from a reference image plus text that specifies how the image should be modified.

  • Method

    ARTEMIS jointly learns text-guided lightweight attention modules for Explicit Matching and Implicit Similarity, combining their scores for retrieval.

  • Results

    ARTEMIS consistently outperforms the state of the art across several datasets despite its simple architecture and training strategy.

  • Takeaways & Limitations

    The complementary modules support retrieval across datasets where either visual or textual query information is more important.

Abstract

from arXiv · show

An intuitive way to search for images is to use queries composed of an example image and a complementary text. While the first provides rich and implicit context for the search, the latter explicitly calls for new traits, or specifies how some elements of the example image should be changed to retrieve the desired target image. Current approaches typically combine the features of each of the two elements of the query into a single representation, which can then be compared to the ones of the potential target images. Our work aims at shedding new light on the task by looking at it through the prism of two familiar and related frameworks: text-to-image and image-to-image retrieval. Taking inspiration from them, we exploit the specific relation of each query element with the targeted image and derive light-weight attention mechanisms which enable to mediate between the two complementary modalities. We validate our approach on several retrieval benchmarks, querying with images and their associated free-form text modifiers. Our method obtains state-of-the-art results without resorting to side information, multi-level features, heavy pre-training nor large architectures as in previous works.

1 INTRODUCTION

Image search with free-form text modifiers combines a reference image with natural-language instructions, addressing the limitations of single-modality queries. ARTEMIS separates explicit textual matching from text-guided implicit visual similarity and jointly learns lightweight attention mechanisms for both.

  • Single-modality queries are restrictive because text incompletely describes the target, while images provide richer but ambiguous visual context.
  • The task uses a reference image and text modifier to specify how the reference should be changed into a target image.
  • ARTEMIS measures textual compatibility through Explicit Matching and reference-target relevance through text-guided Implicit Similarity.
  • Each module uses a lightweight text-guided attention mechanism, and ARTEMIS jointly learns both modules and their combination.
  • ARTEMIS consistently outperforms the state of the art across several datasets despite its simple architecture and training strategy.

2 RELATED WORK

Prior work commonly represents the multimodal query through shared or fused features, often adding complex mechanisms or richer representations. ARTEMIS instead uses text-guided lightweight attention without joint global query composition, target cross-attention, or multi-level visual features.

  • Visual retrieval typically uses global image representations for efficient search or local descriptors for more costly but robust matching.
  • Cross-modal retrieval commonly maps text and images into a shared embedding space and compares their global representations.
  • Early text-modifier retrieval methods adapted visual question answering techniques to reason across image and text modalities.
  • Most current approaches compose the reference image and text modifier into a single representation compared with candidate image features.
  • ARTEMIS uses lightweight text-guided attention instead of joint global query features, target cross-attention, or multi-level visual representations.

3 PROPOSED METHOD

ARTEMIS treats text-modifier retrieval as the intersection of text-to-image and image-to-image retrieval. It computes complementary text-guided similarity scores and combines them to rank target images.

  • Queries contain a reference image and text modifier, which are used to retrieve relevant target images from a gallery.
  • Two complementary views of the task: The text can identify target properties explicitly while also indicating which reference-image properties should remain shared implicitly.
  • Proposed approach: ARTEMIS uses text-guided attention to match reference and target characteristics while separately matching textual requirements to target properties.
  • Training pipeline: The jointly trained IS and EM scores are summed into one compatibility score, optimized with a batch-based classification loss over minibatch negatives.
  • Implicit Similarity: Implicit Similarity compares reference and target features after text-guided weighting, using cosine similarity between the weighted representations.
  • Explicit Matching: Explicit Matching projects text into visual space and text-guidedly weights target features before computing compatibility.

4 EXPERIMENTS

ARTEMIS is evaluated on three retrieval benchmarks using standard dataset-specific protocols, ablations, comparisons with prior methods, and qualitative analysis. It consistently achieves strong or state-of-the-art performance while remaining effective across datasets, encoders, and query characteristics.

  • Experimental setting: Experiments cover Fashion IQ, Shoes, and CIRR, using human-written textual modifiers and standard Recall@K-based evaluation protocols.Fashion IQ and Shoes focus on fashion retrieval, while CIRR uses open-domain images and additionally reports Recallsubset@K.
  • Experimental setting: ARTEMIS is evaluated with ablations, image-only and text-only baselines, late fusion, independent EM and IS modules, and full-model comparisons.The study examines design choices and compares models using ResNet50 and BiGRU encoders in the ablation setting.
  • Ablation study: The full ARTEMIS model performs best across all three datasets, while its independent modules slightly outperform their corresponding single-modality baselines.The results support complementary contributions from Explicit Matching and Implicit Similarity across datasets with different visual-versus-textual query emphasis.
  • Comparison with the state of the art: 1.1% average improvement over VAL⋆ on the challenge metric is reported for the ResNet50-plus-LSTM setting on Fashion IQ.ARTEMIS remains competitive with commonly used ResNet50 and LSTM backbones, while also reporting smaller-backbone results.
  • Comparison with the state of the art: ARTEMIS achieves state-of-the-art results on Shoes with both LSTM and BiGRU encoders and outperforms methods trained on the same data on all CIRR metrics.On CIRR, the comparison is made against methods using the same training data; a competing model’s stronger setting relies on Oscar pre-training over 6.5 million image-caption pairs.
  • Qualitative results: Grad-CAM examples show IS focusing on visual cues shared by reference and target images, while EM focuses on image parts most related to the text modifier.Examples include collars, hems, soles, heels, colors, fits, side slits, and shoe closures.

5 CONCLUSION

ARTEMIS combines complementary explicit and implicit retrieval modules, each guided by text-based attention, and consistently achieves state-of-the-art performance on several benchmarks.

  • ARTEMIS assesses textual compatibility through Explicit Matching and reference-image relevance through Implicit Similarity.Both modules use the text to guide retrieval differently for each query modality.

APPENDIX

The appendix supplements the main paper with implementation details, an additional dataset experiment, efficiency analysis, and qualitative results.

  • The supplementary document adds implementation details, Fashion200K experiments, efficiency comparisons, and qualitative ARTEMIS results.

A IMPLEMENTATION DETAILS

The implementation preprocesses text and images, uses CNN and recurrent text encoders, and trains the model in staged then end-to-end optimization.

  • Text preprocessing removes nonletters, tokenizes modifiers, and encodes words with 300-dimensional GloVe embeddings.Spelling correction and lemmatization did not improve results.
  • Images are padded to square shape, resized to 256×256, cropped to 224 × 224, and represented using a CNN with GeM pooling.Training uses random horizontal flips and random crops, while inference uses center crops.
  • Training freezes base encoders for 8 epochs, then optimizes the full model end-to-end for 50 epochs.AdamW uses batch size 32, initial learning rate 5×10^-4, decay 0.5 every 10 epochs, and 512-dimensional embeddings.

B EXPERIMENTS ON FASHION200K

Fashion200K provides a large, automatically annotated benchmark with a distinct training setup; ARTEMIS performs competitively, although backbone and auxiliary-data choices affect comparisons.

  • Dataset: Fashion200K contains 201k in-shop fashion images, with 172k training images and 29k test images paired into 31k evaluation queries.Its modifiers use automatically generated “replace att. X by att. Y” templates rather than natural-language annotations.
  • Implementation: Fashion200K training uses a mini-batch of 128 for 100 end-to-end epochs with ResNet18 as the image backbone.This setup follows prior methods for handling the dataset’s large training set.
  • Results: Table 6 reports Fashion200K results with markers distinguishing reimplementations, published results, and models using additional training information.
  • Results: ARTEMIS performs better with an LSTM than a BiGRU on Fashion200K, matching the pattern observed for ResNet18 experiments and the TIRG reimplementation.
  • Results: VAL⋆ performs best using extra captions, while ARTEMIS and CoSMo outperform VAL without that additional data.ARTEMIS achieves the best results in 2 of the 4 evaluation metrics among the compared models described.

C COMPLEXITY AND EFFICIENCY STUDY

The study compares ARTEMIS with related retrieval models using parameter count, multiply-accumulate operations, and latency. ARTEMIS adds little complexity and latency relative to the encoder reference, while remaining lighter than CoSMo and VAL.

  • Complexity metrics: Three metrics assess model complexity: trainable parameters, multiply-accumulate operations, and latency time.Latency is measured over all FashionIQ validation queries, target images, and query-target scores on an NVIDIA T4 GPU.
  • Complexity comparison: ARTEMIS and TIRG add very little complexity compared with the encoder reference.Both add only a few operations and parameters to the base encoders; TIRG uses more parameters because its gating mechanisms process concatenated image and text representations.
  • Latency comparison: ARTEMIS and TIRG add some reasonable extra latency compared with the encoder reference.ARTEMIS re-weights pre-computed target representations using text-guided attention before comparison, whereas TIRG uses a simpler direct cosine-similarity pass.
  • Complexity comparison: VAL has more substantial computational complexity than ARTEMIS and TIRG.The comparison reports a more significant increase in GMACs and parameters, while latency could not be measured directly from the available code.
  • Complexity comparison: CoSMo adds around 25% multiply-accumulation operations and more than doubles the latency relative to ARTEMIS and TIRG.Its composition module is more complex than ARTEMIS’s, despite both methods making a single pass on each target.

D QUALITATIVE RESULTS

Qualitative results show that ARTEMIS reasons over multiple visual concepts and uses the reference image to infer unspecified properties. The examples also expose weaknesses with sparse information, negation, subjective captions, and multiple plausible targets.

  • Qualitative retrieval: Across FashionIQ, Shoes, and CIRR, ARTEMIS reasons over attributes such as color, pattern, style, shape, viewpoint, and object details.The qualitative examples show these capabilities across dress, shirt, toptee, shoe, and general image queries.
  • Qualitative retrieval: The model uses the reference image to fill gaps when the text modifier leaves properties unspecified.Examples include style, cut, length, collar form, sleeve length, sole shape, and animal breed.
  • Limitations: Negated text is a documented weakness: qualitative failures retain unwanted properties such as colorful, loose, sheer, laced, or strapped designs.The authors relate this difficulty to negation appearing in only 3.5% of FashionIQ queries, leaving few training examples.
  • Limitations: Caption quality limits retrieval because annotators sometimes express subjective judgments or omit details needed to distinguish the target.An unspecified required color leads to varied model guesses among candidate images.
  • Limitations: Many candidate images may be acceptable for one query beyond the annotated ground truth, making larger-K recall more lenient toward plausible alternatives.The authors identify this as an inherent task limitation and note that CIRR’s Rs@K addresses possible false negatives through additional annotation.
  • Attention visualisations: The EM module focuses on image regions related to captioned requirements, while IS highlights shared characteristics between reference and target images.Heatmaps associate EM with patterns, colors, and decorations, and IS with structural properties such as collars, sleeves, heels, laces, openings, and soles.
Loading 2203.08101v2…