PaperlayerView exact revision on arXiv

Source-linked AI summary

Modality-Agnostic Attention Fusion for visual search with text feedback

Eric Dodds, Jack Culpepper, Simao Herdade, Yang Zhang, Kofi Boakye

arXiv:2007.00145v1cs.CVcs.CLcs.LG

TL;DR

The paper studies finding a desired image from an image-plus-text description of fine-grained changes. It introduces MAAF, which fuses image and text tokens with attention, and reports strong results across existing and newly introduced benchmarks.

  • Problem

    Image retrieval must handle queries combining an image with natural-language descriptions of fine-grained changes beyond discrete objects or attributes.

    Source: §1 · p. 1

  • Method

    MAAF extracts image and text tokens, stacks them into one sequence for attention, and combines the outputs into a query embedding.

    Source: §1 · p. 1, §2.1 · p. 2

  • Results

    MAAF outperforms the state of the art on Fashion IQ and CSS3D, is competitive on Fashion-200k and MIT States, and beats strong baselines on two new benchmarks without modification.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Natural language can express fine-grained image differences beyond predefined lists of attributes or objects in settings such as catalog search.

    Source: §1 · p. 1

    Fashion IQ is not exhaustively labeled, so appropriate retrievals may not be labeled as correct.

    Source: S4. Retrieval examples on Fashion IQ · p. 13

Abstract

from arXiv · show

Image retrieval with natural language feedback offers the promise of catalog search based on fine-grained visual features that go beyond objects and binary attributes, facilitating real-world applications such as e-commerce. Our Modality-Agnostic Attention Fusion (MAAF) model combines image and text features and outperforms existing approaches on two visual search with modifying phrase datasets, Fashion IQ and CSS, and performs competitively on a dataset with only single-word modifications, Fashion200k. We also introduce two new challenging benchmarks adapted from Birds-to-Words and Spot-the-Diff, which provide new settings with rich language inputs, and we show that our approach without modification outperforms strong baselines. To better understand our model, we conduct detailed ablations on Fashion IQ and provide visualizations of the surprising phenomenon of words avoiding "attending" to the image region they refer to.

1. Introduction

The section frames retrieval as finding fine-grained image changes from an image-plus-text query, then introduces MAAF's shared attention over image and text tokens. It reports results across existing and richer-language settings, alongside component ablations and attention visualizations.

1. Introduction

  • The task uses an image plus text describing how to alter it to find a desired image, with emphasis on fine-grained differences beyond discrete objects or attributes.The section identifies catalog search as an application and natural language as more flexible than predefined attribute or object lists.Source: §1 · p. 1
  • MAAF extracts tokens from text and images, stacks them into one sequence for attention, and combines the outputs into a vector for nearest-neighbor search.The strongest Fashion IQ setup uses an LSTM for text, multiscale convolutional feature maps for images, and average pooling within each token class.Source: §1 · p. 1
  • MAAF surpasses the state of the art on Fashion IQ and CSS3D, is competitive on Fashion-200k and MIT States, and beats strong baselines on two new benchmarks without modification.The new benchmarks are based on Birds-to-Words [10] and Spot-the-Diff [26].Source: §1 · p. 2
  • The paper introduces two challenging benchmarks based on Birds-to-Words [10] and Spot-the-Diff [26], providing richer language settings than datasets with only single-word modifications.MAAF outperforms strong baselines on these benchmarks without modification.Source: §1 · p. 2
  • The authors quantitatively evaluate the utility of retrieval-system components through baseline and ablation experiments.Source: §1 · p. 2
  • Visualizations show words avoiding their referent regions and attending to complementary regions, a behavior the authors attribute to the nature of the task.Source: §1 · p. 2

2. Model

MAAF learns a shared embedding space for ranking catalog images against image-plus-text queries. It extracts image and word tokens, fuses them with attention, and pools the outputs for retrieval.

2. Model

  • MAAF learns a shared embedding space by extracting image and word tokens, then ranking catalog images by similarity to an image-plus-text query.Image tokens come from two ResNet blocks, while the LSTM provides one token per input word.Source: §2 · p. 2

2.1. Concatenated self-attention

  • MAAF concatenates image and text tokens and applies alternating self-attention and fully connected layers, sharing attention projections so tokens can interact across modalities.Using the identity attention-map transformation instead of the usual softmax significantly improved CSS performance.Source: §2.1 · p. 2

2.2. Image-text cross attention

  • The alternative cross-attention design keeps image and text sequences separate while enabling their interactions; the paper compares it with modality-agnostic attention on Fashion IQ.The authors hypothesized that modality-agnostic self-attention would work better.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

2.3. Embeddings

  • MAAF pools tokens separately for each image resolution and for text, averages the resulting three vectors, normalizes them, and compares query and catalog embeddings by cosine similarity.The embedding alternatives include pooling all tokens equally, passing catalog images through attention, and including text tokens in pooling.Source: §2.3 · p. 3

2.4. Loss and training

  • Training uses a batch-based classification loss in which each query is matched to its paired target; early experiments found triplet loss worse by recall@k.The base model uses two attention blocks, eight heads, and 256-dimensional fully connected layers; training runs for 150k iterations except on CSS.Source: §2.4 · p. 3

3. Experiments

The experiments compare MAAF with reported retrieval systems and test design choices on CSS and Fashion IQ. They also examine how attention architecture and text encoding affect performance.

3.1. CSS dataset

  • On CSS, larger configurations overfit, so experiments use block 4 alone and one attention block; replacing softmax with identity significantly improves R1.CSS supports controlled analyses using modification phrases for synthesized images in 3-by-3 grid scenes.Source: §3.1 · p. 4

3.2. Fashion IQ

  • Fashion IQ contains about 60K product images and natural-language difference descriptions across three clothing categories, with minibatches drawn from one category.The paper focuses on single models rather than ensembles.Source: §3.2 · p. 4

3.2.1 Comparisons to reported results and ablations

  • MAAF beats TIRG directly and exceeds other reported single models on Fashion IQ validation, except a model using non-public data; some entries surpass it with ensembles.The authors note that the exception is similar to TIRG, but its result without the extra data is unavailable.Source: §3.2.1 · p. 4

3.2.2 Attention model variations

  • The primary single-stream attention variant performs best among the tested two-stream variations on Fashion IQ.The reported comparison is in Table 4, but the supplied passage gives no scores.Source: §3.2.2 · p. 4

3.2.3 Text Representation

  • A randomly initialized LSTM is the most effective text encoder; bag-of-words is about 27% relatively worse, while a single-block Transformer scores lower.The authors attribute bag-of-words performance to the simplicity of the modifying sentences.Source: §3.2.3 · p. 4
  • Fixed GloVe embeddings generally hurt performance, except for a single linear layer, where they provide a useful prior for the small, fashion-specific vocabulary.The authors relate the broader result to general-vocabulary co-occurrence embeddings mapping the specialized vocabulary to a small part of embedding space.Source: §3.2.3 · p. 5

3.3. Birds-to-Words

This section describes retrieval evaluations on Birds-to-Words, Spot-the-Diff, and Fashion200k. The first two provide richer language settings, while Fashion200k uses single-word modifications.

3.3. Birds-to-Words

  • Birds-to-Words pairs bird images with an average of 4.8 human-written descriptions, each averaging 32.1 tokens; retrieval is evaluated on a test set disjoint from training and validation.The dataset has 3,347 image pairs, and training was limited to 30K batches to avoid overfitting.Source: §3.3 · p. 5

3.4. Spot-the-Diff

  • Spot-the-Diff uses captions describing differences between similar surveillance snapshots, making retrieval challenging; the test set contains 2,107 triples from 1,270 image pairs.The dataset covers 11 viewpoints at different times.Source: §3.4 · p. 5

3.5. Fashion200k

  • Fashion200k contains women’s clothing image pairs differing by one attribute, with that attribute’s word as the modifying caption; the Fashion IQ configuration was reused with essentially no changes.Results are compared with existing methods and a TIRG reproduction, but the supplied passage provides no scores.Source: §3.5 · p. 5

4. Qualitative results and visualizations

The visualizations examine averaged attention maps and modulated images to characterize how MAAF responds to modifying words. They show word-specific patterns on CSS and related effects on Fashion IQ.

CSS attention visualizations

  • CSS position words attend less to the image region they name and more to other regions, based on attention maps averaged across heads and examples.Figure 3 encodes larger weights in white; Figure 4 illustrates the attention-map structure used to derive the maps.Source: §4 · p. 6, Figure 3 · p. 7, Figure 4 · p. 7

Interpreting attention patterns

  • The authors suggest that modification words may avoid their referents because those query-image regions are uninformative about the target, unlike words in image-caption grounding.This explanation is presented as a possible account of the observed behavior.Source: §4 · p. 6

Color-word attention visualizations

  • Color words also systematically ignore the objects of their named colors in CSS visualizations using one minus attention, excluding examples where the object is being added.Figure 5 displays these averaged modulated images with the overall mean subtracted.Source: §4 · p. 7, Figure 5 · p. 7

Fashion IQ attention visualizations

  • On Fashion IQ, some heads highlight named clothing parts; additive changes such as “longer dress” show high weights there, while subtractive “shorter dress” changes show negative weights there.The visualization averages attention maps over the first 200 test images.Source: §4 · p. 7, Figure 6 · p. 7

5. Related Work

Related work spans fine-grained retrieval, attention-based image and language modeling, and joint visuolinguistic representations. The paper distinguishes its use of multiscale positional feature maps from approaches based on detected regions or fused text vectors.

5. Related Work

  • Prior work also combines image and language in captioning, VQA, referring expressions, and other tasks, following advances in semantic visual and contextual text features.Image captioning and VQA are among the early task areas cited [46] [38] [2] [33] [11].Source: §5 · pp. 7–8

5.1. Fine-grained image retrieval

  • Fine-grained retrieval work uses labeled product attributes and matching images, with triplet-loss learning and later studies of sampling and loss variations [22] [40] [20].Fashion datasets include Street-to-Shop, DeepFashion, and DeepFashionv2 [13].Source: §5.1 · p. 8

5.2. Image and text modeling with attention

  • Attention has improved object recognition and language tasks; Transformers [45] model self- and cross-attention, while BERT [9] adds large-corpus self-supervised pretraining.The cited language applications include machine translation and language modeling [4] [45] [9] [7].Source: §5.2 · p. 8

5.3. Joint visuolinguistic representations

  • Joint visuolinguistic systems often use object-detector regions, suited to familiar multi-object scenes; MAAF instead uses multiscale positional maps to capture global and local features within single objects.Related models use detector regions [43] [44] [28] [31] [55] [1] [6] [29] or image patches [12].Source: §5.3 · p. 8
  • Hosseinzadeh and Wang use fixed pixel regions with cross-attention and are weaker where comparison is possible; Chen, Gong, and Bazzani fuse text as one vector before attention.The latter approach performs worse on Fashion IQ but better on single-word Fashion200k; the authors suggest it may suit simpler text.Source: §5.3 · p. 8

S1. Detailed model ablations on Fashion IQ

Fashion IQ ablations examine pooling, catalog-image attention, text-token pooling, and attention-block variants. Image-through-attention performs consistently better, while other reported gains are smaller or comparable to trial variation.

S1. Detailed model ablations on Fashion IQ

  • Resolution-wise pooling supports the hypothesis that balancing 7x7 and 14x14 image tokens helps, but its gain is comparable to inter-trial variation.The model separately average-pools each resolution’s tokens, then averages the resulting embeddings.Source: S1. Detailed model ablations on… · p. 11
  • Image-through-attention models perform around 2 percentage points better consistently, without adding parameters because they reuse the query attention blocks.Without ITA, catalog images are represented directly by ResNet features; with ITA, they use the attention mechanism with a null caption.Source: S1. Detailed model ablations on… · p. 11
  • Including text-token outputs in the final average pooling tends to provide a small benefit.Source: S1. Detailed model ablations on… · p. 11
  • Fashion IQ performance is not very sensitive to attention-layer fully connected width; the paper’s other results use 256 units.Source: S1. Detailed model ablations on… · p. 11, Table S2 · p. 12

S1.1. Cross-attention variations

  • Two-stream attention differs by conditioning text-token weights on contextualized image tokens, normalizing image-image and image-text weights separately, and adding learned parameters.Source: S1.1. Cross-attention variations · p. 12
  • The one-stream design was favored because it allows the same basic input interactions as two-stream attention with fewer parameters.The authors expected multiple one-stream blocks could capture benefits of using image tokens contextualized by self-attention when computing image-text attention weights.Source: S1.1. Cross-attention variations · p. 12

S2. Performance by type of modifier

The Fashion IQ analysis tests performance on subsets defined by modifier keywords and reports that MAAF outperforms TIRG across those subsets.

S2. Performance by type of modifier

S3. Visualization of Attention Weights on Fashion IQ

Fashion IQ visualizations show attention patterns for words describing clothing parts and length changes, alongside a procedure for averaging maps across similar images.

S3. Visualization of Attention Weights on Fashion IQ

S4. Retrieval examples on Fashion IQ

The Fashion IQ retrieval examples show additional MAAF results, while noting that dataset labels are not exhaustive and describing the text preprocessing used.

S4. Retrieval examples on Fashion IQ

S5. MIT States

The MIT States evaluation tests image retrieval with adjective feedback on objects whose nouns are unseen during training. Table S5 compares the results with existing methods.

S5. MIT States

  • The test set uses unseen nouns, requiring systems to apply known adjectives to unfamiliar objects.MIT States contains about 60k images, each labeled with an object/noun and a state/adjective.Source: S5. MIT States · p. 13
  • Queries pair an image with the target’s state/adjective, retrieving a target image with the same object/noun.Source: S5. MIT States · p. 13
  • Table S5 reports MIT States results alongside existing methods, as reported in and [23].Source: S5. MIT States · p. 13, Table S5 · p. 14

Related papers

Loading 2007.00145v1…