Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models

• Composed image retrieval uses an image-language pair to find a target; the image conveys a scene’s gist while text specifies finer details, but relevant aspects can be ambiguous.Source: §1 · p. 1
• Real-life scenes contain rich object interactions, but existing datasets focus on simpler fashion [13] or synthetic objects [40], limiting study of the visual reasoning needed to identify what to preserve or change.The model must attend to relevant attributes and infer implicit agreements across visual and language contexts while ignoring backgrounds and irrelevant objects.Source: §1 · p. 1
• CIRR uses open-domain NLVR2 images [35] with rich annotations, emphasizes visually similar candidates, and provides fully labeled subsets to avoid false negatives from unlabeled images.Source: §1 · p. 2
• CIRPLANT uses pretrained vision-and-language knowledge and outperforms current methods on CIRR while reaching state of the art on the existing fashion dataset.It jointly embeds the query, then compares it with candidate-image features using ℓ2 distance rather than exhaustively scoring every query-target pair, while retaining language-conditioned image feature modification [8] [40].Source: §1 · p. 2

• Composed image retrieval combines a reference image with text describing desired differences; prior work explored the setup in fashion, while this work targets open-domain, fine-grained visual distinctions.Other retrieval queries use attributes [13], natural language, or sketches, while image-only CBIR supports product search and face recognition [34] but offers little scope for iterative refinement.Source: §2 · p. 2
• Prior composed-retrieval methods use residual gating [40], region-based features, transformers, or modality-agnostic tokens [8]; this work instead adapts pre-trained V&L knowledge without a task-specific monolithic architecture.Compositional learning also spans VQA, image captioning [1] [2], and video retrieval, where joint embeddings capture salient visual and textual information and their interactions.Source: §2 · p. 2
• Vision-and-language pre-training learns transformer representations from large image-text corpora for task fine-tuning, but composed retrieval requires controlled modification of image features.The approach is motivated by the success of such models in other V&L tasks.Source: §2 · p. 2
• Earlier retrieval datasets were often repurposed with automatically generated descriptions and simple visual or linguistic differences, while later benchmarks added human-written descriptions for footwear, fashion, birds, and scenes.Examples include CSS [40], Fashion200k [13], MIT-States, Shoes, Fashion-IQ, Birds-to-Words [9], and Spot-the-Diff [19].Source: §2 · pp. 2–3
• The new dataset targets open-domain retrieval with related image pairs, mitigates false-negative targets through its collection strategy, and includes auxiliary annotations for query ambiguities.It is not restricted to fashion and associates images only when they are visually and semantically related.Source: §2 · p. 3

3.1. Vision-and-Language Pre-trained Models

• VLP models inspired by BERT [7] concatenate text tokens, image-region features, and optional tokens into variable-length sequences, which transformer layers process with multi-head self-attention.A common downstream-task pattern prepends [CLS] to aggregate modality information for prediction.Source: §3.1 · p. 3

3.2. Adaptation to Composed Image Retrieval

• CIRPLANT uses OSCAR to map a reference image-text pair into an embedding close to the target image, modifying the reference feature conditioned on text.The retrieval task is framed as finding the closest target image in a large corpus.Source: §3.2 · p. 3
• Instead of regional features, the model projects ResNet [14] features from before its final FC-layer through a learned FC-layer and ℓ2-normalization to form one image token.The same representation is used for candidate target images; the paper motivates global features as compatible with VLP models and potentially easier to modify toward the target.Source: §3.2 · p. 3, Figure 2 · p. 4
• Rather than using [CLS], the model uses the image-token output as its joint embedding, aggregating word information through residual connections to modify the reference feature, like REF and VLN-BERT [16].The model uses OSCAR without optional object-label inputs and follows Li et al. for text processing.Source: §3.2 · p. 3
• The model uses the soft triplet-based loss of Vo et al. [40] with ℓ2 distance to bring composed features closer to targets and pull them away from negatives.Training randomly samples a negative for each pair and averages the loss over sampled triplets.Source: §3.2 · p. 4

4. The CIRR Dataset

• Over 36,000 annotated query-target pairs form CIRR, addressing prior datasets’ limited visual complexity and false negatives by distinguishing targets from similar images.Prior datasets treat every image other than the reference and labeled target as a negative, while larger Recall@K values only partly account for false negatives and trade away detailed modification learning.Source: §4 · p. 4

4.1. Data Collection

• CIRR draws real-world images from NLVR2 [35] and forms six-image subsets of visually and semantically similar images using ResNet152 features [14] pre-trained on ImageNet.The selection filters out near-identical images and favors a diverse set of close matches, making retrieval less trivial.Source: §4.1 · pp. 4–5
• Within each subset, nine image pairs support multiple targets per reference and sequential modifications, while workers see the other subset images and write target-specific modifications.This annotation procedure is designed to ensure there are no false negatives within the subset.Source: §4.1 · p. 5
• The data are split randomly into 80% training, 10% validation, and 10% test, and auxiliary annotations are collected but unused in this work.The auxiliary annotations are intended to address ambiguities in implicit human agreement.Source: §4.1 · p. 5

4.2. Analysis on CIRR

• Over 40% of CIRR annotations are compositional, and the sentences are more diverse and longer on average than Fashion-IQ, with references to both images.References to both reference and target image contents may be implicit or explicit.Source: §4.2 · p. 5

5. Experiments

• The evaluation covers CIRR and Fashion-IQ, excluding other discussed datasets because they contain synthetic images or annotations or are domain-wise similar to Fashion-IQ.CIRR is open-domain, while Fashion-IQ covers Dress, Shirt, and Toptee product subtypes with human-generated annotations.Source: §5 · p. 5

5.1. Results

• On CIRR, TIRG performs similarly to the Image-only baseline, MAAF generalizes poorly, and BERT initialization has little effect.The authors suggest that CIRR’s diverse, fine-grained modifications challenge cross-modal composition, and that spatial image tokens may not generalize well to these concepts.Source: §5.1 · pp. 6–8
• On CIRR, OSCAR-initialized CIRPLANT outperforms all other methods by a significant margin, while uninitialized CIRPLANT also achieves higher RecallSubset scores than other methods.The authors attribute the model’s fine-grained retrieval performance to transformer composition of image and word-token features, and its overall CIRR performance to V&L pre-training.Source: Table 3 · p. 7, §5.1 · p. 8
• On Fashion-IQ, OSCAR-initialized CIRPLANT outperforms most methods, including TIRG, but VAL and MAAF perform better.The authors report that OSCAR initialization improves performance and suggest that domain shift may limit their model relative to VAL and MAAF.Source: Table 4 · p. 7, §5.1 · p. 8

5.2. Qualitative Results

• Qualitative CIRR examples show pre-training helps retrieve images involving less-frequent concepts such as pavement, while a failure requires preserving a dog’s breed using subtle cues such as pointy ears.CIRPLANT also captures language cues such as “takes people around,” which implies people in the back of a carriage; the authors credit its word-token attention.Source: Figure 4 · p. 7, §5.2 · p. 8

5.1. Results

• RecallSubset results place state-of-the-art models near random, while Text-only and Random-Image+Text baselines outperform them significantly.RecallSubset ranks targets among images from the same set and is not affected by false negatives; target-specific descriptions can help within that smaller set without identifying the target in the full corpus.Source: §5 · p. 6, §5.1 · p. 8

6. Conclusion

• CIRR emphasizes distinguishing visually similar open-domain images and is designed to support subtle visiolinguistic reasoning and iterative retrieval with dialogue.The dataset is publicly available.Source: §6 · p. 8
• CIRPLANT uses a transformer-based architecture and vision-and-language pre-training to compose image and text features.Source: §6 · p. 8
• Validation on CIRR and an existing fashion dataset demonstrates the design’s generalizability and the effectiveness of vision-and-language pre-training.Source: §6 · p. 8

A. Implementation Details

• CIRR experiments use state-of-the-art methods’ default configurations.This applies to the methods evaluated on the proposed CIRR dataset.Source: A. Implementation Details · p. 11
• TIRG and its baselines use ImageNet-pretrained ResNet18 image encoders and randomly initialized LSTM text encoders.The listed baselines include Random, Image/text-only, Random Image+Text, and Concatenation.Source: A. Implementation Details · p. 11
• On CIRR, TIRG does not benefit from more complex ResNet features or word-embedding initialization, and batch-based classification loss causes serious overfitting.The models are therefore trained with soft-triplet-based loss.Source: A. Implementation Details · p. 11
• MAAF uses a pretrained ResNet50 with an LSTM and is trained with batch-based classification loss.This setup follows Dodds et al. [8].Source: A. Implementation Details · p. 11
• MAAF and TIRG share a codebase, use hidden size 512, and are optimized with vanilla SGD.The optimization setup follows Vo et al. [40].Source: A. Implementation Details · p. 11
• CIRPLANT’s Fashion-IQ setup keeps the CIRR hyperparameters without tuning and samples each training minibatch from one subtype.The single-subtype minibatch sampling follows Dodds et al. [8] because Fashion-IQ’s three subtypes differ greatly.Source: A. Implementation Details · p. 11

B. Additional Metrics

• Precision@K and Recall@K are equal because each query has only one true positive, so P@K is not shown.Source: B. Additional Metrics · p. 11
• mAP@K comparisons among CIRR methods are described as similar to the Recall comparisons in Table 3.Table 5 presents mAP scores, but the supplied passages do not provide its cell values.Source: Table 5 · p. 11, B. Additional Metrics · p. 11

C. Auxiliary Annotations in CIRR

• Auxiliary annotations target ambiguities from implicit human agreements; although unused in this work, they may help future work clarify and interpret them.Source: C. Auxiliary Annotations in CIRR · p. 11
• For each reference-target pair, AMT workers saw the image pair and main modification sentence, then answered each question with a sentence or marked it not applicable.Statistics appear in Table 6, and the collection interface and examples are shown in Fig. 12 and Fig. 11, respectively.Source: C. Auxiliary Annotations in CIRR · p. 11
• Workers were asked which object characteristics persisted across the images and which changed but were irrelevant to the modifying sentence.Source: C. Auxiliary Annotations in CIRR · p. 11
• Workers were also asked whether camera angle, focus, or viewpoint changed, and whether the background or lighting changed.Source: C. Auxiliary Annotations in CIRR · p. 11
• Table 6 reports auxiliary-annotation statistics, with availability for each pair determined by its visual contents and main annotation.Source: Table 6 · p. 11, C. Auxiliary Annotations in CIRR · p. 11

Image subsets

• Each six-image subset is constructed to contain visually similar images with appreciable differences, filtering out images during the process and later avoiding heavy overlap between subsets.The construction procedure is illustrated in Figure 5.Source: D. Collection Details on CIRR · p. 11, Figure 5 · p. 12

Image pairs

• Nine image pairs are drawn from each subset, supporting consecutive modification paths and multiple targets from a shared reference image.Figures 6 and 7 illustrate closed-loop dialogue paths and multiple outcomes from one reference image, respectively.Source: D. Collection Details on CIRR · p. 11, Figure 6 · p. 13, Figure 7 · p. 13
• 71.1% of subsets have closed-loop dialogue paths; path lengths vary because subsets may overlap slightly and workers can reject poor-quality pairs.Overlapping subsets can connect paths across subsets, while rejected pairs leave dialogues incomplete (Figure 8).Source: D. Collection Details on CIRR · p. 13, Figure 8 · p. 13

Annotation collection on AMT

• For main annotations, AMT workers write sentences that lead only to the true target and can flag poor-quality pairs for removal; auxiliary annotations clarify pair ambiguities.The collection interface also elicits human RecallSubset retrieval judgments on the test split.Source: D. Collection Details on CIRR · p. 13

Quality control

• Preselection filters out over 60% of submitted workers, and selected workers’ annotations have over 90% acceptance; about 30% of each selected worker’s annotations are manually reviewed.Remaining annotations are checked automatically for checkbox abuse, irresponsible content, and guideline violations.Source: D. Collection Details on CIRR · pp. 13–14

Human performance

• Human RecallSubset@1 performance on the test split is collected by asking three AMT workers per pair to select the most probable target and averaging their submitted results.The score is listed in Table 3, row 7; no value is given in these passages.Source: D. Collection Details on CIRR · p. 14

E. Additional Analysis on CIRR

• CIRR covers 124 of NLVR2’s 1,000 synsets, with 136.6±73.1 images per synset; Fig. 9 shows their distribution.The five most common synsets are bookcase, bookshop, dobreman, timber wolf, and pug; the passage lists acorn, skunk, orange, ox, broccoli, and padlock among the least common.Source: E. Additional Analysis on CIRR · p. 14
• CIRR’s image-pair construction uses visual similarity rather than distinguishing similar synsets, and synsets may not fully characterize images because annotations emphasize fine-grained visual comparisons.For example, dobreman and French bulldog are not distinguished as separate concepts when forming pairs.Source: E. Additional Analysis on CIRR · p. 14
• CIRR is comparable in size to existing composed-image-retrieval datasets and provides rich auxiliary annotations for open-domain images.Table 7 compares CIRR with existing datasets, but the supplied passage gives no specific table values.Source: E. Additional Analysis on CIRR · p. 14
• Unlike Fashion-IQ, CIRR’s collection procedures ensure no false negatives within each image subset.Fig. 10 demonstrates false negatives in Fashion-IQ, while CIRR’s guarantee is discussed in Sec. D.Source: E. Additional Analysis on CIRR · p. 14

F. Additional Examples of CIRR

• Additional examples show negative retrieval results from both TIRG and CIRPLANT and emphasize CIRR’s challenge of distinguishing visually similar images.The examples appear in Fig. 11.Source: F. Additional Examples of CIRR · p. 14
• CIRR’s auxiliary annotations explicitly interpret errors, including implicit human agreements across visual and language modalities, and may support fine-grained analysis or future training signals.These are suggested uses for composed image retrieval research.Source: F. Additional Examples of CIRR · p. 14

G. Dataset File Description

• Table 9 summarizes information provided for each image pair and presents the data structure used in the files.Further details are available on the project website.Source: G. Dataset File Description · p. 14, Table 9 · p. 20
• Pair counts are generated on-the-fly; each pair has two sentences, subtype pairs and images overlap, and approximately 100,000 low-detection-score images could be removed.The reported total image count includes those images.Source: G. Dataset File Description · p. 15
• CIRR is comparable in pair count to existing composed image retrieval datasets while providing richer annotations for open-domain images.Source: Table 7 · p. 15
• The asterisk is used for cross-referencing image pairs between cap.sample.json and cap.ext.sample.json; when constructing target soft, labels [The same image], [No differences worth mentioning], and [Images that are too different] receive 1.0, 0.5, and -1.0, respectively.Source: G. Dataset File Description · p. 20
• Auxiliary annotation options are described, but test-split annotations are not public; evaluation is instead provided through the project website’s test-split server.Source: G. Dataset File Description · p. 20