Zero-shot Composed Text-Image Retrieval

• CIR retrieves images from a reference image and textual information describing desired changes, capturing richer user-intention semantics than image-to-image or text-to-image retrieval.Source: §1 · pp. 1–2
• Supervised CIR requires reference-caption-target triplets that are expensive to construct manually, so practical training datasets tend to be limited in scale.Source: §1 · p. 2
• The pipeline automatically builds CIR training datasets from Internet image-caption data by revising captions and retrieving targets with similar captions.Template-based and large-language-model caption revisions produce two datasets.Source: §1 · p. 2
• TransAgg uses transformer-based adaptive aggregation to fuse modalities, and can be applied to downstream benchmarks without finetuning after training on the constructed datasets.Source: §1 · p. 2
• On CIRR and FashionIQ, the zero-shot model performs on par with or significantly above existing SOTA models, and is sometimes comparable to fully supervised ones.Source: §1 · p. 2

• Standard image retrieval uses either dual towers with cosine similarity or multimodal encoders to score image-image or text-image pairs, but these methods cannot exploit complementary modalities for fine-grained queries.Source: §2 · p. 2
• CIR fusion methods include TIRG’s residual and gating modules, CIRPLANT’s pretrained multimodal transformers [18], and CLIP4CIR’s two-stage CLIP training.Source: §2 · pp. 2–3
• Zero-shot CIR approaches include Pic2Word’s image-token mapping [25], CompoDiff’s diffusion training with synthesized triplets [9], and CASE’s semi-automatic triplet construction [15].The paper contrasts these with its fully automated retrieval from a large corpus of real images, without human intervention.Source: §2 · p. 3

3.1 Problem Scenario

• A composed query combines a reference image and relative caption to rank retrieval-set images by cosine similarity, with positives defined by conditional edits to the reference.Training samples are reference-image, relative-caption, and target-image triplets.Source: §3.1 · p. 3

3.2.1 Architecture

• TransAgg uses pretrained visual and text encoders, a Transformer for cross-modal interaction, and adaptive aggregation to combine the resulting features.The experiments primarily use pretrained BLIP or CLIP encoders.Source: §3.2.1 · p. 4
• Adaptive aggregation transforms global visual and textual features into fusion features, then projects them to weights for combining visual, fusion, and textual representations.The fusion features are formed from the image global patch and text global token through an MLP.Source: §3.2.1 · p. 4

3.2.2 The Training Objective

• Training uses batch-based classification loss to bring each query close to its positive target and separate it from negative instances.The temperature is τ = 0.01, and similarity is computed with cosine similarity.Source: §3.2.2 · p. 5

3.3 Dataset Construction

• The dataset pipeline starts with Laion-COCO image-caption pairs, edits captions, and retrieves target images to form CIR training triplets.The caption-editing alternatives are sentence templates and large language models.Source: §3.3 · p. 5

3.3.1 Generating Relative Caption

• Template generation applies predefined rules for eight semantic operations, including replacing a noun phrase with a similar alternative for compare-and-change edits.The example replacement uses noun-phrase similarity scores from 0.5 to 0.7.Source: §3.3.1 · p. 6
• The LLM route prompts ChatGPT (gpt-3.5-turbo) to generate both a relative instruction and an edited target-image caption from the reference caption.The prompt asks for simple edits across semantic aspects and cautions against adding imaginary content.Source: §3.3.1 · p. 6

3.3.2 Target Image Retrieval

• Target-image retrieval embeds edited captions and Laion-COCO captions with a Sentence Transformer, then keeps images whose cosine similarity exceeds a threshold.The resulting candidates supply target images for reference-image, relative-caption, and target-image triplets.Source: §3.3.2 · p. 6

4.1 Experimental Setups

• The zero-shot evaluation uses about 16K triplets per constructed dataset, tests on CIRR and FashionIQ, and reports Recall@K plus CIRR RecallSubset@K.CIRR includes general search over the full validation corpus and fine-grained search among six similar images; FashionIQ covers Dress, Shirt, and Toptee.Source: §4.1 · p. 7

4.2 Ablation Study

• BLIP encoders yield the best performance in the backbone comparison, and fine-tuning more parameters improves results in most cases.The authors use BLIP encoders in subsequent experiments.Source: §4.2 · p. 7, Table 1 · p. 8
• Ablations favor template-based editing over LLM-based construction, show greater impact from adaptive aggregation than transformer fusion, and favor fine-tuning both encoders.Table 2 compares FashionIQ variants with fusion removed or adaptive aggregation replaced by static aggregation.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 Comparison with State-of-the-art

• On CIRR, TransAgg achieves state-of-the-art results on all metrics except Recall@50; on FashionIQ, it ranks among the top two, competitive with CompoDiff using 16k/32k rather than over 18M triplets.Table 3 compares CIRR test and FashionIQ validation results.Source: §4.3 · pp. 8–9, Table 3 · p. 9

4.4 Failure Cases of Dataset Construction

• Sentence-transformer similarity may miss crucial sentence information and retrieve the wrong target; automatically generated Laion-COCO captions can also be inaccurate.The paper illustrates retrieval failures in the first two rows of Figure 3.Source: §4.4 · p. 9

4.5 Qualitative Results for CIR

• Qualitative examples show retrieval without downstream fine-tuning, including an edit that preserves an animal’s category while adding blue sky.Figure 4 marks the ground-truth image with a red box among the top five retrieved images.Source: §4.5 · p. 9

5 Conclusion

• The method performs on par with or significantly above existing SOTA models on two public benchmarks, with some zero-shot results comparable to fully supervised ones.The conclusion attributes this evaluation to a retrieval-based dataset-construction pipeline and TransAgg.Source: §5 · p. 9
• The proposed pipeline automatically constructs CIR datasets from Internet image-caption data, while TransAgg adaptively aggregates information across modalities.The paper obtains two datasets based on templates and a large language model.Source: §5 · p. 9
• Figure 3 illustrates dataset-construction failures involving semantically mismatched captions, a mistaken watering-can retrieval, and incorrect numerical values.In the first case, edited and target captions have high similarity despite differing in meaning.Source: Figure 3 · p. 10
• Figure 4 presents CIRR retrieval examples with the reference image, relative caption, and top five retrieved images; the ground truth is boxed in red.These examples are described as qualitative results for a model trained on automatically constructed data without downstream fine-tuning.Source: Figure 4 · p. 10

A Appendix

• The appendix first details construction of the Laion-CIR-Template dataset.It then presents more detailed experiment comparisons.Source: A Appendix · p. 13
• It reports results for training on a combined Laion-CIR-Template and Laion-CIR-LLM dataset.Source: A Appendix · p. 13
• It presents failure cases from the dataset-construction pipeline and interpretable heatmaps for analyzing TransAgg reasoning patterns.Source: A Appendix · p. 13

Cardinality

• Cardinality edits replace a number in the reference caption with another number or “a group of.”The relative caption is constructed from predefined templates.Source: §A.1 · p. 13, Table 4 · p. 13

Addition

• Addition selects a caption noun and a different noun with similarity between 0.5 and 0.7, then adds “with {noun}” to the edited caption.Source: §A.1 · p. 13, Table 5 · p. 13

Negation

• Negation removes a selected noun phrase from the reference caption and uses a template to form the relative caption.Source: §A.1 · p. 13, Table 6 · p. 13

Direct Addressing

• Direct addressing selects a target image with description similarity of 0.5 to 0.7 and uses its caption as the relative caption.Source: §A.1 · p. 13

Compare & Change

• Compare-and-change replaces a selected noun phrase with another phrase whose similarity score is between 0.5 and 0.7.A predefined template forms the relative caption, and the replacement creates the edited caption.Source: §A.1 · pp. 13–14, Table 7 · p. 14

Comparative Statement

• Comparative statements replace a selected adjective with its antonym and form a relative caption using the antonym’s comparative form.Source: §A.1 · p. 14

Viewpoint

• Viewpoint edits append “small” or “big” to a selected noun according to the relative caption’s meaning.The relative caption is generated using templates.Source: §A.1 · p. 14, Table 8 · p. 14

Statement with Conjunction

• Conjunction randomly combines two of seven scenarios, joins their relative captions with “and,” and applies both sets of editing rules.Source: §A.1 · p. 14

A.2 Detailed Experimental Results

• The section presents more detailed experimental results.Source: §A.2 · p. 14

A.2.1 Pretrained backbone and finetuning

• The evaluation covers different backbones and fine-tuning types on CIRR.The complete results are presented in Table 9.Source: §A.2.1 · p. 14, Table 9 · p. 14
• The evaluation covers different backbones and fine-tuning types on FashionIQ.The complete results are presented in Table 10.Source: §A.2.1 · p. 14, Table 10 · p. 15
• The section compares generalization across backbone and fine-tuning configurations.The supplied passages identify the comparison dimensions but do not report specific outcomes.Source: §A.2.1 · p. 14, Table 9 · p. 14, Table 10 · p. 15

A.2.2 Traininig on combination of Laion-CIR-Template and Laion-CIR-LLM

• Approximately 32k samples make up Laion-CIR-Combined, and using more data tends to lead to better results.The combined dataset is used to train TransAgg.Source: §A.2.2 · p. 15
• The combined-dataset results are reported on the CIRR test set and FashionIQ validation set.They appear in Tables 11 and 12, respectively.Source: §A.2.2 · p. 15, Table 11 · p. 15, Table 12 · p. 15

A.2.3 Comparison with state-of-the-art

• The comparison evaluates zero-shot composed image retrieval methods on CIRR and FashionIQ.The comparison is shown in Tables 13 and 14, respectively; the supplied passages do not state the outcome.Source: §A.2.3 · p. 15, Table 13 · p. 17, Table 14 · p. 17

A.3 Explainability

• In the examples, attention focuses on the dog's head and on the knees and knee pads as requested by the captions.These are the first-row and second-row examples in Figure 5.Source: §A.3 · p. 15, Figure 5 · p. 16
• Attention heatmaps link a bolded relative-caption token to image patches.Figure 5 places each heatmap alongside the reference image, relative caption, and target image.Source: Figure 5 · p. 16