Source-linked AI summary
SQUARE: Semantic Query-Augmented Fusion and Efficient Batch Reranking for Training-free Zero-Shot Composed Image Retrieval
Ren-Di Wu, Yu-Yen Lin, Huei-Fang Yang
TL;DR
Training-free zero-shot composed image retrieval must capture user intent from a reference image and textual modifications, but embedding-based approaches can treat modification text as a coarse signal. SQUARE combines MLLM-enriched query embeddings with joint candidate reranking, and achieves state-of-the-art or near-state-of-the-art performance across multiple benchmarks.
Problem
Embedding-based approaches can treat modification text as a coarse signal, underusing its rich semantic structure.
Source: §1 · p. 2Method
SQUARE enriches embedding-based composed queries with MLLM semantics and reranks top candidates through joint multimodal reasoning in a two-stage framework.
Source: §1 · p. 3Results
SQUARE achieves state-of-the-art or near-state-of-the-art performance across multiple benchmarks.
Source: §5 · p. 19Takeaways & Limitations
The paper identifies recent MLLMs, including lightweight variants, as promising rerankers.
Source: §5 · p. 19Embedding-based approaches can treat modification text as a coarse signal, leaving its semantic structure underused.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Composed Image Retrieval (CIR) aims to retrieve target images that preserve the visual content of a reference image while incorporating user-specified textual modifications. Training-free zero-shot CIR (ZS-CIR) approaches, which require no task-specific training or labeled data, are highly desirable, yet accurately capturing user intent remains challenging. In this paper, we present SQUARE, a novel two-stage training-free framework that leverages Multimodal Large Language Models (MLLMs) to enhance ZS-CIR. In the Semantic Query-Augmented Fusion (SQAF) stage, we enrich the query embedding derived from a vision-language model (VLM) such as CLIP with MLLM-generated captions of the target image. These captions provide high-level semantic guidance, enabling the query to better capture the user's intent and improve global retrieval quality. In the Efficient Batch Reranking (EBR) stage, top-ranked candidates are presented as an image grid with visual marks to the MLLM, which performs joint visual-semantic reasoning across all candidates. Our reranking strategy operates in a single pass and yields more accurate rankings. Experiments show that SQUARE, with its simplicity and effectiveness, delivers strong performance on four standard CIR benchmarks. Notably, it maintains high performance even with lightweight pre-trained, demonstrating its potential applicability.
1 Introduction
SQUARE is a training-free, coarse-to-fine framework for zero-shot composed image retrieval. It enriches VLM-based queries with MLLM semantics, then reranks top candidates through joint multimodal reasoning.
1 Introduction
- SQUARE addresses the limitation that embedding-based methods can treat modification text as a coarse signal and underuse its semantic structure.Caption-based retrieval also faces limits from the quality, specificity, and faithfulness of generated descriptions.Source: §1 · p. 2
- SQAF adds MLLM-generated target-image descriptions to embedding-based queries, strengthening query-target alignment and improving global retrieval, especially with lightweight VLMs.The semantic cues also make the composed query more interpretable.Source: §1 · p. 3
- EBR presents labeled, boxed top candidates in one grid so an MLLM can compare them jointly in a single pass for faster, more accurate reranking.This encourages reasoning about inter-image differences rather than assessing each image in isolation.Source: §1 · p. 3
- SQUARE is a training-free, coarse-to-fine framework designed to work without model fine-tuning or annotated triplets.Its modular design allows different VLMs and MLLMs to be selected for different efficiency and effectiveness trade-offs.Source: §1 · p. 3
- Experiments on four CIR benchmarks show strong performance, including effectiveness with lightweight backbone models.The framework is presented as adaptable to differing computational budgets and performance requirements.Source: §1 · p. 3
2 Related Work
Related work traces ZS-CIR from trained mapping networks toward training-free methods that use pretrained vision-language models and multimodal reasoning. SQUARE differs by augmenting an embedding-based query with MLLM-generated target descriptions and jointly reranking candidates in one pass.
2.1 Zero-Shot Composed Image Retrieval
- Textual-inversion approaches train mappings from reference images to pseudo-word tokens; Pic2Word introduces a lightweight mapping network, while SEARLE adds optimization-based token generation and distillation.These tokens are combined with modification text for retrieval.Source: §2.1 · pp. 3–4
- KED enriches pseudo-word tokens with image-caption details, while Context-I2W selects intent-relevant visual cues; their learned mapping networks rely on large image-text collections, motivating training-free alternatives.Training-free ZS-CIR uses pretrained models directly as off-the-shelf tools.Source: §2.1 · p. 4
2.2 Training-Free Zero-shot Composed Image Retrieval
- Training-free methods include Slerp’s interpolation of reference-image and modification-text embeddings, while caption-based approaches compose reference captions with edits or generate multiple interpretations.Reference captions produced independently of modification text can omit edit-critical visual details, and generating diverse captions risks extra computation and redundant or noisy descriptions.Source: §2.2 · p. 4
- SQUARE uses MLLM-generated target descriptions to add semantic information to an embedding-based query rather than using captions directly for text-to-image retrieval.The resulting query representation is described as better aligned with the intended target image.Source: §2.2 · p. 5
2.3 MLLMs for Reranking
- Prior MLLM reranking includes binary checks of individual candidates and staged local verification followed by pairwise comparisons; SQUARE instead jointly compares candidates using visual marks in a single pass.The paper presents this single-pass strategy as enabling faster inference.Source: §2.3 · p. 5
3 Methodology
SQUARE adds MLLM reasoning to training-free composed image retrieval by enriching the query and reranking candidate images. It first builds a global ranking and shortlist, then refines that ordering against the reference image and text modification.
3 Methodology
- SQUARE combines a VLM-based query with an MLLM-generated target caption, then reranks top candidates against the reference image and modification text.The pipeline uses the caption to enrich the composed query before candidate reranking.Source: §3 · p. 5
3.1 SQAF: Semantic Query-Augmented Fusion for Richer Composed Queries
- SQAF has an MLLM infer a target-image description from the reference image and modifier, encodes it with the VLM text encoder, and fuses it with the VLM query.The parameter β controls the influence of the MLLM description in the fused query.Source: §3.1 · pp. 6–7
- SQAF ranks gallery images by similarity to the enriched query and passes the top-K candidates to EBR for finer-grained alignment.The method selects an ordered candidate list from VLM image embeddings and similarity scores.Source: §3.1 · p. 7
3.2 EBR: Efficient Batch Reranking for Finer-grained Alignment
- EBR arranges annotated candidates in a grid and has the MLLM compare them with the reference image and modifier to produce a relevance-based updated ranking.Colored boxes and numeric labels let the MLLM refer to specific candidates; the original inputs make EBR independent of how the shortlist was obtained.Source: §3.2 · pp. 7–8
- If the MLLM returns only some candidate indices, EBR preserves the initial order of the remaining candidates and appends them to complete the ranking.The final ranked image list follows the resulting index sequence.Source: §3.2 · p. 9
4 Experiments
The experiments evaluate SQUARE’s retrieval performance across four CIR benchmarks and examine how its fusion and reranking components vary with model and configuration choices. Ablations and qualitative examples also assess the contributions and limitations of these components, including challenges with spatial and compound-attribute queries.
4.1 Datasets and Setup
- Evaluation spans CIRR (R@K, K=1,5,10), CIRCO (mAP@K, K=5,10,25,50), clothing categories (average R@10/R@50), and four GeneCIS tasks (R@K, K=1,2,3).Source: §4.1 · pp. 9–10
4.2 Main Results
- On CIRR, SQUARE with EBR achieves the highest retrieval accuracy among compared methods; its compact reranking avoids token-intensive chain-of-thought prompting.SQAF without EBR outperforms OSrCIR across nearly all settings and CLIP backbones, except ViT-G/14 on the CIRR subset.Source: Table 1 · p. 10, §4.2 · p. 11
- SQUARE achieves superior CIRCO performance across CLIP backbones, with reranking yielding an average 6.66% boost in mAP@5.With larger backbones, LDRE and SEIZE are competitive among non-MLLM methods.Source: Table 1 · p. 10, §4.2 · p. 11
- On FashionIQ validation, SQUARE without EBR outperforms all other methods on R@10 and R@50 with ViT-B/32 and ViT-L/14 backbones; EBR further boosts R@10.With ViT-G/14, SQUARE without EBR and SQUARE achieve higher R@10 than SEIZE but slightly underperform in R@50. EBR reranks only the top 16 candidate images.Source: §4.2 · p. 11, Table 2 · p. 12
4.3 Ablation Studies
- For SQAF, CLIP ViT-G/14 achieves the highest mAP across evaluated ranks, followed by ViT-L/14 and ViT-B/32; BLIP variants are also competitive.The results indicate that larger, more expressive backbones generally improve retrieval, while the framework can use different backbone architectures.Source: §4.3 · p. 12
4.2 Main Results
- On GeneCIS, SQUARE achieves the highest R@1 with all backbones on most tasks, but underperforms on change object, where irrelevant content can distract the MLLM.Despite this limitation, it remains competitive overall, particularly at the top ranks.Source: §4.2 · p. 11, Table 3 · p. 13
4.3 Ablation Studies
- On CIRCO, excluding MLLM-generated target-image descriptions substantially lowers mAP, while ChatGPT-4.1 achieves the highest mAP across evaluated ranks.The ablation suggests the VLM alone does not fully capture user intent and that stronger MLLMs can improve intent understanding.Source: Table 5 · p. 13, §4.3 · p. 13
- In the reported reranker comparison, ChatGPT-4.1 consistently outperforms the others; ChatGPT-4.1-mini’s average mAP is less than 1% lower, while ChatGPT-4o yields unsatisfactory results.The authors suggest ChatGPT-4o’s results may be due to its optimization for speed and efficiency.Source: §4.3 · p. 14
- The 3×3 grid improves average mAP by 5.35% over w/o EBR, while 4×4 gains 3.66%; 5×5 and 6×6 fall below w/o EBR.The authors nevertheless adopt 4×4 as a practical choice because it refines 16 candidates rather than 9.Source: Table 7 · p. 14, §4.3 · p. 14
- Reranking with the reference image and modification text outperforms relying solely on MLLM-generated captions, which can introduce irrelevant or misleading details.The supplied example describes a cat as beside a bowl rather than in it, conflicting with the intended modification.Source: Table 8 · p. 15, §4.3 · pp. 15–16
4.4 Qualitative Analysis
- SQUARE retrieves targets more effectively than CIReVL, and EBR raises targets that SQUARE without EBR often leaves at lower ranks, especially on challenging CIRCO queries.Without EBR, fine-grained distinctions among similar gallery images remain difficult.Source: Figure 7 · p. 16, §4.4 · pp. 17–18
- Caption-based reranking can mislead: a caption that places a cat beside rather than inside a bowl steers retrieval toward the wrong relationship.Although captions provide fine-grained semantic cues, they may be inaccurate or conflict with the modification text.Source: Figure 8 · p. 17, §4.4 · p. 18
4.3 Ablation Studies
- Combining MLLM descriptions with VLM-based queries performs best: excluding descriptions is inferior, description-only beats VLM-only, and excessive description weighting degrades retrieval.The fusion weight β controls description influence; over-reliance can overshadow VLM intent or introduce noise.Source: §4.3 · p. 17
4.4 Qualitative Analysis
- Failure cases include spatial reasoning about a dog’s head being closer to the camera and compound fashion edits; the method can miss the exact target yet return semantically relevant alternatives.The fashion example requires simultaneous changes to color, neckline style, and neckline height.Source: Figure 9 · p. 18, §4.4 · pp. 18–19
5 Conclusion
SQUARE is a two-stage training-free framework for zero-shot composed image retrieval, combining semantic query enrichment with joint candidate reranking. Experiments report state-of-the-art or near state-of-the-art performance across multiple benchmarks, including substantial gains with smaller VLMs.
5 Conclusion
- SQUARE combines SQAF, which fuses an MLLM-generated semantic caption with a VLM-based query embedding, and EBR, which jointly compares candidates in one pass.The paper describes SQAF as improving global retrieval quality and EBR as producing more accurate, semantically coherent rankings.Source: §5 · p. 19
- SQAF enhances the composed query by integrating an MLLM-generated semantic caption with the VLM-based embedding query.The paper says this improves global retrieval quality.Source: §5 · p. 19
- EBR uses MLLMs to compare all candidate images jointly in a single forward pass, producing more accurate and semantically coherent rankings.The paper also reports that recent MLLMs, including ChatGPT-4.1, can serve as effective rerankers, with lightweight variants delivering strong results.Source: §5 · p. 19
- Experiments show state-of-the-art or near state-of-the-art performance across multiple benchmarks.The paper also reports substantial improvements with smaller VLMs, sometimes surpassing results previously achieved only with larger models.Source: §5 · p. 19
- SQUARE yields substantial improvements with smaller VLMs such as CLIP B/32, sometimes surpassing results previously achievable only with larger models such as CLIP L/14.The paper presents this as evidence of the framework’s potential applicability.Source: §5 · p. 19