Source-linked AI summary
Benchmarking Composed Image Retrieval for Applied Earth Observation
Bill Psomas, Dionysis Christopoulos, Thanasis Petropoulos, Nikos Efthymiadis, Ioannis Kakogeorgiou, Ondřej Chum, Yannis Avrithis, Giorgos Tolias, Konstantinos Karantzalos
TL;DR
RSCIR offers flexible image-and-text queries for targeted search in large EO archives, but its transferability to EO imagery and operational relevance remain underexplored. This study benchmarks adapted composition methods across six vision-language backbones on PatternCom and introduces xView2-CIR for scene-preserving, change-centric retrieval. Training-free methods are strong practical baselines, while change-centric retrieval differs substantially from attribute-based retrieval.
Problem
The transferability of composed retrieval methods to EO imagery and their relevance to operational workflows remain underexplored despite the need for targeted search in large satellite archives.
Method
The study benchmarks representative composition methods on PatternCom across six vision-language backbones and introduces xView2-CIR for disaster and damage monitoring.
Results
Training-free composition strategies are highly effective in EO settings, with FreeDom strongest on PatternCom, while xView2-CIR shows that change-centric retrieval differs substantially from attribute-based editing.
Takeaways & Limitations
RSCIR is a complementary interface for controllable archive exploration and evidence gathering, rather than a replacement for pixel-level change detection, damage assessment, or rapid mapping.
Takeaways & Limitations
xView2-CIR is relatively small and imbalanced, especially for rare disaster categories, so results on those categories are indicative rather than definitive.
Abstract
from arXiv · showhide
Remote sensing composed image retrieval (RSCIR) enables search in large satellite image archives using composed queries that combine a reference image with a textual modifier. Although RSCIR offers a flexible interface for expressing targeted retrieval intent, the transferability of modern composition methods to Earth observation (EO) imagery and their relevance to operational EO workflows remain underexplored. We address this gap through a unified benchmark and an application-oriented study. First, we systematically adapt and evaluate representative composed image retrieval methods with six vision-language backbones on PatternCom under a standardized protocol, analyzing their behavior across backbones, composition strategies, and query types. Second, we introduce xView2-CIR, a change-centric dataset for disaster and damage monitoring, where retrieval is conditioned on scene identity and a target post-event state. Our results show that training-free composition methods provide strong and scalable baselines for EO retrieval, while change-centric retrieval presents different challenges from attribute-based retrieval, particularly due to the need to preserve scene identity. Overall, this study establishes a practical benchmark for RSCIR and positions composed retrieval as a complementary tool for remote sensing image retrieval, archive exploration, and change analysis. The dataset and code are available at https://github.com/billpsomas/rscir.
1. Introduction
RSCIR combines a reference image with a natural-language modifier to express targeted retrieval intent in EO archives. This work benchmarks RSCIR methods and introduces a change-centric dataset for operationally relevant retrieval.
- Motivation: Single-modality queries restrict retrieval when users need visually similar scenes with targeted attribute or state changes.Operational examples include lower parking-lot occupancy, changed facility quantity, post-wildfire imagery, and recent construction.
- Motivation: RSCIR addresses this limitation by combining a reference image with a textual modifier while preserving relevant visual context.
- Contributions: The study establishes a unified PatternCom benchmark with EO adaptations of representative composition methods evaluated across six vision-language backbones.
- Contributions: xView2-CIR evaluates change-centric retrieval by requiring a pre-event reference, a textual post-event state, and the corresponding post-event image from the same location.
- Contributions: The study analyzes when composed retrieval is useful and how its behavior differs between attribute-based and change-centric settings.
2. Related Work
Remote sensing image retrieval searches large archives through image or text queries, while composed image retrieval combines both modalities to express more specific intent. Recent vision-language models enable zero-shot composition approaches that avoid task-specific training.
- Remote Sensing Image Retrieval: Remote sensing image retrieval research covers unisource and cross-source settings, with single-label and multi-label formulations.
- Composed Image Retrieval: Composed image retrieval retrieves images that remain visually related to a query image while matching textual modifications.
- Composed Image Retrieval: Traditional composed retrieval relies on labeled query-image, query-text, and target-image triplets, limiting early applications.
- Zero-Shot Composed Retrieval: Zero-shot composed retrieval composes embeddings without task-specific training through textual inversion, memory-based inversion, caption-and-LLM pipelines, or diffusion.
- Vision-Language Models: Vision-language models provide joint image-text embedding spaces that support zero-shot transfer across multiple vision tasks.
3. Task & Methodology
RSCIR ranks database images using a reference image and textual modifier, with relevance defined either by class and target attribute or by scene identity and target state. The methodology combines VLM embeddings with unimodal, fusion, inversion, captioning, diffusion, supervised, and training-free composition methods.
- 3.1. Problem formulation: A composed query q = (y, t) pairs a reference image y with textual modifier t and ranks database images by composed similarity.The reference may carry a semantic class or scene label, while the image and text specify source and target attributes or states.
- 3.1. Problem formulation: The benchmark evaluates same-class target-attribute retrieval and same-scene target-state retrieval as distinct relevance protocols.The second protocol preserves geographic scene identity while satisfying the requested state or change.
- 3.1. Problem formulation: Pre-trained VLM visual and text encoders map images and words into a shared l2-normalized embedding space for scoring.
- 3.2. Baselines: Unimodal baselines score images from either text-image or image-image similarity, while multimodal baselines average or multiply the two scores.Score fusion addresses the composed query by combining signals from both modalities, though scale differences can bias simple averaging.
- 3.3. Methods: The evaluated composition methods span textual inversion, memory-based query construction, caption-and-LLM composition, diffusion, supervised pooling, and training-free feature calibration.
- 3.3. Methods: Pic2Word and SEARLE form textual queries from image embeddings using learned or test-time-optimized pseudo tokens.SEARLE uses per-query optimization with an LLM/GPT-based linguistic regularizer.
- 3.3. Methods: FreeDom retrieves textual anchors nearest to the image embedding, concatenates them with the modifier, and improves robustness through visual query expansion.
- 3.3. Methods: CIReVL uses captioning and LLM rewriting, CompoDiff injects text through diffusion, MagicLens learns supervised multimodal pooling, and BASIC calibrates frozen VLM features.
4. Experiments
Experiments establish xView2-CIR as a scene-preserving change-retrieval benchmark and show that composition behavior differs substantially between attribute modification and post-event state retrieval. Training-free fusion and vocabulary-based methods are strong baselines, but change-centric retrieval is more vulnerable to scene drift.
- Experimental setup: xView2-CIR pairs a geo-registered pre-event reference with a post-event text modifier, requiring retrieval of the same location in the target state.Its relevance criterion rejects images from different locations even when they depict the requested disaster state.
- PatternCom: On PatternCom, image-only retrieval usually beats text-only retrieval, while simple multimodal fusion improves over both unimodal baselines.Text × image is often stronger than text + image but is less stable on some backbones, including SigLIP.
- PatternCom: FreeDom achieves the best average mAP across all six backbones, including 46.61% on SigLIP, with CIReVL and BASIC forming the next tier.WeiCom remains competitive and lightweight, whereas Pic2Word, SEARLE, and CompoDiff are generally less competitive or unstable.
- PatternCom: PatternCom favors methods that preserve class identity while injecting attribute-specific textual cues, making strong textual surrogates and domain-grounded vocabularies beneficial.FreeDom’s qualitative results consistently respect both the query class and target attribute value.
- xView2-CIR: WeiCom is the most robust xView2-CIR method overall, achieving the best TOTAL score for both OpenAI CLIP and SigLIP, while text + image is more reliable than text × image across disaster types.The results suggest calibrated modality fusion is suited to identity-preserving change retrieval.
- xView2-CIR: Qualitative xView2-CIR results show that generic composition often retrieves the correct disaster state in the wrong geographic area, especially for hurricanes and wildfires.Distinctive geometry or landmarks can anchor the location, but effective retrieval must jointly preserve instance identity and apply the state constraint.
- LEVIR-CC: LEVIR-CC examples frequently retrieve plausible construction changes under the same location and target-state criterion, indicating usefulness for attribute-based EO retrieval.The examples cover building and road construction using change-oriented modifiers.
- Ablations: Moderate textual expansion improves average mAP from 28.99% to 35.89% on PatternCom and from 9.59% to 16.84% on xView2-CIR at m=n=7.Larger expansions plateau, while visual expansion harms xView2-CIR, reducing Total mAP from 9.59% at k=1 to 4.51% at k=500.
5. Limitations
The study identifies dataset, evaluation, comparability, and prompt-design boundaries, framing the work as a foundation rather than a final RSCIR solution.
- xView2-CIR is relatively small and imbalanced, especially for rare disaster categories, so those results are indicative rather than definitive.
- The evaluation covers image-level retrieval but not finer-grained localization or dense temporal search.
- Some composition methods are tied to particular backbones or representation spaces, limiting strict cross-backbone comparability.
- Prompt design and auxiliary vocabularies influence performance, particularly in change-centric scenarios, and require more systematic study.
- The authors position the study as a benchmark-and-analysis foundation rather than a final solution to composed retrieval in Earth observation.
6. Conclusion
The paper benchmarks RSCIR on PatternCom and introduces xView2-CIR for scene- and state-conditioned change retrieval. It finds that strong backbones and training-free methods transfer effectively, while identity-preserving change retrieval differs from attribute-based editing and supports RSCIR as a practical complement to existing EO workflows.
- The study establishes a standardized PatternCom benchmark and introduces xView2-CIR for disaster and damage retrieval requiring scene identity and a target post-event state.
- Stronger vision-language backbones transfer directly to RSCIR, and training-free composition strategies can be highly effective in EO settings.
- FreeDom is strongest on PatternCom, while lightweight WeiCom and BASIC remain competitive and attractive for practical deployment.
- Change-centric retrieval differs substantially from attribute-based editing because preserving scene identity changes the retrieval challenge.
- RSCIR complements standard RSIR and change-analysis pipelines by providing controllable, semantically guided access to large EO archives.
7. Data and Code Availability Statement
The study provides its supporting code, data, benchmark construction materials, and reproduction scripts in a public repository.
- The code and data are publicly available in the project repository.
- The repository includes xView2-CIR data and scripts for data preparation, evaluation, and reproducing the reported results.
A.1. Dataset statistics
PatternCom statistics motivate attribute-balanced macro-averaged mAP, while xView2-CIR provides a first change-centric benchmark with one same-location positive per query and limited rare categories.
- PatternCom reports composed queries and positives by attribute type, class, and target value, with variable positive counts motivating attribute-balanced macro-averaged mAP.
- xView2-CIR is a first evaluation benchmark rather than a complete operational disaster-monitoring dataset.
- Small disaster categories motivate reporting both macro-averaged and overall metrics and caution against over-interpreting fine-grained differences on rare categories.
- xView2-CIR reports query counts by disaster type and textual modifier, with each pre-event query matched to exactly one post-event image from the same location.
A.2. Vision–language backbones
The benchmark evaluates six vision–language backbones sharing a ViT-L/14 visual backbone, spanning general-purpose and remote-sensing-adapted CLIP-family models.
- Six evaluated vision–language models all use a ViT-L/14 visual backbone.
- CLIP LAION-2B uses the publicly released laion2b_s32b_b82k OpenCLIP checkpoint trained on 2.3 billion web-collected image–text pairs.
- RemoteCLIP is initialized from OpenAI CLIP and fine-tuned on synthetic-caption image–text pairs derived from annotated remote sensing datasets.
- OpenAI CLIP is the original model trained on 400M web image–text pairs, providing a general-purpose baseline.
- SigLIP uses a sigmoid-based contrastive loss rather than the standard softmax contrastive loss and is trained on WebLI.
- CLIP LAION-RS and SkyCLIP-50 are remote-sensing-adapted models fine-tuned respectively on a 726K-image LAION-RS subset and 2.6M SkyScript-50 image–text pairs.
A.3. Vocabulary construction
The study adapts composed retrieval methods to EO semantics through remote-sensing vocabularies and evaluates training-free, inversion, caption-guided, and diffusion-based composition under varied settings.
- Vocabulary construction: The vocabulary prompt requests short, explicit, self-contained entries across diverse land-use, land-cover, and object-detection themes, generated in batches until at least 2,000 entries.
- Vocabulary construction: Requested themes include biodiversity, sustainability, agriculture, marine environments, cultural sites, recreation, transportation, construction, education, healthcare, and natural ecosystems.
- Adapted methods: BASIC combines calibrated image–image and text–image similarities with semantic projection, query expansion, Harris regularization, and RSText-150, while WeiCom fuses modalities with equal contribution at λ=0.5.
- Adapted methods: FreeDom performs memory-based textual inversion using proxy images and HybridText-23k, whereas CompoDiff, Pic2Word, SEARLE, and CIReVL use diffusion, learned inversion, test-time inversion, and caption-editing pipelines.
- Sensitivity and ablations: Moderate textual expansion and diffusion budgets are effective, while BASIC depends mainly on centering and semantic projection; query expansion can harm xView2-CIR because similar proxies introduce off-location evidence.