PaperlayerView exact revision on arXiv

Source-linked AI summary

RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models

Xun Liang, Honghui Yang, Weihang Pan, Ruisi Zhao, Boyuan Pan, Yao Hu, Wenxiao Wang, Binbin Lin, Deng Cai

arXiv:2609.16847v1cs.CVcs.AIcs.IR

TL;DR

Region-level retrieval models and benchmarks have limited support for matching user-specified image regions, despite applications such as e-commerce search and retrieval-augmented generation. RegRet combines a Region-Aware Encoder with staged training and introduces REGMB, a four-task benchmark. It reports improvements over strong baselines on regional retrieval while maintaining comparable or better global-level retrieval results.

  • Problem

    Existing multimodal retrieval models primarily target global tasks, while region-level contrastive training data and diverse evaluation benchmarks are scarce.

    Source: §1 · p. 2

  • Method

    RegRet uses a Region-Aware Encoder to balance ROI features with background context and a three-stage pipeline of localized captioning and contrastive learning.

    Source: §1 · p. 2

  • Results

    RegRet exceeds average LMM performance by 14.3% on REGMB and 15.0% on existing public benchmarks; REGMB fine-tuning adds 7.4% and 7.8% over zero-shot, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    REGMB supplies 225k region-level contrastive pairs across four multimodal retrieval tasks, and RegRet reports on-par or better performance on the global-level M-BEIR benchmark.

    Source: §1 · p. 2

Abstract

from arXiv · show

Region-level retrieval aims to align user-specified image regions with relevant regions or textual descriptions, playing a crucial role in realworld applications such as e-commerce product search and RAG. Although recent Large Multimodal Models (LMMs) have made significant strides in multimodal retrieval, they primarily focus on global-level tasks and struggle to capture effective region-level representations. To bridge this gap, we present RegRet, an LMM-based Region-level Retrieval framework that enhances the regional representations without compromising overall global retrieval performance. At its core, RegRet integrates a Region-Aware Encoder to capture detailed regional features while balancing them with the global background context. To further enhance the fine-grained understanding and discriminability of representations, we design a multi-stage training pipeline that includes detailed localized captioning and regional contrastive learning tasks. In addition, considering the absence of region-level contrastive training data and the limited diversity of evaluation tasks in current benchmarks, we introduce the REGMB benchmark. It comprises 225k contrastive pairs, covering four multimodal retrieval tasks. Extensive experiments validate the effectiveness of our approach. RegRet outperforms strong baselines in the zero-shot setting. Further training with contrastive learning leads to an average improvement of more than 20\% on both REGMB and public benchmarks, while achieving comparable or better results on global-level retrieval tasks.

1 Introduction

Region-level retrieval matters when a query specifies an image region, but existing multimodal retrieval models and benchmarks have limited support for this setting. RegRet combines region-aware encoding and staged training, while REGMB supplies region-level training and evaluation data.

1 Introduction

  • Existing multimodal retrieval models are not designed for region-level tasks, and current benchmarks offer limited training data and scenario coverage.Region-level information can distinguish a carpet's texture from a room's shared layout, which may otherwise make a negative candidate seem more similar.Source: §1 · p. 2
  • RegRet pairs a Region-Aware Encoder with the context encoder to balance regional features and background context, then trains through three stages.The stages are localized captioning for RAE pretraining, text-only contrastive learning, and regional contrastive learning.Source: §1 · p. 2
  • RegRet exceeds average LMM performance by 14.3% on REGMB and 15.0% on existing public benchmarks; REGMB fine-tuning adds 7.4% and 7.8% over zero-shot, respectively.RegRet also achieves leading region-level performance in zero-shot and fine-tuned settings and on-par or better results on M-BEIR.Source: §1 · pp. 2–3
  • REGMB provides 225k region-level contrastive pairs across four multimodal retrieval tasks for training and evaluation.The benchmark addresses the lack of large-scale region-level contrastive training data and the scarcity of evaluation benchmarks.Source: §1 · p. 2
  • RegRet preserves global-level retrieval capability while improving region-level performance.The paper reports on-par or superior performance to state-of-the-art LMMs on M-BEIR.Source: §1 · pp. 2–3

2 Related Works

Prior work extends LLM embedding methods to multimodal retrieval, while region-level representation learning and retrieval benchmarks remain less developed. The cited work describes varied regional encoding strategies and a need for broader, trainable benchmark coverage.

2 Related Works

  • LMM embedding methods extend text-retrieval approaches to multimodal retrieval, with later work using synthetic data or refined contrastive loss.The passage names E5 and NV-Embed, followed by LamRA, Vlm2Vec, E5-V, MMEMBED, MME5, and RzenEmbed.Source: §2 · p. 3
  • Region-level representation methods align cropped regional features or long captions with fine-grained text, while LMM prompting and localized encoding approaches have retrieval-specific drawbacks.The passage notes possible background misinterpretation, distractor noise, and conflicts between regional and global features.Source: §2 · p. 3
  • Global retrieval benchmarks cover diverse fused-modal tasks, but prior region-level benchmarks are limited in task scope and do not support training.The passage contrasts COCO box-classification reformulation and scenario-specific image-to-image datasets with ILIAS, which supports regional image-image and image-text retrieval.Source: §2 · p. 3
  • RegRet processes the image and ROI separately with CE and RAE, then uses an LLM to integrate their tokens into a retrieval embedding.RAE layers cross-attend to corresponding CE layers; CE tokens can be used alone for global retrieval or jointly with RAE tokens.Source: Figure 2 · p. 4

3 Methods

RegRet formalizes ROI-conditioned multimodal matching and combines a context encoder with a region-aware pathway, then trains it through three stages.

3.1 Task Formulation

  • Retrieval ranks candidates by semantic alignment between ROI-conditioned query and candidate representations; image ROIs may be patches or detector proposals.The query and candidate can contain images, text, or interleaved formats.Source: §3.1 · p. 4

3.2 Model Design

  • The context encoder processes the full image, while the region-aware encoder refines its middle-layer features before the LLM produces retrieval embeddings.The regional tokens aim to retain fine-grained ROI information and necessary background context.Source: §3.2 · p. 5

3.2.1 Regional Visual Prompts

  • Visual marks can leave regions underemphasized, crops can discard useful context, and auxiliary images can introduce distractors from the original image.RegRet instead encodes the crop separately with RAE, which can be used alongside or independently from context-encoder features.Source: §3.2.1 · p. 5

3.2.2 Region-Aware Encoder

  • RAE uses layer-wise self-attention for regional features and cross-attention to gather context from corresponding CE layers.Layer-wise coordination accesses early-layer details such as material and texture; separate projectors and frozen CE parameters let RAE be optimized while retaining global retrieval capability.Source: §3.2.2 · pp. 5–6, Figure 3 · p. 6

3.2.3 LMM For Multimodal Embedding

  • A learnable [EMB] token summarizes the preceding instruction and multimodal input into the embedding used for retrieval.The embedding is taken from the token's last hidden state.Source: §3.2.3 · p. 6

3.3 Training Pipeline

  • Training progresses from localized-caption RAE pretraining, to text-only contrastive learning, then to regional contrastive tuning with global- and region-level pairs.The final stage includes negatives between global and regional samples; the language backbone is optimized with InfoNCE.Source: §3.3 · pp. 6–7

4 REGMB: A Region-Level Multimodal Retrieval Benchmark

REGMB addresses limited region-level training data and evaluation diversity with two retrieval-granularity metatasks spanning four tasks.

Benchmark scope

  • REGMB contains 200k training and 25k test contrastive pairs from six datasets, targeting gaps in region-level training data and cross-modal evaluation tasks.The benchmark subjects data to privacy filtering and manual region-level annotation.Source: §4 · p. 7

Metatask 1: Region-Level Retrieval

  • Its region-level metatask has image-to-text, text-to-image, and image-to-image retrieval, including cross-scene XGoods and XLife pairs.The benchmark uses detailed captions for region-based text tasks and human-annotated social-media pairs for the cross-scene image task.Source: §4 · p. 7

Metatask 2: Context-Level Retrieval

  • The context-level metatask is image-text-to-image retrieval, where matching requires background context, such as distinguishing flowers by the metal vase behind them.VisMin and ImgDiff contribute edited region-level image pairs, with XGoods also sampled as complementary data.Source: §4 · pp. 7–8

5 Experiments

Experiments test RegRet’s regional and global retrieval performance, including its architecture, training data, and prompting choices. Results favor RegRet on regional benchmarks while indicating that it retains global retrieval ability.

5.1 Implementation Details

  • RegRet uses Qwen2-VL 3B and 8B backbones in three training stages, from regional pretraining through mixed image- and region-level fine-tuning.The stages use 800k regional image-text pairs, 780k text pairs, then 1.8M image-level and 200k region-level pairs.Source: §5.1 · p. 8

5.2 Baselines

  • Baselines span LLM-, CLIP-, and fine-grained region-level models; prompting strategies vary by task, and RegRet-8B-zs uses only global-level training.The selected strategies are auxiliary images for I2T and T2I, cropping for I2I, and visual markers for IT2I.Source: §5.2 · pp. 8–9

5.3 Evaluation Benchmarks

  • Regional evaluation uses REGMB and public benchmarks, while global retrieval is evaluated on M-BEIR; ROxford has ROIs only in queries.REGMB reports Recall@5 for most tasks and Recall@1 for vismin and imgdiff.Source: §5.3 · p. 9, Table 2 · p. 10, Table 3 · p. 10

5.4 Analysis

  • 22.8% average improvement on unseen public benchmarks follows REGMB training; zero-shot RegRet-8B-zs outperforms baselines on REGMB and public benchmarks.RegRet-8B also improves by 7.4% on average over its zero-shot version.Source: Table 1 · p. 9, §5.4 · p. 10, Table 2 · p. 10
  • RegRet gains on most M-BEIR splits versus LamRA, but the local/global trade-off remains: decoupled RAE and CE support performance across tasks.The reported gain is particularly notable on InfoSeek, where it increases by 1.2%.Source: §5.4 · pp. 10–11, Table 3 · p. 10

5.5.1 RAE Architecture

  • Separate projectors reduce local/global interference, and layer-wise coordination further improves regional performance by aligning CE and RAE features.The ablations compare shared-projector, encoder-decoder, and coordinated RAE variants under the same training process.Source: §5.5.1 · pp. 11–12, Table 4 · p. 11, §5.5 · p. 11

5.5.2 Training Data.

  • 67.7%→80.9%: training LamRA with REGMB improves its performance, while RegRet also gains regional retrieval accuracy from regional contrastive pairs.The data ablation compares pure text, image-level M-BEIR, and regional REGMB training data.Source: Table 5 · p. 11, §5.5.2 · p. 12

5.5.3 Regional Prompt Strat-

  • Even the strongest tested regional prompting strategy falls short of RegRet; auxiliary images improve over cropping and visual markers.The prompting ablation compares visual marks, cropping, and auxiliary images.Source: §5.5.3 · p. 12, Table 6 · p. 12

6 Case Study

Case studies illustrate how RegRet handles background context and fine-grained visual attributes differently from prompting-based baselines.

6 Case Study

  • RegRet uses background requirements when selecting a candidate, rejecting similar clothing that fails to match the requested café setting.The example contrasts RegRet with LamRA’s auxiliary-image result, which is dominated by the racecar query’s background crowd.Source: §6 · p. 12
  • For ROI-focused retrieval, RegRet can discard background distractors that cause an auxiliary-image baseline to match a racecar query to “crowd.”Source: §6 · p. 12
  • RegRet’s fine-grained understanding captures material, finding cloth jackets where the baseline returns leather, an ability attributed to detailed-caption pretraining.The case study also shows that cropping can remove background context needed to interpret an ROI correctly.Source: §6 · p. 12

7 Conclusion

RegRet is designed for diverse region-level multimodal retrieval and pairs a Region-Aware Encoder with the REGMB benchmark. The paper reports improved retrieval performance and identifies future directions in visual document retrieval and RAG.

7 Conclusion

  • RegRet uses a Region-Aware Encoder to enhance region-level representations for diverse multimodal retrieval tasks.Source: §7 · p. 12
  • REGMB is built to support training and evaluation of regional retrieval models.Source: §7 · p. 13
  • The authors report state-of-the-art performance and propose extending RegRet to visual document retrieval and RAG.Source: §7 · p. 13

Appendices

The appendix discusses why region indicators matter in candidate images and how full-image embeddings can miss the relevant local features.

A Incorporating RoIs into Candidates

  • An explicit indicator such as an RoI guides attention to the relevant area; without it, retrieval may rely on global visual similarity.Source: A Incorporating RoIs into Candidates · p. 17
  • A single full-image embedding can overlook local features and favor dominant visual cues, as in LamRA’s red-vehicle false matches.The true positive is closer in LamRA’s semantic space to “Two cars in front of a building” than to “A red van with white strips”.Source: A Incorporating RoIs into Candidates · p. 17
  • Including RoIs in query and candidate inputs focuses retrieval on the relevant region and can downweight interfering background features.Source: A Incorporating RoIs into Candidates · p. 17

B Inference Time Analysis

RegRet’s inference cost depends on the prompting strategy: it is slower than Qwen2VL-7B with cropping, but faster in an auxiliary-image setting. The training pipeline’s three stages use distinct data, trainable parameters, and batch sizes.

B Inference Time Analysis

  • With visual-marker prompting, RegRet and Qwen2VL-7B have comparable inference times; with cropping, RegRet is slower because it processes both the full image and the RoI.The authors note that RegRet achieves more than 20% performance improvement over Qwen-based baselines using cropping, making the added inference cost acceptable.Source: B Inference Time Analysis · p. 18
  • In auxiliary-image inference, RegRet takes 7.1 versus Qwen2VL-7B’s 9.1 and is 22% faster than the auxiliary-image baseline.The comparison is reported for the I2I task on REGMB.Source: Figure 6 · p. 17, B Inference Time Analysis · p. 18

C.1 Training Parameters

  • Training uses three stages: RAE modules and projector, then the language model on pure-text contrastive pairs, then multimodal contrastive fine-tuning with LoRA.The stages use batch sizes 64, 1104, and 1104, respectively, and finish within 20 hours on 24 NVIDIA H20 GPUs.Source: §C.1 · p. 18

C.2 Training Strategy Ablations

The Stage-III ablation compares crop, concatenated, random, and mixed visual prompting on REGMB. Mixed prompting performs best by balancing context for one metatask against fine-grained localization for the other.

C.2 Training Strategy Ablations

  • For REGMB tasks 1–3, training supplies only RAE tokens; task 4 concatenates CE and RAE tokens to support contextual understanding without injecting excessive background into RAE.This mixed prompting design is intended to preserve fine-grained representation quality.Source: §C.2 · pp. 18–19
  • Mixed visual prompting achieves the best performance across the two metatasks in the Stage-III REGMB ablation.The alternatives are crop-only RAE tokens, concatenated CE and RAE tokens, and random selection between those strategies.Source: Table 7 · p. 18, §C.2 · p. 19
  • Metatask 1 uses only RAE tokens to preserve localized detail, while metatask 2 receives CE tokens for contextual information.The authors state that excessive context may distract from metatask 1’s fine-grained understanding.Source: §C.2 · p. 19

D Public Evaluation Benchmarks

The public evaluation uses three representative retrieval benchmarks with ROI annotations, applying a unified cropping strategy to baseline models for fair comparison. Each benchmark uses a task-specific subset and metric.

D Public Evaluation Benchmarks

  • The evaluation selects ROxford, DeepFashion2, and ILIAS because publicly available ROI-annotated datasets are scarce, and applies unified cropping to baselines for fair comparison.ROxford results use its hard subset; DeepFashion2 uses the validation set because the official test candidate pool is unavailable; ILIAS uses its 4.72k core subset instead of the 1M distractor pool.Source: D Public Evaluation Benchmarks · p. 19
  • ROxford covers building scenes and reports mAP on the hard subset, whose ROIs are provided only for query images.The paper omits top-50 because each query has many junk samples.Source: D Public Evaluation Benchmarks · p. 19
  • DeepFashion2 uses consumer-to-in-shop retrieval and average Recall@1, while ILIAS covers I2I and I2T retrieval and reports mAP@50.The paper describes Recall@1 as stricter than mAP for DeepFashion2 and uses the ILIAS core subset to avoid the time cost of indexing 1M candidates.Source: D Public Evaluation Benchmarks · p. 19

E Dataset Details

REGMB combines six sources into region-level query-candidate pairs, with hard negatives to make retrieval challenging. Its datasets span commercial and daily-life imagery, segmentation, captioning, and image editing.

E Dataset Details

  • REGMB aggregates four public datasets—SAM, COYO, ImgDiff, and Vismin—and two proprietary social-media datasets, XGoods and XLife, pairing region-based queries with one to six hard negatives.The sources contribute bounding-box annotations for regional query-candidate pairs.Source: E Dataset Details · p. 19
  • XGoods pairs commercial product images with user photos of the same item; human annotators select positives and the remaining candidates serve as hard negatives.The dataset uses YOLO v8 to detect key objects and SigLIP2 regional features to retrieve candidates; its Task 3 has 69k training and 3.3k evaluation samples, and Task 4 has a 3.4k test split.Source: E Dataset Details · p. 20
  • XLife contains social-media images of daily life and contributes 4.2k query-candidate pairs to the final test set; SAM, COYO, ImgDiff, and Vismin provide region, caption, or edit-based retrieval data.SAM supplies Tasks 1–2 from segmentation regions; COYO supplies region-caption pairs; ImgDiff and Vismin use source images and edit instructions to form image-text-to-image queries.Source: E Dataset Details · p. 20

Related papers

Loading 2609.16847v1…