PaperlayerView exact revision on arXiv

Source-linked AI summary

RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition

Ziyu Liu, Zeyi Sun, Yuhang Zang, Wei Li, Pan Zhang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, Jiaqi Wang

arXiv:2403.13805v2cs.CVcs.AIcs.LG

TL;DR

CLIP and MLLMs can struggle to distinguish fine-grained categories across extensive vocabularies. RAR retrieves top-k candidates from multimodal memory for MLLMs to rank, and the paper reports improvements across fine-grained, few-shot, and zero-shot recognition benchmarks.

  • Problem

    CLIP and MLLMs face challenges with fine-grained recognition across extensive vocabularies, limiting recognition where many similar categories must be distinguished.

    Source: §I · p. 1

  • Method

    RAR stores multimodal embeddings in external memory, retrieves top-k class candidates, and has MLLMs rank them for final predictions.

    Source: §I · p. 1, B. Multimodal Retriever · p. 3

  • Results

    RAR reports improvements across five fine-grained benchmarks, 11 few-shot datasets, and two zero-shot detection datasets, including an average 6.2% gain on 11 image-classification datasets in the 4-shot setting.

    Source: §I · p. 1

  • Takeaways & Limitations

    RAR is presented as a plug-and-play approach for integrating retrieval and ranking with MLLMs across recognition tasks.

    Source: §I · p. 2, §V · p. 13

Abstract

from arXiv · show

CLIP (Contrastive Language-Image Pre-training) uses contrastive learning from noise image-text pairs to excel at recognizing a wide array of candidates, yet its focus on broad associations hinders the precision in distinguishing subtle differences among fine-grained items. Conversely, Multimodal Large Language Models (MLLMs) excel at classifying fine-grained categories, thanks to their substantial knowledge from pre-training on web-level corpora. However, the performance of MLLMs declines with an increase in category numbers, primarily due to growing complexity and constraints of limited context window size. To synergize the strengths of both approaches and enhance the few-shot/zero-shot recognition abilities for datasets characterized by extensive and fine-grained vocabularies, this paper introduces RAR, a Retrieving And Ranking augmented method for MLLMs. We initially establish a multi-modal retriever based on CLIP to create and store explicit memory for different categories beyond the immediate context window. During inference, RAR retrieves the top-k similar results from the memory and uses MLLMs to rank and make the final predictions. Our proposed approach not only addresses the inherent limitations in fine-grained recognition but also preserves the model's comprehensive knowledge base, significantly boosting accuracy across a range of vision-language recognition tasks. Notably, our approach demonstrates a significant improvement in performance on 5 fine-grained visual recognition benchmarks, 11 few-shot image recognition datasets, and the 2 object detection datasets under the zero-shot recognition setting.

I. INTRODUCTION

CLIP and MLLMs both face difficulties with extensive, fine-grained vocabularies, motivating RAR’s retrieval-and-ranking augmentation. The paper evaluates RAR across fine-grained, few-shot, and zero-shot recognition tasks and positions candidate-based ranking alongside prior MLLM approaches.

I. INTRODUCTION

  • CLIP’s performance wanes on vast vocabularies and fine-grained categories, where ambiguous language descriptions and synonyms complicate distinguishing similar classes.MLLMs also face difficulty with extensive vocabularies and fine-grained categorizations, despite their proficiency when candidate numbers are manageable.Source: §I · p. 1
  • RAR retrieves top-k class names with a multimodal retriever, then uses MLLMs to rank candidates as final predictions.The approach is also described as preserving MLLMs’ existing knowledge while augmenting them with external knowledge.Source: §I · pp. 1–2
  • Across 11 image-classification datasets, RAR improves 4-shot performance by an average of 6.2%, with gains on LVIS and V3Det zero-shot object recognition.The reported gains are 6.4% on LVIS and 1.5% on V3Det; evaluation also covers five fine-grained recognition benchmarks.Source: §I · p. 1
  • The paper presents RAR as plug-and-play across MLLMs and reports that it outperforms baselines across visual-recognition tasks.Its stated evaluation includes 11 classification datasets and two object-detection datasets.Source: §I · p. 2

II. RELATED WORK

  • RAR uses candidate prompting for MLLM screening, whereas Fine-R is described as using prompt design for better descriptions and attributes in fine-grained perception.The paper also states that MLLMs can re-rank top results obtained through vision-language models such as CLIP.Source: §II · p. 2

III. METHODOLOGY

The methodology section moves from background on CLIP, MLLMs, and retrieval augmentation to RAR’s multimodal retriever and downstream retrieving-and-ranking pipeline.

III. METHODOLOGY

  • The methodology first provides background on CLIP, MLLMs, and retrieval augmentation in LLMs.This background is identified as Section III-A.Source: §III · p. 2
  • RAR’s multimodal retriever is presented as the next methodology component.The paper identifies this component as Section III-B.Source: §III · p. 2
  • The methodology then describes applying RAR to downstream tasks through retrieving and ranking.This downstream-task pipeline is identified as Section III-C.Source: §III · p. 2

A. Preliminaries

The preliminaries describe CLIP’s image-to-category matching, MLLMs’ sequence generation, and retrieval augmentation that supplies external-memory context to an LLM.

A. Preliminaries

  • CLIP combines image and text encoders and maps an input image to its most semantically similar category.The category is selected from the full category list using cosine similarity.Source: A. Preliminaries · pp. 2–3
  • RAR’s pipeline retrieves top-k categories from memory and uses MLLMs to refine them through ranking into final predictions.The pipeline caption describes memory containing image or text embeddings.Source: Figure 2 · p. 3
  • An MLLM conditions on text and visual input tokens to generate an output-token sequence.The visual tokens are extracted from the input image via an image encoder.Source: A. Preliminaries · p. 3
  • Retrieval augmentation selects a subset of external-memory documents and feeds them to the LLM alongside the original input.The retrieved context is used with the input to generate the target output.Source: A. Preliminaries · p. 3

B. Multimodal Retriever

The multimodal retriever stores CLIP image and text embeddings in memory for category retrieval, with task-specific memory construction and indexing. For object detection, it adds image preprocessing to support embedding extraction.

B. Multimodal Retriever

  • CLIP image and text encoders create embeddings stored in memory; n-shot memory uses n images per class, while some zero-shot settings store only text embeddings.The memory is built from randomly sampled training data.Source: B. Multimodal Retriever · p. 3
  • For object detection, image embedding extraction adds cropping and resizing, and the described procedure also blurs non-target regions.The procedure crops proposal regions and resizes them; blurring is intended to direct MLLM attention to target objects.Source: Figure 3 · p. 4, B. Multimodal Retriever · p. 4
  • HNSW indexing accelerates approximate nearest-neighbor retrieval over large embedding memories while maintaining competitive precision against exact search.Brute-force search computes similarities against every memory vector and becomes less efficient at millions of embeddings.Source: B. Multimodal Retriever · p. 4

C. Inference with Retrieving And Ranking

RAR retrieves a shortlist from memory and supplies it with the input image to an MLLM, which ranks candidates for the final prediction. The inference setup varies for object detection, and ranking can be supported by prompts, fine-tuning, or in-context examples.

C. Inference with Retrieving And Ranking

  • For classification, the image embedding navigates the memory index to retrieve top-k related categories, which act as a preliminary filter of candidate classes.The visual encoder matches the encoder used to construct the multimodal retriever.Source: C. Inference with Retrieving And Ranking · p. 4
  • For object detection, RAR uses image-to-text rather than image-to-image retrieval because cropped regions are small and CLIP features are limited at low resolution.The retrieved text categories are those with the highest similarity.Source: C. Inference with Retrieving And Ranking · p. 4
  • The MLLM combines retrieved category labels with the image embedding in a ranking prompt to produce the final category prediction.The final prediction is represented as ranking the top-k retrieved embeddings using the ranking prompt.Source: C. Inference with Retrieving And Ranking · p. 4
  • The ranking prompt presents the retrieved top-k categories for the MLLM to order, with Fig. 4 illustrating the prompt using fine-grained Stanford Cars examples.The example caption gives k = 5 as one possible shortlist size.Source: Figure 4 · p. 5, C. Inference with Retrieving And Ranking · p. 5
  • Ranking can be tuned with roughly 30,000 Aircraft-derived examples or guided with an in-context ranking example, which avoids explicit retraining.The fine-tuning examples are constructed from nearest-neighbor image groups and shuffled category labels.Source: C. Inference with Retrieving And Ranking · p. 5

IV. EXPERIMENTS

The experiments cover fine-grained recognition, few-shot image recognition, and zero-shot object recognition, followed by ablations of design choices.

IV. EXPERIMENTS

  • The evaluation spans fine-grained visual recognition, few-shot image recognition, and zero-shot object recognition.The accompanying table describes statistics for datasets in fine-grained, few-shot, and zero-shot region-recognition settings.Source: §IV · pp. 5–6, Table I · p. 6
  • Fine-grained recognition, few-shot recognition, and zero-shot object recognition are the three task areas identified for experiments.The zero-shot task uses object detection datasets.Source: §IV · pp. 5–6
  • The experiments also include ablation studies of the method's design choices.Source: §IV · p. 6

A. Experimental Setup

The setup evaluates fine-grained, few-shot, and zero-shot recognition on classification and detection benchmarks using task-specific metrics. Prompts provide retrieved categories for ranking, and implementation uses a frozen CLIP encoder and HNSW retrieval.

A. Experimental Setup

  • Five fine-grained datasets are evaluated with clustering accuracy (cACC) and semantic similarity accuracy (sACC).The datasets are Bird-200, Cars-196, Dog-120, Flower102, and Pet-37.Source: A. Experimental Setup · p. 6
  • Few-shot image recognition uses 11 datasets and reports top-1 accuracy (%).The set includes general, textual, scene, satellite, facial-expression, car-type, and fine-grained datasets.Source: A. Experimental Setup · p. 6
  • Zero-shot object recognition uses LVIS and V3Det, reporting APr, APc, APf, and APall for LVIS and mAP for V3Det.LVIS has 1,203 categories; V3Det has 13,204 categories.Source: A. Experimental Setup · p. 6
  • The prompt supplies retrieved categories for MLLMs to rank; implementation uses frozen CLIP ViT B/16 embeddings, HNSW, and top-k retrieval, generally with k = 5.The 4-shot setting uses k = 4; 30k FGVC-Aircraft examples support one-epoch fine-tuning at learning rate 1e−5.Source: A. Experimental Setup · p. 6

B. Fine-Grained Visual Recognition

RAR is evaluated for fine-grained recognition using a memory built from three unlabelled images per category. It achieves the top averaged clustering and semantic-similarity results across five datasets, while not reaching state of the art on Dog-120 and Flower-102.

B. Fine-Grained Visual Recognition

  • RAR uses three unlabelled images per category to build its retrieval memory and evaluates clustering accuracy alongside category-name semantic similarity.cACC assesses clustering images from the same category; sACC measures similarity between cluster category names and ground truth.Source: B. Fine-Grained Visual Recognition · p. 6
  • 58.5% cACC and 65.3% sACC: RAR achieves the top averaged results across five fine-grained recognition datasets.The reported metrics are clustering accuracy (cACC) and semantic similarity accuracy (sACC).Source: B. Fine-Grained Visual Recognition · p. 7, Table II · p. 7
  • RAR does not reach state-of-the-art results on Dog-120 or Flower-102, where some baselines use near-exhaustive category knowledge bases.WordNet covers all Dog-120 ground-truth categories, while CaSED includes 101 of 102 Flower-102 categories; RAR achieves the best results on the majority of fine-grained datasets.Source: B. Fine-Grained Visual Recognition · p. 7

C. Few-Shot Image Recognition

Across few-shot image recognition experiments, RAR combines CLIP-based retrieval with MLLM ranking and improves classification results across shot settings and datasets. Additional evaluations examine larger retrieval sets, stronger feature extraction, full-training-set memory, and how candidate-list size affects CLIP and MLLMs.

C. Few-Shot Image Recognition

  • With CLIP ViT-L/14@336 as feature extractor, RAR’s gains were 4.9% on 4-shot and 4.7% on 8-shot settings; top-5 accuracy also rose 1.3% averaged across eleven datasets.The top-5 evaluation used the 4-shot setting.Source: C. Few-Shot Image Recognition · p. 7, Table IV · p. 9, Table V · p. 9
  • RAR significantly outperforms GPT-4V in accuracy, with average precision exceeding GPT-4V by 12.6% across eleven datasets, regardless of whether the base model is LLaVA, Intern-IXC2, or Qwen-VL.The GPT-4V results were copied from [45].Source: C. Few-Shot Image Recognition · p. 7, Table VI · p. 9
  • In selected fine-grained examples, GPT-4V identified cues such as a car’s “coupe” shape, an aircraft’s long fuselage, and prominent pet ears that CLIP may overlook.The examples were chosen from cases where CLIP failed to classify correctly.Source: C. Few-Shot Image Recognition · p. 8
  • CLIP+KNN exceeded 90% accuracy in four of six datasets at top-10, while MLLM distinction ability declined as input category counts rose from 10 to 30.The MLLM category-count test used Caltech101 and Flowers102 and ensured the correct category was included.Source: C. Few-Shot Image Recognition · p. 9
  • RAR first retrieves likely candidates with CLIP and then uses MLLMs to select among them, combining their complementary classification strengths.The paper describes CLIP as narrowing the candidate set and MLLMs as making the finer selection.Source: C. Few-Shot Image Recognition · p. 9
  • Across 1-, 2-, 4-, 8-, and 16-shot settings, RAR beat CLIP+KNN by 7.4%, 6.8%, 6.2%, 6.8%, and 6.3%, averaging a 6.7 percentage-point increase.For 4-shot top-1 accuracy, the eleven-dataset average rose from 57.0 to 63.2 (%).Source: C. Few-Shot Image Recognition · pp. 7, 9, Table III · p. 8

D. Zero-Shot Object Recognition

For zero-shot object recognition, RAR adds retrieval and MLLM ranking to object proposals with supplied boxes. It improves reported performance on LVIS and V3Det, including rare classes and a 13,204-class setting.

D. Zero-Shot Object Recognition

  • With InternLM-XC2 on LVIS, RAR improved over CLIP by 8.4 percentage points and over RegionCLIP by 6.4 percentage points.The evaluation used the LVIS v1.0 validation set.Source: D. Zero-Shot Object Recognition · p. 9, Table VII · p. 10
  • On LVIS rare classes, RAR’s APr exceeded CLIP by as much as 19.6%.RAR’s peak performance was on APr, while CLIP’s performance increased progressively from APr through APc to APf.Source: D. Zero-Shot Object Recognition · p. 10
  • On V3Det’s 13,204 classes, RAR with InternLM-XC2 surpassed the CLIP baseline by 1.5 percentage points in APall.The paper describes V3Det as an extremely large fine-grained category setting.Source: D. Zero-Shot Object Recognition · p. 10

E. Ablation Experiments

The ablations examine RAR’s sensitivity to fine-tuning data, ranking strategy, retrieval settings, and proposals. They report robustness across several choices, alongside retrieval-speed and accuracy trade-offs.

Different Fine-tuning data

  • RAR is not sensitive to the fine-tuning dataset across 10 additional datasets; FGVC-Aircraft yields 82.7% versus 82.0% for Stanford-Cars on Qwen-VL.InternLM-XC2 shows the same trend, and the authors choose FGVC-Aircraft for fine-tuning.Source: E. Ablation Experiments · p. 11

Fine-tuning vs In-Context Learning

  • Fine-tuned ranking improves accuracy over in-context learning across almost all datasets for both Qwen-VL and InternLM-XC2.The comparison uses FGVC-Aircraft fine-tuning versus training-free in-context prompts.Source: E. Ablation Experiments · p. 11, Table IX · p. 12

Ablation Study on Effects of the Parameter k

Results on More MLLM and Embedding Models

  • RAR improves results across embedding strengths, gaining +15.4, +8.5, and +7.3 over OpenCLIP ViT-B/16, OpenCLIP ViT-g/14, and EVA-CLIP ViT-L/14, respectively.With weaker OpenCLIP RN50, RAR gains +21.2 on average.Source: E. Ablation Experiments · p. 12, Table XI · p. 13

Ablation study on HNSW and brute-force retrieval methods

  • HNSW retrieval is significantly faster than brute-force search, but its accuracy drops on some datasets and improves on others.The experiments compare average single-retrieval time and accuracy using OpenCLIP ViT-B/16 in the 4-shot setting.Source: E. Ablation Experiments · p. 12, Table XII · p. 13

Ablation study on different retrieval methods

  • RAR’s retrieval method achieves near-optimal top-1 accuracy; IndexIVF and IndexLSH perform worse, especially IndexLSH on DTD.The alternatives tested include IndexHNSW, IndexPQ, IndexIVF, and IndexLSH.Source: E. Ablation Experiments · p. 12, Table XIII · p. 13

Ablation study on different proposals

  • With Detic-generated proposals, RAR retains a clear improvement over baseline, while inaccurate detections lower absolute AP versus ground-truth boxes.The compared models use the same ground-truth boxes except CLIP w/mask, which uses ground-truth masks.Source: E. Ablation Experiments · p. 12

V. CONCLUSION

The conclusion presents retrieval-and-ranking with MLLMs as a way to address their limited few-shot and zero-shot performance on fine-grained datasets. RAR ranks retrieved results and is proposed for recognition settings with expanding category variety and volume.

V. CONCLUSION

  • RAR combines retrieval with MLLM ranking to address limited few-shot and zero-shot perception on fine-grained datasets.The conclusion identifies fine-grained recognition, zero-shot image recognition, and few-shot object recognition as target tasks.Source: §V · p. 13
  • The authors describe RAR as seamlessly integrable into various MLLMs for applications where category variety and volume continue to expand.Source: §V · p. 13
  • The conclusion suggests retrieval-augmented MLLMs may also benefit future reasoning and generation tasks.Source: §V · p. 13
Loading 2403.13805v2…