PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook

Xu Zheng, Ziqiao Weng, Yuanhuiyi Lyu, Lutao Jiang, Haiwei Xue, Bin Ren, Danda Paudel, Nicu Sebe, Luc Van Gool, Xuming Hu

arXiv:2503.18016v1cs.CV

TL;DR

RAG can address limitations in model knowledge, but its application to computer vision remains an emerging area. This survey reviews RAG across visual understanding, generation, and embodied vision, synthesizes its advances and challenges, and identifies directions for future work.

  • Problem

    RAG in computer vision is an emerging area, while models still face challenges including hallucinations, outdated knowledge, and limited domain-specific expertise.

    Source: A. Background · p. 1, C. Related Works · p. 3

  • Method

    The survey systematically reviews RAG for visual understanding, visual generation, and embodied vision, and presents a taxonomy of techniques across vision tasks.

    Source: B. Contributions · p. 2

  • Results

    The survey concludes that RAG can improve computer-vision model performance by leveraging external knowledge to address knowledge limitations, outdated information, and domain-specific expertise.

    Source: §VI · p. 14

  • Takeaways & Limitations

    The survey provides a foundation for future exploration of retrieval-augmented techniques in computer vision.

    Source: B. Contributions · p. 2, §VI · p. 14

    The survey identifies retrieval efficiency, modality alignment, computational cost, and domain adaptation as challenges hindering wider adoption of RAG in computer vision.

    Source: B. Contributions · p. 2

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) has emerged as a pivotal technique in artificial intelligence (AI), particularly in enhancing the capabilities of large language models (LLMs) by enabling access to external, reliable, and up-to-date knowledge sources. In the context of AI-Generated Content (AIGC), RAG has proven invaluable by augmenting model outputs with supplementary, relevant information, thus improving their quality. Recently, the potential of RAG has extended beyond natural language processing, with emerging methods integrating retrieval-augmented strategies into the computer vision (CV) domain. These approaches aim to address the limitations of relying solely on internal model knowledge by incorporating authoritative external knowledge bases, thereby improving both the understanding and generation capabilities of vision models. This survey provides a comprehensive review of the current state of retrieval-augmented techniques in CV, focusing on two main areas: (I) visual understanding and (II) visual generation. In the realm of visual understanding, we systematically review tasks ranging from basic image recognition to complex applications such as medical report generation and multimodal question answering. For visual content generation, we examine the application of RAG in tasks related to image, video, and 3D generation. Furthermore, we explore recent advancements in RAG for embodied AI, with a particular focus on applications in planning, task execution, multimodal perception, interaction, and specialized domains. Given that the integration of retrieval-augmented techniques in CV is still in its early stages, we also highlight the key limitations of current approaches and propose future research directions to drive the development of this promising area.

I. Introduction

RAG supplements model inputs with relevant external knowledge to address challenges including hallucinations, outdated information, and limited domain expertise. This survey turns to computer vision, where applying retrieval augmentation is complicated by the multimodal nature of visual data.

A. Background

  • RAG addresses LLM challenges including hallucinations, outdated knowledge, and limited domain-specific expertise by supplying relevant retrieved factual information.The passage frames this as a motivation for RAG in generative AI.Source: A. Background · p. 1
  • RAG retrieves relevant external knowledge and combines it with model inputs, often requiring minimal adaptation and no additional training.The supplied discussion describes its language-based applications and motivation before turning to computer vision.Source: A. Background · p. 1
  • Visual understanding and generation rely on multimodal sources such as images, videos, and 3D models, complicating the use of traditional text-based RAG.The passage names object identification, anomaly detection, segmentation, and text-to-image generation as examples of vision tasks.Source: A. Background · p. 1
  • In vision, external knowledge can inform object interactions and spatial relationships in scene generation and provide up-to-date visual references for image classification.The survey also identifies image generalization and understanding as challenges for large vision-language models [18].Source: A. Background · p. 2

B. Contributions

The survey organizes retrieval-augmented techniques across computer-vision understanding, generation, and embodied vision, and identifies current limitations and future directions. It positions its review as the first focused on RAG in computer vision and surveys image-understanding approaches including retrieval-augmented recognition and captioning.

B. Contributions

  • The survey reviews RAG across visual understanding, visual generation, and embodied vision, identifies limitations and future directions, and covers multimodal learning in robotics and autonomous driving.Its proposed directions include real-time retrieval optimization, cross-modal fusion, privacy-aware retrieval, and retrieval-based generative modeling.Source: B. Contributions · p. 2
  • The survey presents itself as a resource for synthesizing existing work and informing future development of RAG in computer vision.The related-work discussion also identifies Figure 2 as the paper’s systemic overview.Source: B. Contributions · p. 2, C. Related Works · p. 3

C. Related Works

  • The survey describes itself as the first to focus on RAG in computer vision, extending earlier RAG reviews of language models and AIGC.It presents its purpose as synthesizing methods that combine retrieval with visual perception.Source: C. Related Works · pp. 2–3

A. Image Understanding

  • Retrieval-augmented recognition uses external information for long-tailed visual data; RAC adds retrieval to classification, while RALF retrieves related classes for open-vocabulary detection.A segmentation approach retrieves similar samples into a memory bank and conditions predictions on them.Source: A. Image Understanding · p. 3

1) Pattern Recognition:

  • Image-captioning methods retrieve style-aware images, captions, or paired image-caption embeddings; SAMLL-CAP finds dynamic retrieval more resilient than fixed top-k selection.The approaches include retrieving object names from external visual memory to prompt LLMs.Source: 1) Pattern Recognition · p. 3

2) Image Captioning:

Video RAG spans methods for video understanding and generation, with approaches ranging from direct retrieval to tool-assisted, multimodal pipelines. The survey highlights efficiency, long-range temporal reasoning, consistency, and security as continuing challenges.

B. Video Understanding

  • Video RAG methods include multiexpert prompting, modular retrieval, video-to-text pipelines, and divide-and-conquer strategies, while balancing efficiency with comprehensive understanding remains challenging.The discussion names MM-REACT, FlashRAG, VideoRAG, Visual RAG, ViTA, and OmAgent.Source: B. Video Understanding · p. 3

2) Long Video Retrieval and Understanding:

  • Long-video systems still struggle to balance processing speed and comprehension accuracy, preserve consistency over extended sequences, and handle complex temporal dependencies.The cited discussion also notes memory constraints and long-range dependency modeling as open challenges.Source: 1) Video Understanding · p. 4
  • Long-video pipelines range from direct database retrieval to LLM augmentation, chunking, iterative reranking, and tool-assisted multimodal reasoning.The taxonomy distinguishes five architectures, from a basic retrieval pipeline to multi-step LLM processing with specialized tools.Source: 2) Long Video Retrieval and… · p. 4

1) Video Understanding:

  • Video-generation systems use retrieved motion structures to guide synthesis, reusing existing content while adjusting its appearance.The survey describes potential for personalized, contextually accurate videos and identifies narrative consistency, visual coherence, and security as ongoing concerns.Source: 2) Long Video Retrieval and… · p. 5

3) Other Retrieval and Application Tasks:

Multimodal retrieval-augmented understanding spans visual question answering, document understanding, medical visual question answering, and general-purpose tasks. Its benchmarks vary in application focus, while existing VQA approaches can underuse image information relative to language knowledge.

C. Multimodal Understanding

  • Multimodal retrieval-augmented understanding remains an early-stage area, with applications spanning VQA, document understanding, medical VQA, and general-purpose multimodal tasks.The survey also outlines a multimodal retrieval pipeline and identifies MuRAG and REACT as pioneering works.Source: C. Multimodal Understanding · p. 5

1) Benchmark Datasets:

  • Benchmark datasets are grouped as traditional evaluation, general-purpose, and technology-focused sets, reflecting different application requirements and evaluation priorities.The survey’s technology-focused category includes RAG and hallucination-detection datasets.Source: C. Multimodal Understanding · p. 5, 1) Benchmark Datasets · p. 6
  • Traditional benchmarks cover VQA, document VQA, and fundamental vision tasks, including open-ended answering, multi-page retrieval, image captioning, and fine-grained detection.The cited examples include VQA v2, M3DocVQA, MS-COCO, and V3Det.Source: C. Multimodal Understanding · p. 5, 1) Benchmark Datasets · p. 6
  • General-purpose benchmarks assess multimodal capabilities, but current datasets have insufficient evaluation dimensions and imbalanced task coverage.The passage names ELEVATER, MMBench, SEED-Bench, and MMStar.Source: 1) Benchmark Datasets · p. 6

C. Multimodal Understanding

  • Many VQA methods emphasize language knowledge and overlook valuable information in images.The passage frames multimodal databases as a promising way to address this limitation.Source: 1) Benchmark Datasets · p. 6

2) Visual Question Answering (VQA):

The survey traces multimodal RAG for VQA from multimodal retrieval toward methods that preserve visual detail and improve reasoning. For document understanding, it contrasts OCR-based text retrieval with page-level multimodal retrieval and end-to-end methods, noting limits in visual information and multi-page handling.

2) Visual Question Answering (VQA):

  • Multimodal databases offer a promising solution: MuRAG was first to construct one, combining retrieval and language augmentation to generate high-quality outputs, while image-to-text conversion can lose crucial visual details.FLMR combines late interaction with multi-dimensional representations to capture fine-grained cross-modal relevance and preserve intricate visual features.Source: 2) Visual Question Answering (VQA) · pp. 6–7
  • RMR adds in-context learning for reasoning, while RagLLaVA uses knowledge-enhanced re-ranking and training-time noise injection for robustness.Source: 2) Visual Question Answering (VQA) · p. 7

3) Document Understanding:

  • DocVQA handles textual queries over documents, but page-image approaches demand accurate multimodal embedding and struggle to preserve relationships across long, multi-page documents.Source: 3) Document Understanding · p. 7
  • OCR-based RAG summarizes document images into text, improving document understanding but risking loss of critical visual information such as graphics.Source: 3) Document Understanding · p. 7
  • ColPali uses vision-language models to create contextualized page embeddings for visually rich retrieval, but focuses on retrieval rather than generation.Source: 3) Document Understanding · p. 7
  • VisRAG [13] and M3DocRAG add generation and end-to-end evaluation, with M3DocRAG targeting multi-page, multi-document scenarios.The survey also identifies most multimodal RAG methods as single-task systems and notes high pretraining and parameter costs for multi-task approaches.Source: 3) Document Understanding · p. 7

4) General-purpose Multi-task Integration:

The survey describes multi-task RAG for vision-language models as underexplored amid pretraining and parameter costs. It contrasts a unified-vector-space pipeline with heterogeneous-data retrieval and surveys methods for incorporating world knowledge and retrieved information into vision tasks.

4) General-purpose Multi-task Integration:

  • Multi-task RAG for vision-language models remains underexplored given pretraining costs and added parameter overhead; pipeline (b) uses a unified vector space, unlike pipeline (c), which handles heterogeneous data.Pipeline (b) supports efficient cross-modal comparison and lower integration complexity, while pipeline (c) increases database-management complexity.Source: 4) General-purpose Multi-task… · p. 8
  • ReVeaL integrates world knowledge through knowledge-aware self-supervised learning, while REAVL studies retrieval of world knowledge from knowledge graphs for tasks including VQA and image captioning.Source: 4) General-purpose Multi-task… · p. 8
  • REACT trains task-specific blocks without changing original weights, while RAVEN handles multiple tasks and SURf and RoRA-VLM selectively use retrieved information to resist irrelevant or misleading data.RAR [18] uses retrieval and ranking with an MLLM for fine-grained knowledge tasks.Source: 4) General-purpose Multi-task… · p. 8
  • Medical vision-language models integrate visual and textual medical knowledge for diagnosis support, but hallucinated non-factual or misleading responses challenge their reliability in healthcare.Source: 4) General-purpose Multi-task… · p. 8

5) RAG in Medical Domain:

For medical vision-language models, the survey presents retrieval of domain-specific knowledge and adaptive context selection as approaches to improving factuality and reliability. In text-to-image generation, it describes factual-information and domain-database interventions for specialized depictions, including historical and cultural images.

5) RAG in Medical Domain:

  • MMED-RAG [95] retrieves domain-specific medical knowledge from structured sources to support accurate, contextually relevant responses, while RULE adaptively selects context to balance external information and model reasoning.The survey identifies limits on retrieved contexts and potential over-reliance on external sources as challenges for direct RAG application to medical models.Source: 5) RAG in Medical Domain · p. 8

A. Image Generation

1) Text-based framework:

  • Text-to-image generation can produce nonfactual demographic distributions for specialized historical or cultural images; FAI [102] incorporates factual gender and racial information, while uses domain-specific databases such as artworks.Source: A. Image Generation · p. 9

2) Vision-based framework:

For image generation, the survey describes retrieval-based methods addressing training cost and realism, alongside multimodal retrieval frameworks. For video generation, Animate-AS retrieves motion structures to guide synthesis, and the survey reports potential benefits for efficient, coherent, contextually accurate video creation.

2) Vision-based framework:

  • Image-generation models face costly real-time training and generated objects that can appear too virtual relative to real-world objects.Source: 2) Vision-based framework · p. 9
  • RA-Diffusion uses nearest-neighbor retrieval from a large image database to compose scenes, reducing model parameters and training compute without generating everything from scratch.It can generalize to alternative image databases without additional training; ImageRAG [104] also dynamically retrieves prompt-relevant images as generation context.Source: 2) Vision-based framework · pp. 9–10
  • iRAG addresses factuality hallucination by retrieving images to improve object realism, while RealRAG [97] reports gains for fine-grained and unseen-object generation.Source: 2) Vision-based framework · p. 10
  • ReMoDiffuse retrieves motion references by semantic and kinematic similarity, aligns them with target motion, and uses condition mixture to mitigate classifier-free guidance sensitivity.The survey characterizes multimodal frameworks as fusing diverse database sources to improve generation completeness and diversity.Source: 2) Vision-based framework · p. 10

B. Video Generation

  • Animate-AS retrieves video clips by motion structure, then uses those structures to guide text-to-video synthesis and reuse content with adjusted appearance.The survey describes RAG-based video generation as offering potential for high-quality, personalized, contextually accurate videos and production efficiency.Source: B. Video Generation · p. 10

C. 3D Generation

The section reviews retrieval-augmented 3D generation alongside embodied AI applications in planning, execution, multimodal perception, and interaction. It describes retrieved 3D references and sensory or task information as inputs to these approaches.

C. 3D Generation

  • 3D generation is limited by sparse, quality-filtered datasets and by text or single-image inputs that cannot fully specify a detailed model.The largest 3D datasets contain around 10 million objects, compared with billions of 2D data pairs.Source: C. 3D Generation · pp. 10–11
  • Phidias retrieves a 3D reference, renders multi-view images, and reconstructs a model; its comparison shows more reasonable content than feed-forward methods.Other approaches retrieve and optimize 3D objects, apply RAG to molecular generation or human motion, or compose scenes from retrieved components; Diorama lacks an optimization stage for image consistency.Source: Figure 10 · p. 11, C. 3D Generation · p. 11, Figure 9 · p. 11

IV. Retrieval-Augmented Generation in Embodied AI

  • RAG in embodied AI retrieves external knowledge to inform decisions, task execution, multimodal perception, and applications in dynamic environments.The section frames these capabilities as supporting agent adaptability and performance.Source: §IV · p. 11

A. RAG in Planning and Task Execution

  • Planning and execution systems retrieve information during interactions, updating records such as instructions, scene graphs, trajectories, and task status.The cited work uses progressive retrieval in MINI-BEHAVIOR; another study retrieves knowledge during task execution to respond to unforeseen circumstances.Source: A. RAG in Planning and Task Execution · pp. 11–12

B. RAG in MultiModal Perception and Interaction

  • Multimodal perception approaches retrieve contextual or stored sensory information, including wireless sensor inputs and prior experiences, to support situational awareness and interaction.The cited examples combine multimodal data with RAG or use non-parametric memory to store and retrieve sensory data.Source: B. RAG in MultiModal Perception and… · p. 12

C. RAG in Specialized Domains

This section surveys RAG applications in specialized domains and outlines challenges and future directions for video understanding and multimodal learning. It also identifies broader needs for adaptive, scalable, and secure retrieval strategies.

C. RAG in Specialized Domains

  • RAG retrieves real-time traffic data and historical patterns for traffic management, and contextual demonstrations for explainable autonomous-driving decisions.The cited autonomous-driving model is reported to generalize well to new environments.Source: C. RAG in Specialized Domains · p. 12

1) RAU in Video Understanding:

  • Video retrieval methods still face efficiency, memory, long-range dependency, narrative consistency, visual coherence, and security challenges.The section points to dynamic allocation of retrieval resources and improved memory management as future directions for long sequences.Source: 1) RAU in Video Understanding · p. 12

2) RAU in Multimodal Learning:

B. Insights and New Outlook for RAG

  • The survey calls for adaptive, scalable, and secure retrieval strategies to improve generalization and reliability in real-world applications.It notes that recent work has addressed factual grounding, efficiency, and cross-modal interactions.Source: 2) RAU in Multimodal Learning · p. 12

1) RAG for Image Generation:

The section reviews RAG in image generation, 3D content generation, embodied AI, and downstream vision applications. It describes unresolved challenges alongside possible uses of retrieved information.

1) RAG for Image Generation:

  • RAG-based image generation is presented as addressing hallucination, domain adaptation, computational efficiency, and generated-image realism; Table VI organizes insights and future directions.The supplied table passage gives its title but not its contents.Source: 1) RAG for Image Generation · p. 13, Table VI · p. 13

2) RAG in 3D Content Generation:

C. RAG in Embodied AI

  • Embodied AI RAG advances task execution, multimodal perception, and specialized applications, while retrieval efficiency, cross-modal reasoning, and real-world scaling remain challenges.Table VII is identified as summarizing insights and future directions.Source: C. RAG in Embodied AI · p. 13

D. Application

  • RAG supports VQA responses and downstream image classification, object detection, and segmentation by retrieving external knowledge.These applications are described as improving performance and adaptability.Source: D. Application · p. 13

1) Multimodal Interaction and Understanding:

The section spans multimodal understanding, visual generation, embodied AI applications, and the survey’s conclusions. Across these areas, it describes retrieval as a way to incorporate external knowledge into vision tasks.

2) Vision Generation Tasks:

  • Visual generation uses retrieved knowledge to create images, videos, and 3D models, with uses including creative design and potential immersive or personalized content.The section distinguishes existing creative uses from potential future applications.Source: 1) Multimodal Interaction and… · p. 13

3) Multifaceted Applications of Embodied AI:

  • Embodied AI retrieves external knowledge in real time to adjust planning and execution; cited applications include autonomous driving, robotics, medical reporting, and industrial quality control.The medical example retrieves databases for reports and diagnoses, while the industrial example uses defect patterns and standards.Source: 2) Vision Generation Tasks · p. 13, 3) Multifaceted Applications of… · p. 14

VI. Conclusion

  • The survey reviews RAG in visual understanding, visual generation, and embodied vision, while identifying retrieval efficiency, modality alignment, and computational cost as challenges to broader adoption.It highlights real-time retrieval optimization, cross-modal fusion, and integration into embodied AI, 3D generation, and robotics as future directions.Source: §VI · p. 14
Loading 2503.18016v1…