PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Generation for Large Language Models: A Survey

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, Haofen Wang

arXiv:2312.10997v5cs.CLcs.AI

TL;DR

RAG research has grown rapidly without a systematic synthesis of its trajectory and evaluation practices. This survey organizes the field’s paradigms, core technologies, tasks, datasets, benchmarks, and evaluation methods, concluding that RAG advances LLM capabilities by integrating model knowledge with external data while leaving robustness and semi-structured-data handling as open challenges.

  • Problem

    Rapid RAG growth lacks systematic synthesis of its trajectory and evaluation practices, limiting consolidated understanding of the field.

  • Method

    The survey reviews over 100 studies across Naive, Advanced, and Modular RAG, analyzing retrieval, generation, augmentation, tasks, datasets, benchmarks, and evaluation methods.

  • Results

    The survey maps RAG’s evolution and identifies three developmental paradigms alongside technologies spanning retrieval, generation, and augmentation.

  • Takeaways & Limitations

    RAG advances LLM capabilities by integrating parameterized model knowledge with extensive non-parameterized data from external knowledge bases.

  • Takeaways & Limitations

    Conventional RAG systems remain challenged by semi-structured data because text splitting can corrupt tables and tables complicate semantic similarity search.

Abstract

from arXiv · show

Large Language Models (LLMs) showcase impressive capabilities but encounter challenges like hallucination, outdated knowledge, and non-transparent, untraceable reasoning processes. Retrieval-Augmented Generation (RAG) has emerged as a promising solution by incorporating knowledge from external databases. This enhances the accuracy and credibility of the generation, particularly for knowledge-intensive tasks, and allows for continuous knowledge updates and integration of domain-specific information. RAG synergistically merges LLMs' intrinsic knowledge with the vast, dynamic repositories of external databases. This comprehensive review paper offers a detailed examination of the progression of RAG paradigms, encompassing the Naive RAG, the Advanced RAG, and the Modular RAG. It meticulously scrutinizes the tripartite foundation of RAG frameworks, which includes the retrieval, the generation and the augmentation techniques. The paper highlights the state-of-the-art technologies embedded in each of these critical components, providing a profound understanding of the advancements in RAG systems. Furthermore, this paper introduces up-to-date evaluation framework and benchmark. At the end, this article delineates the challenges currently faced and points out prospective avenues for research and development.

I. INTRODUCTION · II. OVERVIEW OF RAG · A. Naive RAG

The survey traces RAG’s evolution from Naive RAG through Advanced and Modular RAG, explaining how retrieval, generation, and augmentation address LLM limitations. It also reviews evaluation practices and identifies challenges and future directions.

  • I. INTRODUCTION: LLMs struggle with hallucinations and knowledge-intensive queries requiring information beyond their training data.RAG addresses these limitations by retrieving relevant document chunks from external knowledge bases through semantic similarity.
  • I. INTRODUCTION: The survey synthesizes over 100 RAG studies, mapping the field’s evolution, technical paradigms, research methods, and anticipated future paths.It contextualizes RAG research within the broader landscape of large language models.
  • II. OVERVIEW OF RAG: RAG research is organized into three paradigms: Naive RAG, Advanced RAG, and Modular RAG.Advanced and Modular RAG emerged in response to limitations in Naive RAG, despite RAG methods outperforming native LLMs and remaining cost-effective.
  • I. INTRODUCTION: The survey examines retrieval, generation, and augmentation as central technologies whose synergies form a cohesive RAG framework.These components are analyzed as the core stages through which external knowledge is integrated into LLM responses.
  • I. INTRODUCTION: The survey summarizes RAG evaluation across 26 tasks and nearly 50 datasets, covering objectives, metrics, benchmarks, and tools.It also anticipates future directions aimed at addressing current RAG challenges.
  • A. Naive RAG: Naive RAG uses a “Retrieve-Read” workflow comprising indexing, retrieval, and generation.It became prominent after ChatGPT’s widespread adoption and represents the earliest RAG methodology.
  • A. Naive RAG: Indexing cleans and segments documents, encodes chunks into vectors, and stores them in a vector database for similarity search.Retrieval encodes the query with the same model and selects the top K most similar chunks as expanded prompt context.
  • A. Naive RAG: Naive RAG suffers from retrieval errors, unsupported or low-quality generation, difficult augmentation, redundancy, and overreliance on retrieved content.Retrieval may miss crucial information or select irrelevant chunks, while generation and augmentation can produce hallucinated, incoherent, repetitive, biased, or stylistically inconsistent responses.

B. Advanced RAG

Advanced RAG improves Naive RAG by optimizing retrieval before and after searching, while retaining a chain-like structure. Its methods refine indexing, clarify queries, rerank retrieved chunks, and compress context.

  • B. Advanced RAG: Advanced RAG addresses Naive RAG’s limitations through pre-retrieval and post-retrieval optimization strategies while maintaining a chain-like process.It differs from Modular RAG’s greater flexibility by focusing on targeted retrieval improvements.
  • B. Advanced RAG: Advanced RAG refines indexing with sliding windows, fine-grained segmentation, metadata, and other optimization methods to streamline retrieval.These techniques specifically target indexing issues and retrieval quality.
  • Pre-retrieval process: Pre-retrieval improves indexed content through finer data granularity, optimized index structures, metadata, alignment optimization, and mixed retrieval.Query optimization additionally makes the original user question clearer and more suitable for retrieval.
  • Post-Retrieval Process: Post-retrieval integrates retrieved context through chunk reranking and context compression, relocating the most relevant content toward prompt edges.These strategies are implemented in frameworks including LlamaIndex, LangChain, and HayStack.

C. Modular RAG … III. RETRIEVAL

The paper presents Modular RAG as an adaptable architecture that expands retrieval, generation, and orchestration through specialized modules and configurable interaction patterns. It also contrasts RAG with fine-tuning and identifies retrieval-source, granularity, preprocessing, and embedding-model selection as key retrieval concerns.

  • C. Modular RAG: Modular RAG improves adaptability and versatility by adding or refining components, restructuring modules, and rearranging pipelines for specific challenges.Examples include adding a search module for similarity search and fine-tuning the retriever.
  • 1) New Modules:: Specialized Modular RAG modules support direct multi-source search, memory-guided retrieval, routing, context prediction, and task-specific adaptation.The Search module can access search engines, databases, and knowledge graphs using LLM-generated code and query languages.
  • 2) New Patterns:: Unlike the fixed Retrieve-Read structure of Naive and Advanced RAG, Modular RAG permits module substitution, reconfiguration, addition, and interaction-flow adjustment.This flexibility broadens applicability across different tasks.
  • 2) New Patterns:: New patterns refine, replace, or diversify retrieval through query rewriting, LLM-generated content, model-weight retrieval, and hybrid retrieval strategies.Rewrite-Retrieve-Read uses rewriting and LM feedback, while Generate-Read and Recite-Read replace or redirect traditional retrieval.
  • 2) New Patterns:: Dynamic module arrangements such as DSP and ITER-RETGEN use module outputs to strengthen other modules, while FLARE and Self-RAG enable adaptive retrieval orchestration.ITER-RETGEN iteratively follows a Retrieve-Read-Retrieve-Read flow.
  • D. RAG vs Fine-tuning: RAG is compared with prompt engineering and fine-tuning along external knowledge requirements and model adaptation requirements.Prompt engineering primarily uses inherent model capabilities, whereas RAG supplies tailored external knowledge.
  • D. RAG vs Fine-tuning: RAG supports real-time knowledge updates, external knowledge use, and interpretability, but entails higher latency and ethical considerations around data retrieval.Fine-tuning is more static, requires retraining for updates, enables deeper behavioral and stylistic customization, and demands substantial computational resources.
  • D. RAG vs Fine-tuning: Across knowledge-intensive evaluations, RAG consistently outperforms unsupervised fine-tuning on both training-encountered and entirely new knowledge, while LLMs struggle to learn new facts through unsupervised fine-tuning.The choice between RAG and fine-tuning depends on data dynamics and customization needs.

A. Retrieval Source · 1) Data Structure: · 2) Retrieval Granularity:

RAG performance depends on both the retrieval source and the granularity of retrieved units. Sources have expanded from unstructured text to semi-structured, structured, and LLM-generated content, while granularity requires balancing relevance, redundancy, retrieval burden, and semantic integrity.

  • A. Retrieval Source: Both the retrieval source type and retrieval-unit granularity affect the final generation results in RAG.
  • 1) Data Structure:: RAG increasingly incorporates LLM-generated content, with approaches selectively retrieving based on knownness or replacing retrieval with generated contexts.
  • 1) Data Structure:: Unstructured text remains the most widely used retrieval source, including Wikipedia for open-domain question answering and cross-lingual or domain-specific corpora.
  • 1) Data Structure:: Semi-structured sources such as PDFs combine text and tables, but conventional RAG may corrupt tables during splitting and complicate semantic similarity searches.
  • 1) Data Structure:: Structured sources such as knowledge graphs can provide more precise, typically verified information, while requiring additional effort to build, validate, and maintain.
  • 2) Retrieval Granularity:: Coarse retrieval units may provide more relevant information but include distracting redundancy, whereas fine-grained units increase retrieval burden and may lack semantic integrity.
  • 2) Retrieval Granularity:: Choosing an appropriate retrieval granularity during inference is presented as a simple strategy for improving dense-retriever retrieval and downstream-task performance.
  • 2) Retrieval Granularity:: Text retrieval granularity spans tokens, phrases, sentences, propositions, chunks, and documents, with propositions defined as concise, self-contained atomic factual expressions.

B. Indexing Optimization … 1) Query Expansion:

The paper presents indexing and query optimization techniques that improve retrieval quality by structuring documents, enriching chunks, and reformulating complex or underspecified queries. These methods aim to retrieve more relevant, coherent, current, and reliable context for RAG generation.

  • B. Indexing Optimization: Index construction processes documents into segmented embeddings stored in a vector database, and its quality determines whether correct context is retrieved.The indexing phase includes document processing, segmentation, and embedding transformation.
  • 1) Chunking Strategy:: Fixed-size chunking balances context against noise, processing time, and cost, while recursive splitting and sliding windows address sentence truncation.Examples of fixed chunk sizes include 100, 256, and 512 tokens.
  • 2) Metadata Attachments:: Metadata such as page number, file name, author, category, and timestamp enables filtered retrieval and time-aware RAG that prioritizes fresher knowledge.Timestamp weighting helps avoid outdated information.
  • 2) Metadata Attachments:: Artificial metadata, including paragraph summaries and hypothetical questions generated by LLMs, reduces the semantic gap between queries and document answers.Retrieval compares the original question with the hypothetical question using similarity.
  • 3) Structural Index:: Hierarchical document indexes organize files and chunks through parent-child relationships, with node summaries supporting faster traversal and chunk selection.This structure can mitigate issues caused by block extraction.
  • 3) Structural Index:: Knowledge Graph indexes preserve consistency by delineating connections among concepts and entities, transforming retrieval into instructions that LLMs can comprehend.The approach is described as reducing illusions and improving accurate, contextually coherent responses.
  • C. Query Optimization: Naive RAG retrieval is weakened when original queries are imprecise, complex, poorly organized, ambiguous, or contain specialized vocabulary and abbreviations.The paper identifies direct reliance on the user’s original query as a primary challenge.
  • 1) Query Expansion:: Query expansion enriches retrieval context through multi-query generation, sub-question decomposition, and Chain-of-Verification validation to improve relevance and reduce hallucinations.Complex questions can be decomposed into simpler sub-questions using least-to-most prompting, while validated expanded queries are typically more reliable.

2) Query Transformation: … 2) Fine-tuning Embedding Model:

The section presents query transformation and routing strategies for improving retrieval, alongside embedding approaches that combine sparse and dense signals and fine-tune retrievers for specialized domains or generator alignment.

  • 2) Query Transformation:: RAG retrieves chunks using a transformed query rather than the user’s original query.
  • 2) Query Transformation:: LLMs or smaller models can rewrite queries, with Taobao’s BEQUE improving recall effectiveness for long-tail queries and increasing GMV.
  • 2) Query Transformation:: HyDE generates hypothetical documents for answer-to-answer embedding similarity, while Step-back Prompting abstracts queries into high-level concept questions.
  • 3) Query Routing:: Query routing directs varying queries to distinct RAG pipelines to accommodate diverse scenarios.
  • 3) Query Routing:: Metadata routing extracts query entities and filters chunks using keywords and metadata, while semantic routing uses query meaning; hybrid routing combines both.
  • D. Embedding: Embedding-based retrieval compares question and chunk embeddings, relying mainly on sparse BM25 encoders and dense BERT-architecture retrievers.
  • 1) Mix/hybrid Retrieval :: Sparse and dense retrieval provide complementary relevance information: sparse models can train dense retrievers, improve zero-shot retrieval, and help handle rare-entity queries.
  • 2) Fine-tuning Embedding Model:: Fine-tuning embedding models addresses specialized-domain discrepancies and can align retrievers with generators using LLM-derived supervision or task-specific query generation.

E. Adapter … 2) Context Selection/Compression:

The survey describes adapters for aligning retrieval with downstream tasks and recommends curating retrieved context before generation. It emphasizes reranking, compression, filtering, and critique to reduce noise and improve the information supplied to LLMs.

  • E. Adapter: External adapters help address fine-tuning challenges involving API integration and limited local computational resources.Adapters provide an alternative alignment mechanism when directly fine-tuning models is difficult.
  • E. Adapter: UPRISE retrieves task-suitable prompts, AAR provides a universal adapter, and PRCA adds a pluggable reward-driven contextual adapter for downstream tasks.These approaches adapt retrieval or prompting to improve multi-task or task-specific performance.
  • E. Adapter: Directive fine-tuning integrates knowledge into white-box models by replacing the retriever with a query-driven relevant-document generator.The method addresses fine-tuning difficulties while enhancing model performance.
  • IV. GENERATION: After retrieval, directly passing all information to the LLM is discouraged, motivating adjustments to retrieved content and the LLM.The generation stage therefore includes separate strategies for refining context and adapting the model’s processing.
  • A. Context Curation: Redundant information and overly long contexts can cause the “Lost in the middle” problem, so retrieved content requires further processing.LLMs tend to focus on the beginning and end of long texts while forgetting the middle portion.
  • 1) Reranking:: Reranking reorders document chunks to prioritize pertinent results, reducing the document pool while enhancing and filtering inputs for language-model processing.It may use rule-based metrics such as Diversity, Relevance, and MRR or model-based Encoder-Decoder approaches.
  • 2) Context Selection/Compression:: Excessive context can add noise and diminish key-information perception, making context selection and compression essential in RAG.LLMLingua removes unimportant tokens with small language models, while Filter-Reranker uses SLMs as filters and LLMs as reordering agents; LLM critique can also remove poorly relevant documents.

B. LLM Fine-tuning … C. Adaptive Retrieval

The paper presents fine-tuning as a way to provide domain knowledge and control LLM input/output behavior, while describing retrieval augmentation strategies that make RAG more iterative, recursive, and adaptive. These methods expand contextual coverage and enable LLMs to decide how and when external information should be retrieved.

  • B. LLM Fine-tuning: Targeted fine-tuning can provide missing domain knowledge while adapting LLM inputs, outputs, data formats, and response styles to specific scenarios.Hugging Face fine-tuning data can serve as an initial step.
  • B. LLM Fine-tuning: Fine-tuning can also align LLM outputs with human, model, or retriever preferences through reinforcement-learning feedback.One described approach manually annotates generated answers and uses the feedback for reinforcement learning.
  • V. AUGMENTATION PROCESS IN RAG: Once-only retrieval followed by generation can be inefficient and insufficient for complex problems requiring multi-step reasoning because it supplies limited information.The survey summarizes retrieval optimization approaches in Figure 5.
  • A. Iterative Retrieval: Iterative retrieval alternates searches with generation using the initial query and generated text, providing richer context and improving answer robustness through additional references.It may nevertheless suffer from semantic discontinuity and accumulating irrelevant information; ITER-RETGEN combines retrieval-enhanced generation with generation-enhanced retrieval.
  • B. Recursive Retrieval: Recursive retrieval repeatedly refines queries and uses previous results or chain-of-thought to converge on more relevant information.IRCoT guides retrieval with chain-of-thought and refines the reasoning process using retrieved results.
  • B. Recursive Retrieval: Recursive and multi-hop retrieval address specific data scenarios by hierarchically refining document searches or extracting interconnected information from graph-structured sources.A hierarchical process may summarize a document or lengthy PDF before conducting a secondary retrieval within it.
  • C. Adaptive Retrieval: Adaptive retrieval lets LLMs determine when and what to retrieve, improving the efficiency and relevance of externally sourced information.Flare and Self-RAG exemplify this approach, while Graph-Toolformer proactively uses retrievers, Self-Ask, and few-shot prompts to initiate searches.
  • C. Adaptive Retrieval: WebGPT trains GPT-3 to use search autonomously, whereas Flare triggers retrieval when generated-term probabilities fall below a threshold and Self-RAG uses reflection tokens or thresholds to activate retrieval.WebGPT uses special tokens for searching, browsing results, and citing references; Self-RAG includes “retrieve” and “critic” reflection tokens.

VI. TASK AND EVALUATION … 1) Quality Scores:

This chapter surveys RAG’s main downstream tasks, datasets, and evaluation methods. It frames evaluation around retrieval and generation, using task-specific metrics and quality scores for context relevance, answer faithfulness, and answer relevance.

  • VI. TASK AND EVALUATION: RAG evaluation aims to understand and optimize model performance across diverse application scenarios, covering downstream tasks, datasets, and evaluation methods.The chapter focuses on evaluating RAG systems as their adoption in NLP and the LLM community grows.
  • A. Downstream Task: Question answering remains RAG’s core task, spanning single-hop, multi-hop, multiple-choice, domain-specific, and long-form scenarios.RAG is also expanding to information extraction, dialogue generation, and code search, with corresponding datasets summarized in Table II.
  • B. Evaluation Target: Traditional RAG assessments evaluate performance on specific downstream tasks using established task-appropriate metrics.Examples include EM and F1 for question answering, Accuracy for fact-checking, and BLEU and ROUGE for answer quality.
  • B. Evaluation Target: Retrieval quality measures the effectiveness of context sourced by the retriever using search, recommendation, and information-retrieval metrics.Common metrics include Hit Rate, MRR, and NDCG.
  • B. Evaluation Target: Generation quality assesses whether the generator synthesizes coherent and relevant answers from retrieved context.Unlabeled content is evaluated for faithfulness, relevance, and non-harmfulness, while labeled content emphasizes information accuracy.
  • C. Evaluation Aspects: Contemporary RAG evaluation emphasizes three quality scores and four essential abilities for assessing retrieval and generation.These evaluation aspects collectively target the two principal functions of a RAG model.
  • 1) Quality Scores:: The quality scores are context relevance, answer faithfulness, and answer relevance, covering distinct stages of retrieval and generation.Context relevance favors precise, specific retrieved content; answer faithfulness requires consistency with that context; answer relevance requires directly addressing the question.

2) Required Abilities: · D. Evaluation Benchmarks and Tools · VII. DISCUSSION AND FUTURE PROSPECTS

The paper identifies four abilities for assessing RAG adaptability and efficiency, while distinguishing retrieval- and generation-quality evaluation. It also surveys benchmarks and automated tools, and notes that RAG evaluation metrics remain neither mature nor standardized despite ongoing progress and future challenges.

  • 2) Required Abilities:: RAG evaluation assesses noise robustness, negative rejection, information integration, and counterfactual robustness as abilities affecting performance in complex scenarios.These abilities concern handling noisy documents, declining unsupported answers, synthesizing multiple documents, and rejecting known inaccuracies.
  • 2) Required Abilities:: Noise robustness evaluates handling of question-related documents lacking substantive information, whereas negative rejection evaluates refraining from answering when retrieved knowledge is insufficient.Both abilities test whether retrieval results support an appropriate response rather than merely prompting generation.
  • 2) Required Abilities:: Information integration evaluates synthesizing multiple documents for complex questions, while counterfactual robustness tests disregarding known inaccuracies despite misinformation warnings.These abilities target multi-document reasoning and resistance to explicitly identified inaccuracies.
  • 2) Required Abilities:: Context relevance and noise robustness evaluate retrieval quality, whereas answer faithfulness, answer relevance, negative rejection, information integration, and counterfactual robustness evaluate generation quality.The framework therefore separates retrieval assessment from generation assessment while covering overlapping robustness and reasoning concerns.
  • 2) Required Abilities:: Table III summarizes traditional metrics for evaluation aspects, but these measures do not yet constitute a mature or standardized RAG evaluation approach.Some studies have also developed custom metrics tailored to RAG-specific nuances, although they are not included in the summary.
  • D. Evaluation Benchmarks and Tools: Proposed benchmarks and tools provide quantitative metrics for measuring RAG performance and understanding capabilities across evaluation aspects.Examples include RGB, RECALL, and CRUD benchmarks, alongside automated tools such as RAGAS, ARES, and TruLens.
  • VII. DISCUSSION AND FUTURE PROSPECTS: Despite considerable progress in RAG technology, persistent challenges require further research into current limitations and future research directions.The discussion chapter is devoted primarily to these challenges and prospective directions for RAG.

A. RAG vs Long Context … E. Production-Ready RAG

The paper argues that RAG remains valuable despite expanding long-context capabilities because it improves efficiency and traceability, while future progress depends on robustness, hybrid training, scaling analysis, and production-oriented infrastructure. It also identifies unresolved engineering challenges and a still-incomplete integrated platform ecosystem.

  • A. RAG vs Long Context: RAG remains valuable despite LLMs handling contexts exceeding 200,000 tokens, because chunked retrieval improves operational efficiency and supports reference verification.Long-context prompting can slow inference, whereas RAG makes retrieval and reasoning observable.
  • B. RAG Robustness: Noise and contradictory retrieved information can degrade RAG output quality, making resistance to adversarial and counterfactual inputs a key performance focus.The paper characterizes this risk as “Misinformation can be worse than no information at all”.
  • C. Hybrid Approaches: Combining RAG with fine-tuning is a leading hybrid strategy, with sequential, alternating, and end-to-end joint-training arrangements still requiring investigation.The paper also highlights the need to exploit parameterized and non-parameterized advantages together.
  • C. Hybrid Approaches: CRAG exemplifies modular hybridization by training a lightweight retrieval evaluator that assesses retrieved-document quality and triggers knowledge-retrieval actions according to confidence.This approach introduces a specific functionality into the RAG and fine-tuning workflow.
  • D. Scaling laws of RAG: End-to-end and RAG-pre-trained models remain active research targets, but scaling-law applicability to RAG is uncertain and RAG parameter counts still lag behind LLMs.RETRO++ represents an initial effort to study these issues, including the possibility that smaller models may outperform larger ones.
  • E. Production-Ready RAG: Production adoption of RAG is increasing, but efficient retrieval, document recall in large knowledge bases, and protection against source or metadata disclosure remain critical engineering challenges.These challenges concern both system performance and data security.
  • E. Production-Ready RAG: RAG toolkits are converging toward a foundational technology stack for enterprise applications, yet a fully integrated comprehensive platform still requires further innovation and development.The ecosystem is also moving toward customization, simplification, and specialization for production environments.

F. Multi-modal RAG · VIII. CONCLUSION

RAG has expanded beyond text question answering into multimodal applications spanning images, audio, video, code, and structured knowledge. The survey concludes that Naive, Advanced, and Modular RAG reflect this evolution, while broader adoption increases the need for representative evaluation methodologies.

  • F. Multi-modal RAG: RAG has transcended text-based question answering by integrating retrieval and generation across diverse modal data and domains.
  • F. Multi-modal RAG: RA-CM3 retrieves and generates both text and images, while BLIP-2 supports zero-shot image-to-text conversion using frozen image encoders and LLMs.
  • F. Multi-modal RAG: GSS retrieves and stitches audio clips for speech translation, while UEOP and KNN-based attention fusion advance speech recognition and domain adaptation.
  • F. Multi-modal RAG: Vid2Seq augments language models with temporal markers to predict event boundaries and textual descriptions in a unified output sequence.
  • F. Multi-modal RAG: RBPS retrieves objective-aligned code examples for test assertion generation and program repair, while CoK integrates knowledge-graph facts as question-answering hints.
  • VIII. CONCLUSION: The survey frames RAG as combining language models’ parameterized knowledge with extensive non-parameterized data from external knowledge bases.
  • VIII. CONCLUSION: The survey identifies three developmental RAG paradigms—Naive, Advanced, and Modular—alongside applications across many different tasks.
  • VIII. CONCLUSION: The expanding RAG ecosystem requires refined evaluation methodologies that provide accurate and representative assessments of its contributions to AI research and development.
Loading 2312.10997v5…