PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models for Information Retrieval: A Survey

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, Ji-Rong Wen

arXiv:2308.07107v5cs.CLcs.IR

TL;DR

IR systems face challenges spanning data scarcity, interpretability, and potentially inaccurate generated responses as the field combines traditional retrieval with neural language models. This survey synthesizes LLM applications across query rewriting, retrieval, reranking, reading, and search agents, while identifying efficiency and evaluation challenges. Its conclusion reports benefits across these IR dimensions, alongside unresolved practical limitations.

  • Problem

    IR research must consolidate rapidly evolving LLM-enhanced methods while addressing data scarcity, interpretability, and potentially inaccurate responses.

  • Method

    The survey organizes LLM-for-IR research across query rewriters, retrievers, rerankers, readers, search agents, and related prompting and training approaches.

  • Results

    The survey reports that LLMs improve query understanding, query-document matching, fine-grained reranking, and the presentation of retrieved answers.

  • Takeaways & Limitations

    LLM-enhanced IR spans both augmentation of traditional components and unified search agents that perform multiple retrieval tasks.

  • Takeaways & Limitations

    LLM-based IR remains constrained by expensive and inefficient ranking, long retrieved inputs, and challenges in reformulation, retrieval, comprehension, extraction, and summarization.

Abstract

from arXiv · show

As a primary means of information acquisition, information retrieval (IR) systems, such as search engines, have integrated themselves into our daily lives. These systems also serve as components of dialogue, question-answering, and recommender systems. The trajectory of IR has evolved dynamically from its origins in term-based methods to its integration with advanced neural models. While the neural models excel at capturing complex contextual signals and semantic nuances, thereby reshaping the IR landscape, they still face challenges such as data scarcity, interpretability, and the generation of contextually plausible yet potentially inaccurate responses. This evolution requires a combination of both traditional methods (such as term-based sparse retrieval methods with rapid response) and modern neural architectures (such as language models with powerful language understanding capacity). Meanwhile, the emergence of large language models (LLMs), typified by ChatGPT and GPT-4, has revolutionized natural language processing due to their remarkable language understanding, generation, generalization, and reasoning abilities. Consequently, recent research has sought to leverage LLMs to improve IR systems. Given the rapid evolution of this research trajectory, it is necessary to consolidate existing methodologies and provide nuanced insights through a comprehensive overview. In this survey, we delve into the confluence of LLMs and IR systems, including crucial aspects such as query rewriters, retrievers, rerankers, and readers. Additionally, we explore promising directions, such as search agents, within this expanding field.

1 INTRODUCTION

Information retrieval underpins access to information across search engines and other systems, while evolving from term-based methods toward neural and LLM-enhanced approaches. This survey organizes rapidly developing LLM applications across core IR components and search agents.

  • IR systems retrieve relevant information for users and also support dialogue, question-answering, and recommender systems.
  • IR has evolved from term-based methods through vector-space and statistical language models toward neural approaches capturing semantic and contextual signals.
  • LLMs offer strong language understanding, generation, generalization, and reasoning abilities that researchers seek to apply to IR.
  • The survey reviews LLM applications in query rewriting, retrieval, reranking, reading, and search agents.
  • The survey focuses on techniques for developing and applying LLMs to IR and maintains a periodically updated repository of relevant papers and resources.

2 BACKGROUND

IR systems retrieve and rank relevant information from large repositories, evolving from term-based models toward neural and LLM-based approaches. The survey organizes this pipeline around query rewriting, retrieval, reranking, reading, and emerging search agents.

  • IR systems match and rank textual user queries against indexed databases to retrieve relevant information.
  • Traditional IR evolved from Boolean and vector-space models toward neural models that capture contextual information and support representation learning.
  • Query rewriters refine initial queries, retrievers recall documents efficiently, and rerankers reorder retrieved documents for higher ranking quality.
  • Readers generate intuitive answer texts from retrieved passages, while search agents seek to unify search and result understanding into an automated process.
  • Because fine-tuning LLMs for IR is often impractical, in-context learning and parameter-efficient fine-tuning have become prevailing application methods.

3 QUERY REWRITER

LLM-based query rewriters modify queries for ad-hoc and conversational retrieval using formats such as questions, keywords, and answer-incorporated passages. The survey describes prompting and supervised fine-tuning approaches while highlighting concept drift and model-dependent expansion benefits.

  • Query rewriting improves retrieval by enriching or reformulating queries to address vocabulary mismatch and conversational context.
  • Conversational query rewriters resolve references across dialogue turns by converting context-dependent questions into isolated queries.
  • LLM rewrites can take the form of questions, keywords, or answer-incorporated passages, depending on downstream retrieval needs.
  • Zero-shot prompting uses the LLM’s existing knowledge to generate expanded query terms without domain-specific examples.
  • Supervised fine-tuning tailors LLMs to query rewriting, but its effectiveness depends on constructing training data appropriate to the application scenario.
  • Query expansion can introduce unrelated information, causing concept drift and potentially irrelevant retrieval results.
  • Expansion tends to help weaker ranking models but harm stronger ones, favoring selective use with weaker models or substantially different target datasets.

4 RETRIEVER

The retriever is IR’s first-pass filter, so it must efficiently find broadly relevant documents while maintaining high recall. LLM-based approaches address ambiguous queries, noisy documents, scarce labels, and limited generalization through query refinement, data augmentation, and new retrieval architectures.

  • Retriever role: Retrievers efficiently collect broadly relevant documents for downstream ranking, where high recall is essential to final search quality.The retriever serves as the first-pass document filter in large repositories.
  • Challenges: Human relevance labels are costly, while short ambiguous queries and lengthy noisy documents make intent understanding and relevance extraction difficult.These constraints limit retriever knowledge boundaries and cross-domain generalization.
  • Data and query refinement: LLM-based retrieval research refines queries to express user intent and augments training data for dense retrievers, especially in zero- or few-shot settings.Data augmentation can generate pseudo queries or relevance signals when labeled data are scarce.
  • Training data augmentation: LLM-driven augmentation constructs query-document pairs or soft relevance labels to enlarge retrieval training corpora.ART generates normalized probabilities of each question conditioned on retrieved passages as soft labels.
  • Dense retrievers: LLM-based dense retrievers improve in-domain accuracy and out-of-domain generalizability while adding capabilities such as instruction following and in-context learning.OpenAI’s representation models also show consistent gains with increased model size and embedding dimension.
  • Dense retrievers: RepLLaMA surpasses conventional multi-stage retrieval accuracy using first-stage retrieval alone, although expanded model scale introduces extra computation costs.It is a fine-tuned embedder built on the open-source LLaMA-2-7B model.
  • Open challenges: Open challenges for LLM-based embedders include efficiency and adaptability.These challenges remain despite progress from fine-tuning, post-pretraining, and architectural modifications.
  • Generative retrievers: Generative retrievers encode queries and decode document identifiers, but scaling analyses find large-corpus coverage remains challenging.More parameters often improve performance, and synthetic document-generated queries can significantly enhance retrieval.

5 RERANKER

LLM-based rerankers reorder documents retrieved in the first pass according to query-document relevance. The survey organizes existing methods into four paradigms based on how LLMs are used.

  • Reranker role: Rerankers are second-pass filters that reorder retrieved document lists according to query-document relevance.The initial list may be produced by a retriever such as BM25.
  • Reranking paradigms: The survey categorizes LLM-based reranking into supervised, unsupervised, data-augmentation, and reasoning-intensive paradigms.These paradigms are summarized in Table 5.

5.1 Utilizing LLMs as Supervised Rerankers

Supervised rerankers adapt pretrained LLMs to query-document relevance through task-specific ranking data and varied encoder-decoder architectures and ranking losses.

  • 5.1 Utilizing LLMs as Supervised Rerankers: Supervised fine-tuning adapts pretrained LLMs to reranking because pretraining does not provide sufficient awareness of ranking or query-document relevance.Task-specific datasets such as MS MARCO provide relevance and irrelevance signals for parameter adjustment.
  • 5.1 Utilizing LLMs as Supervised Rerankers: Encoder-based rerankers such as monoBERT encode a query-document pair and derive its relevance score from the [CLS] representation.The pair is formatted as “[CLS] query [SEP] document [SEP]” before a linear scoring layer.
  • 5.1 Utilizing LLMs as Supervised Rerankers: Encoder-decoder rerankers commonly generate a binary token such as “true” or “false,” then aggregate pairwise probabilities into document relevance scores.For each document, the model compares it with other documents and aggregates the resulting probabilities.
  • 5.1 Utilizing LLMs as Supervised Rerankers: RankT5 directly predicts numerical relevance scores and optimizes pairwise or listwise ranking losses instead of generation loss.This design aligns the training objective more closely with the ranking task.
  • 5.1 Utilizing LLMs as Supervised Rerankers: Decoder-only models such as RankLLaMA format query-document pairs as prompts and use their representations for relevance calculation.RankLLaMA uses the last-token representation to calculate relevance.

5.2 Utilizing LLMs as Unsupervised Rerankers

Unsupervised rerankers prompt LLMs to rank documents pointwise, listwise, or pairwise, trading off performance, efficiency, cost, and positional robustness.

  • 5.2 Utilizing LLMs as Unsupervised Rerankers: As LLMs exceed 10 billion parameters, prompting becomes an increasingly important alternative to fine-tuning for unsupervised reranking.The main prompting paradigms are pointwise, listwise, and pairwise methods.
  • 5.2.1 Pointwise methods: Pointwise methods independently judge each query-document pair through relevance generation or query generation.Relevance generation outputs labels such as “Yes” or “No,” whereas query generation scores how likely the document is to generate the actual query.
  • 5.2.1 Pointwise methods: Zero-shot query-generation reranking has shown significant performance with T0 and robust ranking ability with instruction-unfine-tuned LLaMA.The relevance score is based on the average log-likelihood of generating query tokens conditioned on the document and prompt.
  • 5.2.1 Pointwise methods: Handcrafted prompts strongly affect reranker performance, while difficulty-based demonstration selection has not significantly outperformed random selection.The evidence identifies demonstration selection as challenging because query-document relationships are complex.
  • 5.2.2 Listwise Methods: Listwise methods rank document lists directly but are constrained by LLM input length, motivating sliding-window reranking.Adjacent windows create dependencies that can reduce inference efficiency.
  • 5.2.2 Listwise Methods: Listwise performance is highly sensitive to prompt document order, and random shuffling can make it worse than BM25.This indicates positional bias in listwise ranking; sliding windows also limit documents per inference and hinder parallelization.
  • 5.2.3 Pairwise Methods: Pairwise methods compare document pairs and aggregate discrete judgments, while setwise methods compare multiple documents per step to reduce comparisons.Pairwise ranking remains inefficient because extensive document-pair comparisons are required.
  • 5.2.4 Comparison and Discussion: Pointwise methods offer lower time complexity and batch inference but lack a performance advantage; listwise methods perform strongly with GPT-4 yet incur high API cost and non-reproducibility.Pairwise methods are competitive with the smaller 20B-parameter FLAN-UL2 but have low efficiency from comparing many pairs.

5.3 Utilizing LLMs for Training Data Augmentation

LLMs support reranker training by generating synthetic explanations, relevance labels, queries, and documents, and by distilling ranking capability into smaller models.

  • 5.3 Utilizing LLMs for Training Data Augmentation: LLM-based data augmentation generates explanations and relevance labels for query-passage pairs, which can train sequence-to-sequence ranking models.ExaRanker and ExaRanker-Open use GPT-3.5 or open-source LLMs for this process.
  • 5.3 Utilizing LLMs for Training Data Augmentation: InPars-Light synthesizes queries for documents through LLM prompting as a cost-effective augmentation strategy.Other approaches generate synthetic documents or explanations for ranking data.
  • 5.3 Utilizing LLMs for Training Data Augmentation: RankVicuna and RankZephyr distill LLM ranking outputs into specialized smaller rerankers through staged training.RankVicuna uses RankGPT3.5 lists as gold rankings, while RankZephyr adds a second stage using RankGPT4 results.

5.4 Reasoning-intensive Rerankers

Reasoning-intensive rerankers incorporate explicit reasoning into relevance assessment through teacher distillation, reinforcement learning, and training-time reasoning with inference-time efficiency.

  • 5.4 Reasoning-intensive Rerankers: Large Reasoning Models improve complex-task accuracy through explicit step-by-step reasoning, a capability relevant to query-intent understanding and cross-document comparison.These capabilities motivate their application to document reranking.
  • 5.4 Reasoning-intensive Rerankers: Reasoning-intensive rerankers use teacher distillation, rule-based reinforcement learning, and reasoning during training while omitting intermediate reasoning at inference.TFRank is described as a “think-free” pointwise ranker using this final strategy.
  • 5.4 Reasoning-intensive Rerankers: TFRank significantly improves performance on reasoning-intensive IR benchmarks such as BRIGHT and R2MED.
  • 5.4 Reasoning-intensive Rerankers: LLM reranking remains expensive and inefficient because it relies on API calls and large parameter counts, creating a cost-efficiency-performance tradeoff.Its adaptability to in-domain, non-standard, and reasoning-intensive datasets also requires more comprehensive exploration.

6 READER

Reader modules extend IR beyond document ranking by generating conclusive answers from retrieved text, while recent work studies retrieval timing, reference placement, attribution, and selective retrieval. The survey also identifies limitations involving passive interaction, context length, noisy references, and unresolved RAG challenges.

  • Reader: Reader modules generate conclusive answer passages from retrieved documents instead of presenting users with only ranked document lists.They use LLMs to comprehend retrieved text and produce more intuitive responses, often incorporating references.
  • Passive Readers: Passive reader methods supply retrieved documents to LLMs as inputs, leaving the LLM as a recipient rather than an active search participant.Representative approaches differ in backbone model, reference insertion point, retrieval timing, and tuning strategy.
  • Retrieval Strategies: Periodic and adaptive retrieval methods obtain additional references during generation, with approaches varying from sentence-level retrieval to whole-passage iteration and confidence-triggered retrieval.RETRO inserts references through cross-attention, IRCoT retrieves for each generated sentence, and FLARE determines retrieval timing from generation probabilities.
  • Limitations: Passive readers cannot issue queries interactively, and long retrieved documents can exceed LLM input limits; RAG systems also face unresolved reformulation, timing, comprehension, extraction, and summarization challenges.Poisoned or even typo-containing references can additionally affect answer generation.
  • Analysis: Relevant-reference placement significantly affects generation, with performance better when the reference appears at the beginning or end of the input.This finding motivates ranking retrieved knowledge before generation.
  • Analysis: More references increase answer attribution but decrease fluency, while always retrieving can hurt question answering when LLMs already know popular entities.Selective retrieval based on entity popularity, confidence, or consistency is proposed to reduce noisy-reference interference.

7 SEARCH AGENT

Search-agent research moves from fixed information-seeking pipelines toward autonomous systems that dynamically reason, search, and synthesize information. The survey organizes this work around architectures and information-seeking mechanisms, while noting trade-offs between single-agent simplicity and multi-agent specialization.

  • Overview: Search agents aim to autonomously search, interpret, and synthesize information through LLM-based reasoning and retrieval.The survey presents them as a developing direction for complex real-world information-seeking tasks.
  • Architecture: Static pipeline architectures assign predefined roles but are limited in adapting to dynamic and complex real-world interactions.Their inflexibility constrains performance in advanced reasoning and problem-solving.
  • Architecture: Single-agent frameworks use one LLM for reasoning, interaction, and answer generation, often allowing end-to-end reinforcement-learning training.ReAct-style systems can generate actions such as think, search, and answer.
  • Architecture: Multi-agent frameworks divide complex search and reasoning among specialized LLMs, improving role-specific effectiveness and efficiency but complicating joint reinforcement-learning optimization.For example, separate agents may handle reasoning and summarization.
  • Information Seeking: Information-seeking modules are categorized as API-based or browsing-based, respectively using search services or interactive environments that simulate human web interactions.Examples include search APIs, sandboxed browsers, and environments such as BrowserGym.

7.3 Optimization

Search-agent optimization progresses from basic retrieval decisions to multi-step reasoning and autonomous operation in open web environments. Methods use supervised trajectories, reinforcement learning, and hybrid data-and-training strategies to internalize planning, retrieval, and tool-use skills.

  • Optimization: Agent-tuning methods seek to internalize planning, reasoning, and tool use so general-purpose LLMs can perform exploratory information acquisition.The survey groups methods by progressively more capable search behaviors.
  • Basic Search: Basic retrieval optimization teaches agents when to search and how to formulate queries, addressing redundant searches and irrelevant information in passive pipelines.Open-RAG uses hybrid adaptive retrieval to control retrieval behavior with generated tokens and trains with distracting information.
  • Multi-step Search: Complex tasks require iterative “think-search-integrate-rethink” cycles for multi-hop question answering and open-domain problem solving.These capabilities are trained with both supervised and reinforcement-learning approaches.
  • Multi-step Search: Supervised methods generate expert-like retrieval trajectories containing intermediate sub-queries, sub-answers, and retrieval decisions for supervised fine-tuning.CoRAG uses rejection sampling, while Auto-RAG synthesizes instruction data for autonomous multi-round retrieval.
  • Multi-step Search: Reinforcement-learning methods treat search as a learnable reasoning action and optimize when and how agents intersperse queries through task-success rewards.ReSearch, R1-Searcher, and Search-R1 use RL-based search training, including two-stage training in R1-Searcher.
  • Autonomous Search: Autonomous web research requires handling noisy or conflicting information without explicit supervision, using planning, cross-validation, and multimodal understanding.End-to-end RL and hybrid training with synthesized data support browser-based planning, verification, and knowledge integration.

7.4 Benchmarks and Resources

Search-agent evaluation combines QA benchmarks for reasoning with task-oriented benchmarks for planning, tool use, and environmental interaction. Resources increasingly bundle benchmarks, training data, models, and toolkits to support both evaluation and development.

  • Evaluation Paradigms: Search-agent evaluation uses QA-style benchmarks and task-oriented benchmarks to measure reasoning, planning, tool use, and environmental interaction.The two paradigms cover both answer quality and practical agent behavior.
  • QA Benchmarks: QA benchmarks range from single-hop factual retrieval to multi-hop reasoning and expert-level challenges.Single-hop datasets test retrieval and reading comprehension, while multi-hop datasets require combining evidence across sources.
  • QA Benchmarks: Expert-level benchmarks stress deep domain knowledge and complex reasoning, including web retrieval designed to exceed what parametric memory can solve.Humanity’s Last Exam and BrowseComp represent these evaluation goals.
  • Task-oriented Benchmarks: Task-oriented benchmarks assess general assistant workflows, code and research, multi-agent coordination, and GUI control.They measure activities such as chained planning, debugging, role assignment, iterative refinement, and interaction with noisy stateful interfaces.
  • Resources: Integrated resources combine evaluation suites with data-generation pipelines, agent models, and toolkits for training and research.The Alibaba-NLP WebAgent repository pairs WebWalkerQA with WebDancer, WebShaper, and WebSailor resources.

8 FUTURE DIRECTION

The survey identifies future directions for LLM-enhanced IR across query rewriting, retrieval, reranking, reading, trustworthiness, bias mitigation, and evaluation. Key challenges include personalization, latency, query realism, answer reliability, source validation, and metrics suited to generated text.

  • Query Rewriter: LLM query rewriters need ranking-aware refinement, deeper conversational-search support, and personalization based on user-specific search behavior.Current methods can identify user intent, but rewritten queries may still yield poor rankings; conversational history and user profiles remain underexplored.
  • Retriever: LLM-based retrievers face high inference latency and may generate synthetic queries that do not align with real user queries, introducing training noise.Suggested directions include transferring capabilities to smaller models, quantization, and reinforcement learning for more realistic query simulation.
  • Reranker: Future rerankers should use user-specific information for personalized search and adapt to response, evidence, and entity ranking through specialized methods such as instruction tuning.Analyzing search histories can support user profiles and personalized result ordering.
  • Reader: Reader-based systems must improve answer reliability because LLMs can produce unfaithful answers even when retrieved references are supplied.The survey calls for studying how references affect generation to improve the credibility of reader-based IR systems.
  • Search Agent: Search agents require document validation because low-quality web information can amplify hallucinations, bias, and offensive content.Proposed safeguards include autonomous credibility and accuracy checks, alongside better training data, algorithmic adjustments, and continuous monitoring.
  • Evaluation and Bias: IR evaluation needs generation-oriented ranking and text-generation measures that address lexical matching limits, subtle semantic differences, and factuality.The survey also highlights source bias, where neural retrievers and rerankers may prefer LLM-generated documents over human-written ones.

9 CONCLUSION

The survey organizes recent LLM applications in IR around query rewriting, retrieval, reranking, and reading, while also examining LLMs as search agents. It concludes that generative answers and broader system roles raise questions about the conventional IR framework and future challenges.

  • 9 CONCLUSION: The survey categorizes LLM-enhanced IR research into query rewriters, retrieval, reranking, and reader modules.It describes improvements in intent identification, contextual query-document matching, fine-grained result ordering, and comprehensive answer generation.
  • 9 CONCLUSION: The survey also reviews LLMs as search agents and questions whether document-ranking IR remains sufficient when systems can directly generate answers.It notes possible IR roles in broader systems, including memory recall for multi-agent simulations.
Loading 2308.07107v5…