PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Search Model: Redefining Search Stack in the Era of LLMs

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei

arXiv:2310.14587v2cs.IRcs.CL

TL;DR

Modern search systems rely on independently optimized components, and the paper proposes a prompt-customized LLM framework to unify many search tasks. Preliminary experiments report competitive performance, while real-world deployment still faces inference, context-length, and other challenges.

  • Problem

    Modern search components are often optimized independently, producing a cumbersome stack while results for long-tail and complex information needs remain unsatisfactory.

    Source: §1 · p. 2

  • Method

    The large search model uses one customized LLM and natural-language prompts to handle information-retrieval tasks, while first-stage retrieval remains separate.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    Preliminary experiments report competitive performance on joint listwise ranking and answer generation; on TREC DL 2019, answer quality averages 8.4 versus 8.9 for gpt-35-turbo.

    Source: §1 · p. 2, §4 · p. 10

  • Takeaways & Limitations

    The framework offers a unified modeling approach for various IR tasks, with the paper presenting it as a conceptual framework supported by preliminary experiments.

    Source: §1 · p. 2, §5 · p. 10

    First-stage retrieval remains separate; retrieval-corpus-in-model approaches are described as immature, with scaling and dynamic updating limitations.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Modern search engines are built on a stack of different components, including query understanding, retrieval, multi-stage ranking, and question answering, among others. These components are often optimized and deployed independently. In this paper, we introduce a novel conceptual framework called large search model, which redefines the conventional search stack by unifying search tasks with one large language model (LLM). All tasks are formulated as autoregressive text generation problems, allowing for the customization of tasks through the use of natural language prompts. This proposed framework capitalizes on the strong language understanding and reasoning capabilities of LLMs, offering the potential to enhance search result quality while simultaneously simplifying the existing cumbersome search stack. To substantiate the feasibility of this framework, we present a series of proof-of-concept experiments and discuss the potential challenges associated with implementing this approach within real-world search systems.

1 Introduction

Modern search systems combine independently optimized components, making the stack cumbersome while long-tail and complex information needs remain difficult. The paper proposes a prompt-customized LLM framework for unifying search tasks and reports preliminary competitive results, alongside challenges to production deployment.

1 Introduction

  • Independent search components make the stack difficult to maintain, while long-tail and complex information needs remain poorly served.The stack spans retrieval, ranking, question answering, and other components.Source: §1 · pp. 1–2
  • LLMs’ language understanding and generation capabilities make them a candidate for unified modeling of search tasks.The paper points to their zero-shot and few-shot capabilities as motivation.Source: §1 · p. 2
  • The large search model handles search tasks except first-stage retrieval with one LLM, using natural-language prompts to customize tasks.The model generates SERP elements such as ranked documents, snippets, and direct answers.Source: §1 · p. 2
  • Preliminary experiments report competitive performance on joint listwise ranking and answer generation, while production use faces inference-cost, long-context, and responsible-AI challenges.The paper calls for further research on benchmarks and model architecture, training, and inference.Source: §1 · p. 2

2 Related Work

Related work covers neural information retrieval, retrieval-augmented generation, and LLM applications to search. The paper positions its framework as a way to unify search tasks that prior LLM work generally addresses individually.

2.1 Neural Information Retrieval

  • Neural IR spans retrieval, reranking, query generation, and question answering, but models can underperform BM25 out of domain.Large-scale contrastive pre-training and combining neural expressiveness with lexical matching are described as ways to improve generalization or robustness.Source: §2.1 · p. 3

2.2 Retrieval-Augmented Generation

  • RAG can improve generated-text factuality and informativeness, but optimal training and retrieved-information use remain open questions.Retrieved information can enter through concatenation, attention-based fusion, or output-probability interpolation.Source: §2.2 · p. 3

2.3 Large Language Models

  • Prior LLM work applies models to specific search tasks, whereas this paper proposes unifying various search tasks with one LLM.The cited applications include pseudo-query and pseudo-document generation, long-form question answering, and prompted ranking.Source: §2.3 · pp. 3–4

3 Large Search Model

The large search model reframes many search tasks as a unified, prompt-customized LLM while retaining separate first-stage retrieval. The section describes the framework’s task coverage, long-context and multimodal directions, and deployment challenges.

3.1 Unified Modeling of Information Retrieval Tasks

  • A large search model unifies diverse IR tasks in a customized LLM, but first-stage retrieval remains embedding- or lexical-based.The framework can potentially replace multiple search components, including online serving, data augmentation, indexing, and human evaluation tasks.Source: §3.1 · pp. 4–5, Figure 1 · p. 4

3.2 Customization through Prompting

  • Natural-language prompts specify different tasks without task-specific parameters, and the framework supports zero-shot generalization to new tasks at inference.In-context learning uses labeled input-output examples and typically lengthy prompts; instructions specify tasks without labeled data.Source: §3.2 · pp. 5–6

3.3 Long Context Modeling

  • Current LLM context limits constrain long-document ranking, retrieval-augmented generation over many passages, and conversational search using growing histories.Concatenating the 100 passages that Izacard and Grave [2021] found best for aggregating information can exceed many LLMs’ maximum input lengths.Source: §3.3 · pp. 6, 8

3.4 Multi-modal Large Search Model

  • Multimodal foundation models could support search that understands web content across modalities and synthesizes multimodal results for users’ needs.Current commercial systems offer some multimodal search through independent image, video, and music modules.Source: §3.4 · p. 8

3.5 Practical Considerations for Deployment

  • Real-world deployment must address inference cost and latency, hallucinations, and alignment with human values, including safeguards against harmful outputs.The paper identifies data filtering, content moderation, and red-team testing as safeguards; it also notes retrieval augmentation and scaling may reduce hallucinations, which remain unsolved.Source: §3.5 · pp. 8–9

4 Proof-of-Concept Experiments

The proof-of-concept fine-tunes LLaMA-7B for listwise ranking and retrieval-augmented answer generation as text-generation tasks. It outperforms BM25 and strong dense retrievers on ranking, while its generated-answer score is below its teacher model’s.

4 Proof-of-Concept Experiments

  • The LLaMA-7B model is fine-tuned on MS MARCO for listwise ranking and retrieval-augmented answer generation, with both tasks framed as text generation.Training uses top-100 passages from E5large-v2 for answer generation and extends context length from 2k to 16k.Source: §4 · p. 9
  • The model outperforms BM25 sparse retrieval and multiple strong dense retrievers on listwise ranking.Table 3 reports MRR@10 for MS MARCO and NDCG@10 for TREC DL 19/20.Source: §4 · p. 10, Table 3 · p. 10
  • On TREC DL 2019 queries, generated answers score 8.4 on a 1–10 scale, versus 8.9 for gpt-35-turbo.Source: §4 · p. 10
  • Manual inspection found generated answers mostly fluent and relevant, though some were less comprehensive than ground truth or repetitive.Table 4 gives an example answer conditioned on the top-100 retrieved contexts.Source: §4 · p. 10, Table 4 · p. 10

5 Conclusion

The conclusion presents the large search model as a unified LLM framework for search tasks, replacing numerous task-specific models. Proof-of-concept experiments support its feasibility, but broader evaluation and work on deployment challenges remain necessary.

5 Conclusion

  • The framework proposes unified modeling of information retrieval tasks with LLMs instead of numerous task-specific encoder or encoder-decoder models.Source: §5 · pp. 10–11
  • Proof-of-concept experiments demonstrate feasibility, while a larger-scale evaluation is needed for a more comprehensive assessment.Source: §5 · p. 11
  • High inference cost, long-context modeling, and potential misalignment risks are challenges requiring further research.Source: §5 · p. 11
  • The authors hope the work will inspire further research as LLM development brings a new wave of innovation to information retrieval.Source: §5 · p. 11

Related papers

Loading 2310.14587v2…