PaperlayerView exact revision on arXiv

Source-linked AI summary

Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs

Jiejun Tan, Zhicheng Dou, Yutao Zhu, Peidong Guo, Kun Fang, Ji-Rong Wen

arXiv:2402.12052v3cs.CL

TL;DR

LLMs need external retrieval for some questions but not others, and existing LLM-based judgments can be computationally expensive. SlimPLM uses a smaller proxy model’s heuristic answers to decide when retrieval is needed and what knowledge to retrieve. Across five datasets, it improves or matches state-of-the-art question-answering performance with little additional computational cost.

  • Problem

    Determining whether an LLM already knows a user question or needs external retrieval remains unresolved, while retrieval can be harmful when unnecessary and existing judgments can require costly LLM computation.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    SlimPLM uses a smaller proxy model to generate heuristic answers, judge retrieval necessity, rewrite answers into claim-specific queries, and filter unnecessary retrieval.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    SlimPLM improves end-to-end LLM question-answering performance across datasets, achieving or exceeding state-of-the-art results with little additional computational cost.

    Source: §4.5 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    A slim proxy model can help LLMs determine when and what to retrieve while reducing reliance on repeated LLM inference.

    Source: §4.5 · p. 7, §5 · p. 9

    SlimPLM is less suitable when nearly all questions either fall outside the LLM’s pre-training knowledge or require no external knowledge, and its three-model pipeline is complex.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The integration of large language models (LLMs) and search engines represents a significant evolution in knowledge acquisition methodologies. However, determining the knowledge that an LLM already possesses and the knowledge that requires the help of a search engine remains an unresolved issue. Most existing methods solve this problem through the results of preliminary answers or reasoning done by the LLM itself, but this incurs excessively high computational costs. This paper introduces a novel collaborative approach, namely SlimPLM, that detects missing knowledge in LLMs with a slim proxy model, to enhance the LLM's knowledge acquisition process. We employ a proxy model which has far fewer parameters, and take its answers as heuristic answers. Heuristic answers are then utilized to predict the knowledge required to answer the user question, as well as the known and unknown knowledge within the LLM. We only conduct retrieval for the missing knowledge in questions that the LLM does not know. Extensive experimental results on five datasets with two LLMs demonstrate a notable improvement in the end-to-end performance of LLMs in question-answering tasks, achieving or surpassing current state-of-the-art models with lower LLM inference costs.

1 Introduction

SlimPLM addresses the costly problem of deciding when an LLM needs retrieval by using a smaller proxy model to generate heuristic answers and guide retrieval. Its pipeline judges retrieval necessity, rewrites answers into claim-focused queries, and improves question answering across five datasets with limited added cost.

  • Retrieval can introduce irrelevant information when LLMs already know the answer, while existing alternatives increase computation through fine-tuning or multiple LLM inferences.This motivates selectively retrieving only for questions exceeding the LLM’s intrinsic knowledge.Source: §1 · p. 1
  • The proxy-model approach is motivated by evidence that smaller and larger language models share knowledge on questions the LLM understands, with larger gaps on questions it does not.The paper’s preliminary analysis supports using the smaller model as a retrieval heuristic.Source: §1 · p. 2
  • SlimPLM uses a smaller proxy model to generate heuristic answers that guide retrieval necessity judgment and identify missing knowledge for the LLM.High-quality heuristic answers support direct generation, while lower-quality answers trigger retrieval of missing information.Source: §1 · p. 2
  • SlimPLM rewrites the heuristic answer into multiple claim-specific queries and filters queries that do not require retrieval.The contribution combines retrieval necessity judgment with claim-based filtering to improve retrieval relevance.Source: §1 · p. 2

2 Related Work

Prior RAG systems retrieve external knowledge before LLM generation, but deciding when and what to retrieve remains challenging. SlimPLM differs by using a smaller LLM’s answer to judge retrieval necessity and formulate more targeted queries.

2.1 Retrieval-Augmented Generation (RAG)

  • 2.1 Retrieval-Augmented Generation (RAG): Standard RAG retrieves documents, optionally rewrites queries and filters results, before an LLM generates the final answer.RAG components commonly include a query rewriter, retriever, filter or reranker, and LLM reader.Source: §2.1 · p. 2

2.2 Retrieval Necessity Judgment

  • 2.2 Retrieval Necessity Judgment: Existing methods fine-tune LLMs, inspect logits, use iterative prompting, or assess question complexity to decide whether retrieval is needed.These approaches include substantial computation, confidence thresholds, thought-action iterations, or prior-question matching.Source: §2.2 · p. 3
  • 2.2 Retrieval Necessity Judgment: SlimPLM analyzes an answer from a smaller LLM to judge retrieval necessity without increasing LLM inference times.The method is presented as distinct from approaches based on the LLM’s own generated answer or logits.Source: §2.2 · p. 3

2.3 Query Formulation

  • 2.3 Query Formulation: SlimPLM analyzes the smaller LLM’s answer to formulate queries that reflect the knowledge required by the question.This contrasts with generating queries from the user question, prior content, masked tokens, or separately rewritten text.Source: §2.3 · pp. 3–4

3 Methodology

SlimPLM uses a smaller proxy model to decide whether retrieval is needed and to identify which knowledge to retrieve. Its framework combines heuristic-answer judgment, claim-based query rewriting, and filtering so retrieval targets knowledge missing from the LLM.

3 Methodology

  • SlimPLM uses a smaller proxy model as a plug-in to determine whether questions require retrieval and to guide relevant knowledge acquisition.The method is designed for retrieval-augmented generation without additional training requirements for the target LLM.Source: §3 · p. 4, §3.2 · p. 4

3.1 Problem Formulation & Framework

  • The framework represents retrieval as generating refined queries, retrieving their relevant documents, and supplying the results to the LLM, with empty results reducing to ordinary generation.The proxy model produces a heuristic answer that supports both retrieval-necessity judgment and query rewriting.Source: §3.1 · p. 4

3.2 Retrieval Necessity Judgment

  • A judgment model uses the user question and heuristic answer to classify retrieval as known or unknown, with labels derived from heuristic-answer quality exceeding or falling below a threshold.Training labels use matching ratios between heuristic answers and multiple short ground-truth answers, and the resulting model is reported to generalize to long-form QA datasets.Source: §3.2 · pp. 4–5

3.3 Retrieval Target Determination

  • SlimPLM decomposes heuristic answers into claims, rewrites both claims and user inputs into queries, and retrieves only rewritten queries classified as requiring external knowledge.Reusing the judgment model at the claim level yields a retrieved set intended to contain only knowledge missing from the LLM.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate SlimPLM across five QA benchmarks using two open-source LLMs, comparing retrieval strategies, component ablations, proxy models, knowledge alignment, and computational cost. SlimPLM generally achieves superior or competitive performance while using substantially lower additional inference cost.

4.1 Datasets

  • The experiments sample 400 questions from each dataset and evaluate long-form answers with ROUGE, short answers with Exact Match, and optional-answer datasets with Hit@1.The datasets are Natural Questions, TriviaQA, ASQA, MuSiQue, and ELI5.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.5 Experimental Results

  • SlimPLM achieves superior or competitive performance across five QA benchmarks, demonstrating effective end-to-end retrieval decisions for LLM question answering.The evaluation uses Table 1 to compare SlimPLM with retrieval and non-retrieval baselines.Source: Table 1 · p. 6, §4.5 · p. 7
  • Compared with LLM-triggered retrieval methods, SlimPLM performs better while requiring only one LLM inference, reducing inference cost.The comparison includes Self-Eval, Self-Ask, and FLARE-style approaches.Source: §4.5 · p. 7
  • Retrieval improves performance on most datasets but not uniformly: approximately 66.4% of ELI5 samples improve with retrieval, while other cases show negative effects.This motivates judging retrieval necessity rather than retrieving for every question.Source: §4.5 · p. 7, Figure 2 · p. 7

4.6 Further Analysis

  • Removing query rewriting, retrieval necessity judgment, or claim-based filtering respectively reduces retrieval relevance, risks misleading the LLM with unnecessary retrieval, or worsens search results.Table 2 defines QR, QJ, and QF as query rewriting, question-level retrieval necessity judgment, and claim-based query filtering.Source: §4.6 · p. 7, Table 2 · p. 8
  • Proxy-model analyses show knowledge alignment between smaller and larger LLMs, with 82.19% overlap on ASQA for samples where the 7B model’s EM exceeds 0.5, while added component costs remain only one-quarter to one-third of a single LLM inference.Llama2-7B-Chat performs best as proxy on most datasets, and its gains align more strongly with Llama2-70B-Chat than Qwen-72B-Chat.Source: §4.6 · pp. 8–9, Table 3 · p. 8, Table 4 · p. 9

5 Conclusion

SlimPLM introduces a proxy-model-based RAG paradigm that determines when and what to retrieve, improving end-to-end question answering with little additional computational cost.

  • SlimPLM uses a smaller LLM as a proxy to support query rewriting, retrieval-necessity judgment, and claim-based query filtering.Source: §5 · p. 9
  • Experiments across various datasets show marked improvement in end-to-end LLM question-answering performance, reaching or exceeding state-of-the-art results.Source: §5 · p. 9
  • The reported enhancement requires little additional computational cost.Source: §5 · p. 9

Limitations

SlimPLM is difficult to use when nearly all questions either fall outside the LLM’s pre-training knowledge or require no external knowledge, and its proxy-based pipeline has recognized capability and complexity limitations.

  • SlimPLM is challenging to use when almost all questions are outside the LLM’s pre-training corpus or almost all questions require no external knowledge.In these settings, full retrieval or no retrieval may be more suitable.Source: Limitations · p. 9
  • Heuristic answers cannot fully reflect the LLM’s true knowledge capability because proxy models and LLMs have a knowledge gap.Source: Limitations · p. 9
  • The current pipeline uses separate proxy, query-rewriting, and retrieval-necessity models, creating complexity that could be reduced by a unified generative framework.Source: Limitations · p. 9

A SlimPLM Implementation Details

SlimPLM’s implementation fine-tunes separate models for query rewriting and retrieval-necessity judgment using data from multiple question-answering datasets, with task-specific balancing and prompting.

B Baseline Implementation Details

Baseline implementation details constrain iterative reasoning, document special prompting choices, and identify an evaluation omission for Self-Ask on ELI5.

  • Baseline methods with multiple reasoning rounds are capped at three iterations because three rounds solve most dataset problems and resources are limited.Source: B Baseline Implementation Details · p. 14
  • Self-Ask results on ELI5 are omitted because its short-answer output does not satisfy the dataset’s long-text annotation setting.Source: B Baseline Implementation Details · pp. 14–15
  • Table 5 reports counts of unknown, known, and dropped samples used for the retrieval-necessity judgment model.Source: Table 5 · p. 15
  • Table 6 presents the RAG prompt used for FLARE.Source: Table 6 · p. 15
  • The baseline appendix documents special prompts for FLARE and Self-Ask, with Self-Ask using the LLM as a reader to extract concise intermediate answers from retrieved documents.Source: B Baseline Implementation Details · p. 15

C Case Study

The case study examines adverse effects of search when retrieved references are misleading or incomplete. It also documents prompts used for query rewriting, retrieval-augmented generation, and Self-Ask tasks.

C Case Study

  • Misleading references can provide the LLM with incorrect information, while incomplete references can cause it to overlook alternative answers.These two scenarios are identified as the main ways searching can adversely affect answers.Source: C Case Study · p. 15
  • Table 7 presents cases involving both misleading and incomplete retrieved references.Source: C Case Study · p. 15, Table 7 · p. 16
  • The study specifies a GPT-4 auxiliary annotation prompt for the query rewriting model.Source: Table 8 · p. 16
  • The appendix provides retrieval-augmented generation prompts for different tasks, including Self-Ask.Source: Table 9 · p. 17, Table 10 · p. 17
Loading 2402.12052v3…