PaperlayerView exact revision on arXiv

Source-linked AI summary

Internet-augmented language models through few-shot prompting for open-domain question answering

Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, Nikolai Grigorev

arXiv:2203.05115v2cs.CLcs.LG

TL;DR

Few-shot LSLMs can hallucinate or rely on incomplete and outdated knowledge, motivating methods that ground answers in current external evidence. The paper uses Google Search with few-shot prompting and no additional learned parameters, finding improved open-domain QA performance and benefits from more inference-time computation.

  • Problem

    Few-shot LSLMs remain susceptible to hallucinations and incomplete or outdated knowledge, creating a need for flexible grounding in factual, current information.

  • Method

    The method retrieves web documents with Google Search and uses few-shot prompting to condition pre-trained LMs on the retrieved evidence without fine-tuning or extra learned parameters.

  • Results

    Across open-domain QA tasks, web-conditioned models improve over closed-book few-shot models, including 15%-30% relative gains on language-generation tasks.

  • Takeaways & Limitations

    More targeted prompting and increased inference-time compute can provide gains beyond scaling model parameters and may reduce reliance on pursuing the biggest model.

  • Takeaways & Limitations

    The experiments remain behind in-domain fine-tuned models, and retrieval improved reasoning tasks only for the largest considered models.

Abstract

from arXiv · show

In this work, we aim to capitalize on the unique few-shot capabilities of large-scale language models (LSLMs) to overcome some of their challenges with respect to grounding to factual and up-to-date information. Motivated by semi-parametric language models (LMs), which ground their decisions in external retrieved evidence, we use few-shot prompting to learn to condition LMs on information returned from the web using Google Search, a broad and constantly updated knowledge source. Our approach does not involve fine-tuning or learning additional parameters, thus making it applicable to any LM, offering therefore a strong baseline. Indeed, we find that LMs conditioned on the web surpass performance of closed-book models of similar, or even larger, model sizes in open-domain question answering. Finally, we find that increasing the inference-time compute of models, achieved via using multiple retrieved evidences to generate multiple answers followed by a reranking stage that uses scores generated by the same LMs, leads to better performance and alleviates lower performance of smaller few-shot LMs. All in all, our findings suggest that it might be beneficial to slow down the race towards the biggest model and instead shift attention towards finding more effective ways to use models, including but not limited to, better prompting or increasing inference-time compute.

1 Introduction

The paper addresses factuality and currency problems in few-shot LSLMs by conditioning them on Internet evidence through prompting. Across open-domain QA settings, this lightweight approach improves performance and motivates greater attention to prompting and inference-time computation.

  • Motivation: Few-shot LSLMs can adapt to many tasks from only a handful of examples, but they remain vulnerable to hallucinations and outdated or incomplete knowledge.The paper frames factual, up-to-date grounding as an important unresolved need.
  • Approach: The proposed system retrieves relevant Internet documents with Google Search and prompts the model to answer while conditioning on that evidence.It uses external retrieval without fine-tuning or additional learned parameters.
  • Results: 15%-30% relative gains over closed-book few-shot prompting were observed on language-generation tasks.The method also produced smaller gains on complex multi-hop questions despite higher retrieval errors.
  • Implications: Conditioning on the web can partly compensate for smaller model size, supporting lightweight use of existing models rather than relying only on parameter scaling.The approach is presented as applicable to virtually any pre-trained LM without fine-tuning or extra learnable parameters.
  • Implications: Increasing inference-time compute through targeted few-shot use and multiple evidence-conditioned answers further supports shifting attention beyond simply building the biggest model.The conclusion emphasizes more effective use of models’ few-shot capabilities alongside additional inference-time computation.

2 Related Work

Prior work shows that external knowledge can make language models more adaptable and grounded, while web-based retrieval broadens coverage beyond curated benchmarks. This paper positions itself as a lightweight, parameter-free few-shot alternative to more intensive approaches.

  • Semi-parametric language models: Semi-parametric language models extend parametric models with external knowledge, helping address domain or temporal shifts and reduce hallucinations.External knowledge can be updated without changing the model’s stored parameters.
  • Web as knowledge source: Web-based question answering expands beyond Wikipedia-centered benchmarks toward more varied interactions and broader knowledge coverage.Commercial search engines have been used to support more diverse fact-checking and web-based datasets.
  • Positioning: This work emphasizes a lightweight approach that uses few-shot prompting without introducing learnable parameters, rather than fine-tuning or reinforcement learning.The authors present it as a generally applicable baseline for understanding the strengths and weaknesses of simple prompting.

3 Few-shot prompting for Internet-augmented Language Models

The method retrieves web evidence, compresses it into question-relevant passages, and uses few-shot prompts to condition answers. It then increases inference-time computation by generating candidates from many passages and reranking them with LM-based scores.

  • 3 Few-shot prompting for Internet-augmented Language Models: The approach has three stages: retrieve relevant documents, prompt the LM with evidence, and rerank multiple candidate answers.It is designed to condition pre-trained LMs on factual and potentially new information without parameter updates.
  • 3.1 Retrieve: Google Search for document retrieval: Google Search retrieves the top 20 URLs for each question, whose HTML is parsed into documents for open-ended web access.Questions are issued verbatim, while the paper notes that complex multi-hop questions expose limits of this simple retrieval strategy.
  • 3.1 Retrieve: Google Search for document retrieval: Long documents are split into six-sentence paragraphs and ranked by TF-IDF cosine similarity to select evidence that fits the model’s input length.The documents average 2,056 words, motivating shorter evidence excerpts.
  • 3.1 Retrieve: Google Search for document retrieval: Google Search provides diverse, continually expanding web content and achieves practical retrieval performance comparable to strong Wikipedia-based retrievers.The paper presents the web as an alternative to static, curated Wikipedia snapshots.
  • 3.2 Prompt: Few-shot prompting for conditioning on evidence: Few-shot prompting extends conventional question-answer examples by inserting an evidence paragraph before each question and answer, using k = 15 examples.The prompt format is Evidence: ..., Question: ..., Answer: ....
  • 3.2 Prompt: Few-shot prompting for conditioning on evidence: Placing evidence before the question consistently performed better than swapping their order across datasets.The authors hypothesize that greater question-answer distance makes it harder for LMs to use longer contexts.
  • 3.3 Rerank: Increasing inference-time compute via answer reranking: Inference-time computation is increased by sampling multiple answers from retrieved evidence rather than only scaling training-time model parameters.The method uses multiple evidence-conditioned candidates and a later scoring stage.
  • 3.3 Rerank: Increasing inference-time compute via answer reranking: Candidate answers are ranked using direct inference, noisy-channel inference, or Product-of-Experts probability factorizations, with LM-derived conditional probabilities.TF-IDF similarity supplies the retrieval probability component, while the remaining probabilities are computed from few-shot-prompted LMs.

4 Experimental Setup

The experiments cover single- and multi-hop QA and fact-checking across generation and classification tasks, using several model sizes and open-book, oracle-evidence, and closed-book comparisons. Evaluation combines task performance with retrieval-oriented measures and controls for unequal candidate-pool sizes.

  • Datasets: The benchmark suite includes NQ, HOTPOTQA, STRATEGYQA, and FEVER, spanning single-hop and multi-hop generation, classification, and fact-checking.Dataset-specific 15-shot prompts use evidence, questions, and answers from the gold documents.
  • Evaluation metrics: Generation tasks are evaluated with exact match, classification tasks with accuracy, and retrieval with answer recall or normalized word overlap.The retrieval score measures whether the conditioning evidence contains or overlaps with the answer signal.
  • Language Models: The study uses 280B-parameter GOPHER alongside 44M, 117M, 400M, 1B, and 7B models from the same family.Input sequence lengths differ across model sizes.
  • Open-book models: OBGoogle generates four candidate answers for each of 50 retrieved paragraphs and selects the candidate with the highest predefined score.This creates a pool of 200 answers per question.
  • Open-book models: Scoring variants include question-answer probability, noisy-channel inference, and Product-of-Experts combinations.The variants correspond to alternative factorizations introduced for answer reranking.
  • Baselines and controls: OBGold conditions on dataset-provided gold evidence, while CB uses only question-answer examples and no external evidence.These provide oracle-evidence and conventional closed-book reference conditions.
  • Baselines and controls: For fair comparison, CB and OBGold also sample 200 answers and retain the highest-probability answer.This controls for OBGoogle’s larger candidate pool generated across retrieved paragraphs.

5 Results

Conditioning Gopher-280B on Google Search evidence improves question-answering performance across all four datasets, with especially strong gains on generation tasks. Retrieval quality, reranking, model scale, and inference-time compute all shape these gains, while updated or complex questions remain challenging.

  • 5.1 Conditioning a large-scale language model on Google search results: Conditioning Gopher-280B on Google Search results improves performance on all four datasets over the same model in the closed-book setting.The open- and closed-book systems use the same underlying language model.
  • 5.1 Conditioning a large-scale language model on Google search results: 30% relative improvement is reached on NQ for generation tasks, where answers are frequently present in the conditioning evidence.Generated answers occur in the evidence in 89.4% of NQ cases and 70.5% of HOTPOTQA cases.
  • 5.1 Conditioning a large-scale language model on Google search results: Google Search outperforms DPR on NQ recall@50, 84%, and is marginally below MDR on HOTPOTQA recall@20, 50.1% versus 52.1%.The generic search system performs zero-shot across tasks and knowledge corpora, unlike the dataset-specific Wikipedia retrievers.
  • 5.2 Ablations: Reranking with factorizations that score more than answer probability improves performance across all datasets, including when smaller prompted models derive the extra scores.The probabilities p(a|q, p) and p(q|p, a) are especially informative, while the RAG-inspired factorization performs worst among the considered factorizations.
  • 5.4 Increasing inference-time compute: Considering more than one retrieved paragraph improves NQ exact match, and five paragraphs with the 7B model surpass the closed-book Gopher-280B model.The comparison is presented as evidence that increased inference-time computation through retrieval and reranking can compensate for much larger parameter counts.

6 Discussion

The method turns pre-trained language models into retrieval-augmented systems using web evidence and few-shot prompting, with further gains from inference-time reranking. The discussion also identifies limits involving fine-tuned-model performance, multi-hop retrieval, safety, and reproducibility.

  • Discussion: The method combines few-shot prompting with Google Search to ground pre-trained language models in broad, current web knowledge.It requires neither fine-tuning nor additional learnable parameters.
  • Discussion: Performance improved across model sizes, with smaller open-book models often surpassing larger few-shot closed-book models.Additional gains came from generating multiple answers from retrieved evidence and reranking them with scores computed by the same models.
  • Discussion: Inference-time interventions may redirect attention from parameter scaling toward more effective use of existing models.The paper specifically highlights targeted few-shot use and increased inference-time compute.
  • Limitations: The approach still trails in-domain fine-tuned models, and retrieval improved reasoning tasks only for the largest considered models.The experiments also cover only a small fraction of simple user interactions where factuality matters.
  • Limitations: Search-result deterioration on multi-hop questions motivates learning-to-search methods that decompose complex queries into simpler sub-queries.The paper identifies black-box search engines and their discrete word-level bottleneck as challenges.
  • Limitations: Using the whole web introduces risks including misinformation and harmful content, while changing search results can reduce reproducibility over time.The authors argue that the benefits outweigh these downsides when the system is used responsibly.

A.1 Illustration of method

Figure 4 schematically represents the method introduced in Section 3.

  • The figure is a schematic representation of the method presented in Section 3.

A.2 Using LSLMs-derived scores for passage reranking

Figure 5 compares NQ answer recall for LSLMs-score reranking of Google Search passages with DPR retrieval on Wikipedia.

  • The figure compares NQ answer recall for LSLMs-derived reranking scores against a DPR retriever using Wikipedia.

A.3 QA Prompts

The appendix provides few-shot QA prompts containing evidence, questions, and answers across NQ and HotPotQA examples. The examples span factual lookup, classification, and multi-hop question answering.

  • A.3 QA Prompts: The prompt appendix explicitly distinguishes open-book prompts with evidence from closed-book prompts that omit the Evidence part.
  • A.3.1 NQ: NQ examples pair questions with short factual answers, including football rankings, song performers, filming locations, and parliamentary nominations.
  • A.3.1 NQ: Additional NQ examples cover McChicken ingredients, Life of Pi, malware, Mum, chess, the Raiders stadium, Starbucks, Wipro, and The Next Iron Chef.
  • A.3.1 NQ: The remaining NQ prompts include evidence and answers about Mum, chess, the Raiders stadium, Starbucks, Wipro, The Next Iron Chef, and plant stems.
  • A.3.2 HotPotQA: HotPotQA examples include a multi-step question about the birth date of the writer of Seesaw, whose evidence identifies Michael Bennett as the writer.

A.4 Prompts for calculating scores

The appendix provides prompts and examples for calculating the scores used in different factorizations. It illustrates evidence-question-answer formatting across diverse open-domain questions and shows variants that omit the answer or reorder the evidence and question.

  • A.4.1 Calculating p(q | ai, pi): The score-calculation prompts use evidence, a question, and an answer to demonstrate how retrieved information supports answering diverse questions.Examples cover music, sports, film, politics, food, television, geography, business, science, and computing.
  • A.4.1 Calculating p(q | ai, pi): The prompt examples include questions about food, literature, malware, television, chess, and voice actors, with answers extracted from corresponding evidence passages.The examples identify a McChicken ingredient, Richard Parker, malware, Lulu and the Lampshades, Tbilisi, Georgia, and T.J. Miller.
  • A.4.1 Calculating p(q | ai, pi): The appendix also demonstrates questions about a song, a television theme, a chess tournament, a film location, and a nominated Rajya Sabha member.The examples answer with the Outfield, Lulu and the Lampshades, Tbilisi, Georgia, Welch, West Virginia, and Mary Kom.
  • A.4.1 Calculating p(q | ai, pi): Additional examples cover the Raiders stadium location, Starbucks’s public offering date, Wipro’s CEO, The Next Iron Chef, and plant-stem functions.The answers are Paradise, Nevada; June 1992; Abid Ali Neemuchwala; Zakarian; and production of new living tissue.
  • A.4.1 Calculating p(q | ai, pi): The appendix includes evidence about The Outfield song “Your Love,” the McChicken, and plant stems alongside questions whose answers are derived from those passages.These examples connect descriptive evidence to short answer outputs such as the Outfield, a breaded chicken patty, and production of new living tissue.
  • A.4.2 Calculating p(q | pi): One prompt variant omits the answer field while retaining the evidence-question structure.This is presented as the alternative calculation of p(q | pi).
  • A.4.3 Calculating p(pi | q): The examples pair retrieved evidence with answers for factual questions about songs, football rankings, films, and other entities.Illustrations include The Outfield, Germany in FIFA rankings, and Welch, West Virginia for The Glass Castle.
Loading 2203.05115v2…