Source-linked AI summary
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, Matei Zaharia
TL;DR
Knowledge-intensive tasks often expose the limits of simple retrieve-then-read systems and unreliable language-model knowledge. DSP composes weakly supervised demonstrations, task-aware search, and grounded prediction between frozen LMs and RMs, with early evaluations reporting state-of-the-art in-context learning results across three task settings.
Problem
Simple retrieve-then-read pipelines can fail when search does not retrieve passages directly answering a question, while LM knowledge can be unreliable.
Method
DSP passes natural-language text and scores between frozen LMs and RMs through composable DEMONSTRATE, SEARCH, and PREDICT programs.
Results
DSP programs established state-of-the-art in-context learning results across open-domain, multi-hop, and conversational question answering, with 37–120% relative gains over corresponding vanilla LMs.
Takeaways & Limitations
DSP makes task-aware retrieval-augmented strategies expressible as short composable programs and supports automatic annotation of complex pipelines from end-task labels.
Takeaways & Limitations
The evaluation is preliminary and assumes human-labeled training data do not provide labels for useful intermediate transformations such as individual-hop queries.
Abstract
from arXiv · showhide
Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM). Existing work has combined these in simple "retrieve-then-read" pipelines in which the RM retrieves passages that are inserted into the LM prompt. To begin to fully realize the potential of frozen LMs and RMs, we propose Demonstrate-Search-Predict (DSP), a framework that relies on passing natural language texts in sophisticated pipelines between an LM and an RM. DSP can express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions, systematically breaking down problems into small transformations that the LM and RM can handle more reliably. We have written novel DSP programs for answering questions in open-domain, multi-hop, and conversational settings, establishing in early evaluations new state-of-the-art in-context learning results and delivering 37-120%, 8-39%, and 80-290% relative gains against the vanilla LM (GPT-3.5), a standard retrieve-then-read pipeline, and a contemporaneous self-ask pipeline, respectively. We release DSP at https://github.com/stanfordnlp/dsp
1. Introduction
DSP extends retrieval-augmented in-context learning beyond rigid retrieve-then-read pipelines by composing task-aware demonstrations, searches, and grounded predictions between frozen language and retrieval models. The framework addresses unreliable LM knowledge and failed direct retrieval by decomposing complex questions into manageable transformations, achieving strong early results across knowledge-intensive tasks.
- Multi-hop example: A multi-hop DSP program decomposes a complex question, retrieves supporting information over two hops, and uses the retrieved passages to produce the final answer.The toy program includes a demonstration stage before iterative search and prediction.
- Motivation: A rigid retrieve-then-read strategy fails when the retrieval model cannot find passages that directly answer the question.The motivating example asks about the castle inherited by David Gregory, requiring information that simple search does not directly retrieve.
- DSP framework: DSP composes DEMONSTRATE, SEARCH, and PREDICT functions that pass natural-language text and scores between frozen LMs and RMs.DEMONSTRATE bootstraps examples, SEARCH gathers corpus information, and PREDICT generates grounded outputs.
- DSP framework: DSP uses weak supervision from labeled question–answer pairs to annotate intermediate queries and passages without hand-labeled transformation examples.The SEARCH and PREDICT transformations are programmatically annotated through the DEMONSTRATE stage.
- Evaluation: 37–120% relative gains were reported across open-domain, multi-hop, and conversational question answering against corresponding vanilla LMs.The evaluations used frozen GPT-3.5 and ColBERTv2 without fine-tuning.
- Contributions: The paper argues that composable, task-aware strategies can replace simple task-agnostic pipelines and establish state-of-the-art in-context learning results for three knowledge-intensive tasks.The claimed contribution is that these strategies are expressible as short programs and can automatically annotate complex pipelines from end-task labels.
2. DEMONSTRATE–SEARCH–PREDICT
DSP defines communication between frozen language and retrieval models through composable DEMONSTRATE, SEARCH, and PREDICT transformations. These programs use text passed between models to build demonstrations, retrieve information, and generate grounded answers for complex tasks.
- Framework: The LM generates final answers, intermediate search queries, and exemplar queries for demonstrations.This extends LM use beyond producing only the final response.
- Framework: DSP programs coordinate frozen language and retrieval models through composable transformations that pass text between them.Transformations operate on Examples, adding or modifying fields without backpropagation.
- DEMONSTRATE: DEMONSTRATE prepares task-specific examples by selecting training data and bootstrapping fields that illustrate desired LM behaviors.DSP can annotate intermediate transformations using only end-task labels when zero-shot processing succeeds on some examples.
- Modularity: DSP's modularity supports strategy changes, updated training examples, and updated retrieval corpora without hand-labeling intermediate transformations.DEMONSTRATE can automatically populate intermediate demonstration fields when developers change the domain or program strategy.
- SEARCH: SEARCH supports multi-step information gathering by composing query generation, passage retrieval, intermediate summarization, and research-path planning.The retrieval model supplies passages for LM-generated queries, while the LM can incrementally plan subsequent hops.
- PREDICT: DSP generalizes self-consistency by sampling multiple pipelines of transformations and comparing multiple candidate predictions directly.The framework can sample different program paths and select or compare top-k candidates, including through LM-generated comparisons.
3. Evaluation
DSP programs are evaluated across open-domain, multi-hop, and conversational question answering using frozen GPT-3.5 and retrieval-based systems. The task-aware programs substantially outperform vanilla, retrieve-then-read, and self-ask baselines in the reported development results.
- DSP evaluation covers open-domain QA, multi-hop QA, and conversational QA, all treated as open-domain tasks without answer-bearing context supplied directly.
- Evaluation reports validation accuracy across Open-SQuAD, HotPotQA, and QReCC, using one development dataset for each task.
- The experiments use frozen GPT-3.5, with ColBERTv2 as the retrieval model and up to 16 randomly sampled training examples per DSP program.
- 126% relative EM gain: on the reported open-domain QA evaluation, task-aware DSP achieves 36.6% EM over the vanilla LM baseline, plus 8% EM and 6% F1 gains over retrieve-then-read.
- 82%, 39%, and 80% relative EM gains: on HotPotQA, task-aware DSP exceeds vanilla LM, retrieve-then-read, and self-ask, respectively.
- 51.4% EM: the task-aware DSP program outperforms the compared Wikipedia-API-search approaches by large margins.
4. Conclusion
The paper frames DSP as a natural-language framework for composing pretrained components into deliberate programs for knowledge-intensive tasks. Its central contribution is opening a broad space of possibilities for in-context learning beyond particular performance gains.
- Tensor-based model composition requires extensive domain expertise, even when combining pretrained components into larger systems.
- DSP uses natural-language instructions and text operations as the core interface between pretrained models in complex systems.
- The authors implement DSP as a Python library and apply it to Open-SQuAD, HotPotQA, and QReCC programs.
- DSP programs deliver substantial gains over previous in-context learning approaches across the evaluated knowledge-intensive tasks.
- The authors identify DSP’s central contribution as revealing a large space of conceptual possibilities for in-context learning beyond any single performance number.