PaperlayerView exact revision on arXiv

Source-linked AI summary

DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, Christopher Potts

arXiv:2310.03714v1cs.CLcs.AIcs.IRcs.LG

TL;DR

Existing LM pipelines depend on brittle, hand-crafted prompt templates, motivating a more systematic way to build and optimize multi-stage systems. DSPy expresses pipelines as composable declarative modules and compiles them using demonstrations and metrics. In two case studies, compiled DSPy programs improved performance across GPT-3.5 and llama2-13b-chat, while supporting relatively small language models.

  • Problem

    Existing LM pipelines use hand-crafted prompt templates that can be brittle and may not generalize across pipelines, language models, domains, or inputs.

  • Method

    DSPy represents pipelines with declarative signatures and parameterized modules, while teleprompters compile programs using training examples and metrics into optimized prompts or finetunes.

  • Results

    Compiled DSPy compositions raised GPT-3.5 quality from 33% to 82% and from 32% to 46%, and llama2-13b-chat quality from 9% to 47% and from 22% to 41%.

  • Takeaways & Limitations

    DSPy programs can build effective LM systems without hand-crafted prompts and can use much smaller, more efficient language models effectively.

  • Takeaways & Limitations

    DSPy fields are generally free-form strings; optional data types for constraining valid values and handling formatting were still being explored.

Abstract

from arXiv · show

The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks. Unfortunately, existing LM pipelines are typically implemented using hard-coded "prompt templates", i.e. lengthy strings discovered via trial and error. Toward a more systematic approach for developing and optimizing LM pipelines, we introduce DSPy, a programming model that abstracts LM pipelines as text transformation graphs, i.e. imperative computational graphs where LMs are invoked through declarative modules. DSPy modules are parameterized, meaning they can learn (by creating and collecting demonstrations) how to apply compositions of prompting, finetuning, augmentation, and reasoning techniques. We design a compiler that will optimize any DSPy pipeline to maximize a given metric. We conduct two case studies, showing that succinct DSPy programs can express and optimize sophisticated LM pipelines that reason about math word problems, tackle multi-hop retrieval, answer complex questions, and control agent loops. Within minutes of compiling, a few lines of DSPy allow GPT-3.5 and llama2-13b-chat to self-bootstrap pipelines that outperform standard few-shot prompting (generally by over 25% and 65%, respectively) and pipelines with expert-created demonstrations (by up to 5-46% and 16-40%, respectively). On top of that, DSPy programs compiled to open and relatively small LMs like 770M-parameter T5 and llama2-13b-chat are competitive with approaches that rely on expert-written prompt chains for proprietary GPT-3.5. DSPy is available at https://github.com/stanfordnlp/dspy

1 INTRODUCTION

Existing LM pipelines rely on brittle, hand-crafted prompt strings, especially when multiple LM calls must interact. DSPy replaces this workflow with parameterized modules and a compiler that optimizes pipeline invocations and prompts.

  • Prompt sensitivity becomes more difficult in pipelines where multiple LM calls must interact effectively.
  • Hard-coded prompt templates are long strings of instructions and demonstrations developed through manual trial and error.
  • DSPy represents LM pipelines as composable text transformation graphs and automatically generates optimized invocation strategies and prompts.
  • DSPy modules translate prompting techniques into declarative, task-adaptive components that can bootstrap demonstrations for pipeline steps.
  • DSPy evaluations span math word problems and multi-hop question answering, including reasoning, retrieval augmentation, reflection, and agent loops.

2 RELATED WORK

Related work establishes prompting, in-context learning, retrieval, tools, and prompt optimization as foundations for LM pipelines. DSPy extends these ideas toward modular optimization of arbitrary multi-stage programs.

  • In-context learning and prompting elicit increasingly sophisticated behavior from foundation models, including systematic reasoning and weak supervision.
  • LM pipelines increasingly combine language models with retrieval systems, multimodal models, APIs, calculators, and agent toolkits.
  • Prior discrete optimization and reinforcement learning methods generally search for effective prompts for a single logical LM call.
  • DSPy generalizes prompt optimization to arbitrary pipelines by bootstrapping multi-stage demonstrations under constraints.
  • The paper evaluates whether modular DSPy programs can build effective LM systems without hand-crafted prompt strings.

3 THE DSPY PROGRAMMING MODEL

DSPy provides signatures, modules, and teleprompters for expressing LM programs as composable computational graphs. Its compiler optimizes these programs using demonstrations, metrics, and configurable language-model or finetuning strategies.

  • DSPy abstracts language models as text-generation devices and optimizes their use within arbitrary computational graphs.
  • 3.1 NATURAL LANGUAGE SIGNATURES CAN ABSTRACT PROMPTING & FINETUNING: A DSPy signature declares typed input and output fields for a text transformation without specifying how a particular LM should be prompted.
  • 3.1 NATURAL LANGUAGE SIGNATURES CAN ABSTRACT PROMPTING & FINETUNING: Signatures can be compiled into pipeline-adaptive prompts or finetunes by bootstrapping useful demonstrations, while reducing brittle formatting and parsing code.
  • 3.1 NATURAL LANGUAGE SIGNATURES CAN ABSTRACT PROMPTING & FINETUNING: The shorthand question -> answer expresses a complete question-answering program whose field names guide the language model’s task interpretation.
  • 3.2 PARAMETERIZED & TEMPLATED MODULES CAN ABSTRACT PROMPTING TECHNIQUES: DSPy modules such as Predict, ChainOfThought, ProgramOfThought, MultiChainComparison, and ReAct implement prompting techniques as interchangeable functions.
  • 3.2 PARAMETERIZED & TEMPLATED MODULES CAN ABSTRACT PROMPTING TECHNIQUES: DSPy modules are parameterized by the language model, instructions, field prefixes, and demonstrations used for prompting or finetuning.
  • 3. THE DSPY PROGRAMMING MODEL: DSPy programs compose modules into arbitrary define-by-run pipelines, including retrieval-augmented generation systems.
  • 3.3 TELEPROMPTERS CAN AUTOMATE PROMPTING FOR ARBITRARY PIPELINES: A teleprompter takes a program, training set, and metric, then returns an optimized program using a selected optimization strategy.

4 THE DSPY COMPILER

DSPy compiles modular programs by generating and selecting demonstrations, parameters, and control-flow variants to improve pipeline quality or cost. Its teleprompters support staged optimization through bootstrapping, parameter selection or finetuning, and higher-order program changes.

  • Compiler overview: Teleprompters optimize DSPy modules through prompting or finetuning, using a program, training inputs, and a validation metric.They unify these optimization modes within the DSPy compiler.
  • Stage 1: Candidate Generation: Candidate generation recursively identifies predictors and proposes instructions, field descriptions, and demonstrations for their parameters.The paper focuses on demonstrations and uses rejection-sampling-like approaches to bootstrap multi-stage systems.
  • Stage 1: Candidate Generation: BootstrapFewShot simulates a teacher or zero-shot program, records multi-stage traces, and filters them with the program metric to obtain potential labels.This process can generate demonstrations for all signatures in the pipeline.
  • Stage 1: Candidate Generation: DSPy can iteratively bootstrap because language models often find training examples that satisfy the constraints enforced by signatures and metrics.The paper characterizes this search as efficient despite LM unreliability.
  • Stages 2–3: Optimization: Parameter optimization selects among discrete candidate demonstrations or instructions with tuning algorithms, or updates predictor weights through finetuning.Quality is typically optimized with a metric and cross-validation over training or validation data.
  • Stage 3: Higher-Order Program Optimization: Higher-order optimization modifies program control flow, including ensembles that run multiple bootstrapped copies in parallel and reduce their predictions.The example reduction function is majority voting.

5 GOALS OF EVALUATION

The evaluation tests whether concise DSPy modules can replace hand-crafted prompts, adapt across language models, outperform expert-written prompts, and support exploration of complex pipelines. It evaluates these hypotheses across diverse task–program pairs and reports GSM8K results in which compiled pipelines reach 49–88% accuracy from 4–20% baselines.

  • H1: DSPy is evaluated on whether concise modules can replace hand-crafted prompt strings without reducing quality or expressive power.This is the first stated evaluation hypothesis.
  • H2: Parameterizing modules and treating prompting as optimization is hypothesized to improve adaptation across language models and possibly outperform expert-written prompts.The evaluation frames this as a testable hypothesis rather than an established result.
  • H3: DSPy’s modularity is hypothesized to enable more thorough exploration of complex pipelines with useful performance characteristics or nuanced metrics.The hypothesis concerns both pipeline exploration and fit to evaluation criteria.
  • Evaluation design: The evaluation uses diverse task–program pairs to make comparisons depend on the language model, program, and compilation strategy.The stated goal is to replace underspecified LM comparisons with well-defined, reproducible runs.
  • Reported result: 49–88% accuracy is achieved after compiling DSPy modules, compared with 4–20% accuracy for different uncompiled settings on GSM8K.Table 1 covers in-context learning pipelines composed of two to four DSPy modules and teleprompters.

6 CASE STUDY: MATH WORD PROBLEMS

The GSM8K case study evaluates compact DSPy programs for math word problems and multiple compilation strategies. Bootstrapping substantially improves performance across programs and both evaluated language models.

  • Programs and evaluation: The study evaluates vanilla, ChainOfThought, and ThoughtReflection DSPy programs on GSM8K, using 200 training and 300 development question–answer pairs.Final evaluations use the official 1.3k-example test set, measuring accuracy of the final numerical answer.
  • Programs and evaluation: ThoughtReflection samples five reasoning chains and compares them with a generic MultiChainComparison module before producing an answer.The modules are not specific to mathematics or a particular language model.
  • Compilation: LabeledFewShot samples k=8 random demonstrations, while bootstrap generates demonstration chains and optimizes their selection through random search.The latter is intended to self-improve program modules rather than merely insert random examples.
  • Compilation: Compiling generally takes minutes or tens of minutes, with more expensive settings requiring only a few thousand parallel program executions.The reported example uses 10–20 trials over 150–300 validation examples.
  • Results: Bootstrap compilation substantially helps vanilla, including through bootstrap×2, because the resulting prompt lets the LM reason using the answer field before final-value extraction.The evaluation metric extracts the final numerical value from the output.
  • Results: Bootstrap matches or surpasses expert human reasoning chains for ChainOfThought, while ThoughtReflection is a clear winner and ChainOfThought is effective with ensembling.Across both LMs, composing two to four DSPy modules and teleprompters raises accuracy from 4–20% to 49–88%.

7 CASE STUDY: COMPLEX QUESTION ANSWERING

The HotPotQA case study compares DSPy pipelines for open-domain multi-hop question answering, including retrieval, ReAct, and custom iterative retrieval. A simple compiled multihop program performs best overall, while compiled llama2-13b-chat becomes competitive with GPT-3.5.

  • Task and setup: The study uses HotPotQA fullwiki with a ColBERTv2 retriever over the official Wikipedia 2017 abstracts dump, reserving 1,000 validation examples for testing.The training set is split 70%/30%, with 200 examples sampled for training and 300 for development reporting.
  • Programs: The evaluated pipelines include a universal question-to-answer baseline, a CoT RAG program, ReAct, and a custom two-hop BasicMultiHop program.BasicMultiHop generates a search query, retrieves passages, and repeats this process across two hops before answering.
  • Results: CoT RAG can self-bootstrap to increase answer EM, but its direct-question retrieval limits passage recall; ReAct and multihop address this with iterative query generation.The comparison identifies iterative retrieval as the response to failures in retrieving relevant passages directly from the original question.
  • Results: The simple multihop program performs best overall, and bootstrap raises quality relative to few-shot variants for both evaluated language models.The study reports that bootstrap or bootstrap×2 can outperform few-shot multihop and expert human reasoning for ReAct.
  • Results: The compiled multihop T5-Large program scores 39.3% answer EM and 46.0% passage accuracy using 200 labeled and 800 unlabeled questions.It uses a teacher program consisting of an ensemble of two multihop llama2-13b-chat programs.
  • Results: Table 2 reports answer exact match and pair-retrieval accuracy for separately compiled pipelines, with cross-validation using the training set rather than development data.One marked result is evaluated on 50% of the test set because of cost.

8 CONCLUSION

DSPy frames AI-system design as programmable text-transformation graphs whose composable modules and optimizers support systematic, reliable LM pipelines. The authors report effective systems with relatively small LMs and identify broader controlled evaluations as future work.

  • DSPy introduces signatures, modules, and teleprompters for composing and optimizing pipelines of pretrained LMs and other tools.The framework has also been used for tasks including information extraction and low-resource synthetic data generation.
  • DSPy supports rapid development of highly effective systems using relatively small language models across two different case studies.The conclusion specifically reports evidence from two case studies and describes the systems as highly effective.
  • Controlled experimental reporting on additional tasks is left for future work.The authors cite reasonable scope and space constraints while deferring such evaluations.

A ADVANCED SIGNATURES

DSPy advanced signatures let users specify typed, structured transformations for LM modules, including explicit instructions and field roles. These signatures can generate search queries and support optional inputs and typed outputs.

  • Advanced signatures can be expressed as Python classes with explicit transformation instructions and field formats or roles.The example defines context and question inputs and a search-query output.
  • A DSPy search-query signature takes context and an optional question to produce a query for answering complex questions.The example uses a natural-language instruction and typed input/output fields.
  • The search-query module can be called with context alone or with both context and a supplied question.The examples show query generation mediated by a generated question and an explicitly provided question.
  • Signatures can use output types such as bool, int, float, list, or dict instead of the default free-form string.The text presents typed outputs as a work-in-progress feature.

B COMPARISON WITH EXISTING LIBRARIES LIKE LANGCHAIN AND LLAMAINDEX

DSPy differs from LangChain and LlamaIndex by targeting automatic prompt optimization for new LM computational graphs rather than mainly providing pre-packaged application components. The comparison highlights the burden of manually engineered prompt strings in existing libraries.

  • LangChain and LlamaIndex primarily provide pre-packaged components, chains, agents, retrieval pipelines, and tool connections for application developers.Their focus contrasts with DSPy’s focus on the fundamental prompt-engineering challenges of building new computational graphs.
  • DSPy introduces signatures, modules, and teleprompters as core composable operators for automatically compiling LM pipelines.Its stated goal is rapid pipeline development through self-improvement rather than manual prompt engineering.
  • 50 LangChain strings exceeded 1000 characters, whereas DSPy contained none, and LangChain included 12 prompts.py and 42 prompt.py files.The informal study was conducted in late September 2023; DSPy had no hand-written prompt demonstrations at that time.
  • The LangChain PAL template averaged 3982 characters, while its database-specific SQL prompts averaged 1058 characters.The passage uses these examples to characterize lengthy, task-specific prompt templates in existing libraries.
  • Automatically bootstrapped, LM- and task-adaptive prompts are presented as more modular than hard-coded prompts for individual database providers.The comparison contrasts expert-written prompts with prompts generated and adapted through DSPy.

C SAMPLE LARGE PROMPTS

The appendix presents long, hand-engineered prompts used for agreement checking, code-based math solving, ReAct, source-grounded QA, SQL querying, and document retrieval or chatbot tasks. A table reports their word and character counts.

  • The appendix lists eight prompt examples spanning text-evidence checking, math word problems, ReAct, source-grounded QA, SQL querying, document retrieval, and IRS chatbots.The examples include prompts from research papers, LangChain, and LlamaIndex.
  • Prompt 1 is an 818-word, 4964-character text-evidence checker for identifying inconsistencies between text and evidence.Its example uses a few-shot reasoning chain for an agreement model.
  • Prompt 2 is a 566-word, 3957-character PAL prompt for solving math word problems by generating code.The displayed example includes Python solution code for arithmetic word problems.
  • Prompt 3 is a 593-word, 3889-character ReAct prompt interleaving Thought, Action, and Observation steps.A separate LangChain zero-shot ReAct example is listed as 101 words and 600 characters.
  • The listed LangChain and LlamaIndex examples include source-grounded QA, MyScale SQL querying, relevant-document retrieval, and IRS chatbot guidance.Their reported sizes range from 129 to 992 words and from 719 to 6197 characters.

D.1 PREDICT

DSPy implements prediction and reasoning as modular LM calls whose parameters can be compiled through demonstration bootstrapping and search. The section also illustrates automatically generated prompts for GSM8K and HotPotQA programs.

  • Predict: Predict modules parameterize the language model, signature, and demonstrations used for each LM call.The implementation selects the appropriate LM, signature, and demonstrations at runtime.
  • ChainOfThought: ChainOfThought extends a module signature with a rationale output field prefixed by step-by-step reasoning instructions.It then delegates execution to a Predict sub-module with the modified signature.
  • Teleprompters: SimplifiedBootstrapFewShot compiles a student by mapping compatible student and teacher Predict modules and constructing a compiled program.The teleprompter uses a metric and training data to bootstrap few-shot demonstrations.
  • Teleprompters: SimplifiedBootstrapFewShotWithRandomSearch evaluates multiple shuffled-training candidates across a configurable number of trials.When no validation set is supplied, the implementation reuses the training set for validation.
  • Teleprompters: SimplifiedBootstrapFewShotWithOptuna selects demonstrations for predictors by optimizing candidate programs over configurable trials.Each trial resets a copy of the student program and chooses demonstrations for its predictors.
  • Automatically Generated Prompts: DSPy automatically generates demonstrations, labels, and selections for GSM8K chain-of-thought and HotPotQA query-generation prompts.The examples include a GSM8K vanilla program compiled with bootstrap×2 and a HotPotQA multi-hop program generating a second-hop query.
Loading 2310.03714v1…