PaperlayerView exact revision on arXiv

Source-linked AI summary

DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines

Arnav Singhvi, Manish Shetty, Shangyin Tan, Christopher Potts, Koushik Sen, Matei Zaharia, Omar Khattab

arXiv:2312.13382v2cs.CLcs.AIcs.PL

TL;DR

LM pipelines need a systematic way to enforce computational constraints beyond heuristic prompt engineering. The paper introduces LM Assertions in DSPy, combining hard and soft constraints with runtime self-refinement and compile-time bootstrapping. Across four case studies, these methods improve constraint compliance and downstream task quality, while TweetGen reveals that conflicting constraints can reduce engagement.

  • Problem

    Existing LM pipeline frameworks cannot directly express arbitrary computational constraints or systematically teach models to follow them.

    Source: §1 · p. 1

  • Method

    The paper integrates hard Assertions and soft Suggestions into DSPy, adding backtracking self-refinement, assertion-driven example bootstrapping, and counterexample bootstrapping.

    Source: Assertion-driven backtracking · p. 2, §3 · p. 4

  • Results

    Across four case studies, LM Assertions improve constraint compliance and downstream task performance, including 37.6% to 98.8% well-formatted QuizGen outputs and 30.5% to 87.2% valid quiz questions.

    Source: Assertion-driven backtracking · p. 2

  • Takeaways & Limitations

    LM Assertions provide a programmable abstraction for expressing constraints and improving LM pipelines through runtime retries and compilation-time examples.

    Source: §1 · p. 1

    On TweetGen, conflicting constraints and imprecise repair instructions were associated with shorter, less engaging tweets when compiling with assertions.

    Source: §5.4.3 · p. 8

Abstract

from arXiv · show

Chaining language model (LM) calls as composable modules is fueling a new way of programming, but ensuring LMs adhere to important constraints requires heuristic "prompt engineering". We introduce LM Assertions, a programming construct for expressing computational constraints that LMs should satisfy. We integrate our constructs into the recent DSPy programming model for LMs, and present new strategies that allow DSPy to compile programs with LM Assertions into more reliable and accurate systems. We also propose strategies to use assertions at inference time for automatic self-refinement with LMs. We report on four diverse case studies for text generation and find that LM Assertions improve not only compliance with imposed rules but also downstream task performance, passing constraints up to 164% more often and generating up to 37% more higher-quality responses. Our reference implementation of LM Assertions is integrated into DSPy at https://github.com/stanfordnlp/dspy

1. Introduction

LM Assertions address the difficulty of enforcing computational constraints in composable LM pipelines without relying on labor-intensive prompt engineering. Integrated with DSPy, they support runtime self-refinement and compile-time optimizations that improve constraint compliance and downstream task performance.

1. Introduction

  • LM Assertions express user-specified properties on LM outputs as computational constraints within a pipeline.They function as programmatic conditions or rules that can enforce desired behavior during execution.Source: §1 · p. 1
  • Assertion-driven backtracking retries failed modules using prior outputs and error messages, while example and counterexample bootstrapping improve prompts during compilation.Counterexamples pair erroneous results with fixes so models can avoid recurring mistakes without backtracking.Source: Assertion-driven backtracking · p. 1
  • Hard Assertions halt a pipeline after exhausted retries, whereas soft Suggestions trigger self-refinement but allow execution to continue.The distinction separates non-negotiable requirements from desirable properties.Source: Assertion-driven backtracking · p. 2
  • 98.8% of QuizGen outputs were well-formatted in JSON, up from 37.6%, with LM Assertions and assertion-driven backtracking.This result is reported for the QuizGen task.Source: Assertion-driven backtracking · p. 2
  • 73.0% of TweetGen outputs were more engaging, up from 2.0%, with assertion-driven example and counterexample bootstrapping.The paper also reports a maximum increase from 30.5% to 87.2% for generating valid quiz questions when using all assertion-driven optimizations.Source: Assertion-driven backtracking · p. 2

2. Background and Motivation

DSPy provides declarative modules and optimizers for composing LM pipelines, but lacks constructs for arbitrary computational constraints. The motivating multi-hop QA example shows how soft assertions can guide query generation and improve retrieval and answer accuracy.

2.1. The DSPy Programming model

  • DSPy composes declarative LM modules into arbitrary pipelines and automatically optimizes demonstrations or instructions against a target metric.Its modules replace handwritten task-specific prompts with reusable prompting components.Source: §2.1 · p. 3
  • DSPy signatures specify module inputs and outputs, while Predict formats prompts, adds demonstrations, and calls the LM.A module can, for example, consume a question and return an answer.Source: §2.1 · p. 3
  • Before LM Assertions, DSPy lacked constructs for arbitrary computational constraints that could guide output refinement or teach the LM to follow specifications during compilation.Signatures provided only soft behavioral hints.Source: §2.1 · p. 3

2.2. Motivating Example

  • In multi-hop QA, long or repetitive search queries can retrieve irrelevant or redundant context, motivating computationally checkable query constraints.The pipeline generates search queries, retrieves context, and aggregates it into an answer.Source: §2.2 · p. 3
  • Soft Suggestions restrict query length and require dissimilar queries across hops, then backtrack with prior attempts and suggestion messages for self-refinement.The example limits queries to fewer than 100 characters and discourages repeated retrieval.Source: §2.2 · pp. 3–4
  • 6.5%–7.9% higher retriever recall and 3.4%–14.4% higher answer accuracy were reported on HotPotQA after enabling two simple suggestions.The motivating pipeline was evaluated on the HotPotQA dataset.Source: §2.2 · p. 4

3. Semantics of LM Assertions

LM Assertions introduce programmatic constraints that guide LM pipeline execution through retry-based self-refinement. Assert enforces non-negotiable invariants, while Suggest provides non-binding guidance and allows execution to continue after unresolved failures.

3. Semantics of LM Assertions

  • LM Assertions express developer-specified conditions that constrain LM outputs and guide pipeline execution.They are integrated into DSPy as programmatic elements for improving reliability, predictability, and correctness.Source: §3 · p. 4

3.1. Semantics of Assert

  • Assert retries a failed LM call with prior attempts and error feedback, then raises an AssertionError and halts after the maximum retry count.Before the retry limit, the pipeline enters a retry state; reaching the limit transitions it to an error state.Source: §3 · p. 4, §3.1 · p. 4

3.2. Semantics of Suggest

  • Suggest retries failed calls and self-refines outputs, but logs a warning and continues to the next module when the maximum retries are exhausted.Unlike Assert, Suggest represents a desirable rather than mandatory condition.Source: §3 · p. 4, §3.2 · pp. 4–5

3. Semantics of LM Assertions

  • Both constructs continue normally when their condition evaluates to true, while failures enter a special retry state before resolving or terminating according to construct type.The default semantics can be extended with custom semantics for both Assert and Suggest.Source: §3 · p. 4, §3.1 · p. 4, §3.2 · pp. 4–5

4. Assertion-Driven Optimizations

Assertion-driven optimizations use failures and retry traces to refine LM outputs and improve demonstrations. They support runtime backtracking, constraint-aware example selection, and counterexamples that teach models to avoid and repair assertion violations.

4.1. Assertion-Driven Backtracking

  • Assertion-driven backtracking retries failing modules with prompts containing prior outputs and error messages, halting Assert failures but advancing past exhausted Suggest failures.An error handler determines the next instruction and updates the failing module's prompt when retries remain.Source: §4.1 · p. 5

4.2. Assertion-Driven Example Bootstrapping

  • Assertion-driven example bootstrapping applies backtracking to the teacher model so demonstrations satisfy intermediate constraints, not only the final-answer metric.This prevents examples with correct final responses but incorrect intermediate outputs from becoming demonstrations.Source: §4.2 · p. 5

4.3. Counterexample Bootstrapping

  • Counterexample bootstrapping mixes failed traces and fixes into prompts, providing negative demonstrations that help students avoid similar assertion failures.These traces can improve constraint compliance even without assertion-driven backtracking at inference time.Source: §4.3 · p. 5
  • Counterexample bootstrapping alone eliminates student-model backtracking overhead while preserving the ability to generate more assertion-compliant responses.The result is reported for programmer-defined assertions.Source: §4.3 · p. 5

5. Evaluation

The evaluation tests LM Assertions across four HotPotQA-derived tasks, comparing zero-shot, compiled, inference-time, and combined assertion strategies. Assertions improve constraint compliance and often downstream performance, while conflicting suggestions limit gains in some settings.

5.1. Tasks & Metrics

  • The study evaluates MultiHopQA, LongFormQA, TweetGen, and QuizGen using intrinsic constraint-compliance and extrinsic downstream-performance metrics.MultiHopQA requires multi-hop retrieval and answering; LongFormQA requires faithful citations; TweetGen requires concise faithful engagement; QuizGen requires JSON choices and distractors.Source: §5.1 · pp. 5–6

5.2. Dataset and Models

  • Experiments use hard HotPotQA examples with 300 training, 300 validation, and 500 test examples, Wikipedia 2017 abstracts, ColBERTv2 retrieval, and gpt-3.5-turbo.Strategies compare Vanilla and Compile baselines with assertion use during inference, compilation, or both; all pipelines receive complex metric-aware instructions.Source: §5.2 · p. 6, §5.3 · p. 6

5.4.1. H1: SELF-CORRECTION VIA LM ASSERTIONS

  • 32.9% more MultiHopQA test-set Suggestions are passed with Infer w/ Assert than with Vanilla, showing stronger constraint compliance across tasks.The evaluation focuses on intrinsic metrics that measure conformity to LM Assertions.Source: §5.1 · p. 6, §5.4.1 · p. 7
  • 98.8% of QuizGen outputs have correct JSON formatting after adding constraints, backtracking, and self-refinement, versus an initially struggling baseline.The same constrained pipeline also achieves 76.6% correctness for including the right answer among choices.Source: §5.4.1 · p. 7

5.4.2. H2: PERFORMANCE VIA SELF-CORRECTION

  • Defining suggestions raises QuizGen’s overall Validity from 30.5% to 70.5%, while comparing Infer w/ Assert with Vanilla yields moderate-to-large improvements on downstream metrics for MultiHopQA, LongFormQA, and QuizGen.Source: §5.4.2 · p. 7
  • Conflicting suggestions can reduce benefits: TweetGen test-set overall Quality shows no significant increase, and assertion-guided compilation can lower engagement when tweets become too short.The authors attribute this to suggestions overriding one another or being difficult for prompt optimization to satisfy simultaneously.Source: §5.4.2 · pp. 7–8, §5.4.3 · p. 8

5.4.3. H3: COMPILING WITH LM ASSERTIONS

  • 73.0% of TweetGen outputs are Engaging after assertion-guided compilation, compared with 2.0% for baseline few-shot compilation.The compiled strategy also gains 30.7% in overall TweetGen Quality, and other tasks almost always improve on intrinsic and extrinsic metrics.Source: §5.4.3 · p. 8
  • The combined C+Infer w/ Assert strategy makes QuizGen valid 87.2% of the time versus 30.5% for the zero-shot baseline.The combined setting performs best for most intrinsic metrics and also reports a 9.1% MultiHopQA gain and a 47.5% TweetGen increase.Source: §5.4.3 · p. 8

6. Related Work

Prior work uses assertions and guardrails to monitor or constrain model behavior, but LM Assertions target computational constraints across composable LM pipelines. The paper extends this setting with pipeline-level backtracking, example selection, and counterexample-based refinement.

  • Runtime assertions in Java and Python check properties but generally warn programmers or abort execution.Source: §6 · p. 8
  • LM Assertions differ from model assertions by enabling failing pipeline modules to backtrack and self-refine through retries.Source: §6 · p. 8
  • LM Assertions also filter better few-shot examples and generate counterexamples with fixing traces to improve compliance with constraints.Source: §6 · p. 8
  • LMQL and NeMo Guardrails impose computation constraints on single LMs but do not address LM-pipeline optimization opportunities.Source: §6 · p. 8
  • The paper connects its assertion framework to DSPy prompt optimization and iterative self-refinement using error feedback and past generations.Source: §6 · p. 8

7. Conclusion

The paper integrates LM Assertions into DSPy to express computational constraints, define retry semantics, and support self-correction and better prompt compilation. Across four case studies, the approach improves both assertion-specific and downstream task metrics.

  • LM Assertions express arbitrary computational constraints for LMs used as components of larger programs, with runtime retry semantics in DSPy.Source: §7 · p. 9
  • The implementation abstracts self-refinement across arbitrary steps and pipelines while informing DSPy compilation of higher-quality prompts.Source: §7 · p. 9
  • Evaluations across four case studies report substantial gains on intrinsic assertion metrics and extrinsic downstream task metrics.Source: §7 · p. 9
  • The proposed capabilities let DSPy programs autonomously backtrack, self-correct, and compile better few-shot examples.Source: §7 · p. 9

A. Implementation

The implementation adds Python-style assertions and suggestions to DSPy, with configurable error handling and retry-based backtracking. The resulting programs apply these mechanisms to citation-focused long-form QA and structured quiz generation.

A.1. API Design

  • Both assertions and suggestions accept boolean constraints, optional error messages, and optional target modules for backtracking.Source: §A.1 · p. 11

A.2. Error Handlers

  • Assertion and suggestion failures raise distinct errors that composable handlers can reroute, disable, suppress, or log.Source: §A.2 · p. 11
  • Default handlers retry failed modules R times; assertions then raise an error, whereas suggestions are logged without halting the pipeline.Source: §A.2 · p. 11

A.3. Backtracking

  • The Retry meta-module feeds failed predictions and user-defined error messages back into prompts so modules can refine later generations.Source: §A.3 · p. 11

B.1.1. TASK

B.1.3. CONSTRAINTS SPECIFIED

  • Its suggestions require citations every 1–2 sentences and require preceding text to faithfully support the cited context.Source: §B.1.3 · p. 12, Figure 3 · p. 12

B.2.1. TASK

B.2.2. METRICS

QuizGen evaluates formatting, correct-answer inclusion, and distractor plausibility, while its constraints enforce those properties. The subsequent TweetGen task applies analogous answer, engagement, length, and faithfulness requirements.

B.2.2. METRICS

  • QuizGen measures Valid Formatting, Correct Answer Inclusion, and Choices’ Plausibility as intrinsic metrics.Source: §B.2.2 · p. 13
  • JSON formatting is checked by parsing key-value pairs, while correct-answer inclusion is verified against the HotPotQA answer.Source: §B.2.2 · p. 13
  • Distractor plausibility is assessed by a DSPy program using the question, generated choices, and an LM evaluation question.Source: §B.2.2 · p. 13
  • The composite quiz score averages the three intrinsic metrics only when formatting and correct-answer inclusion both hold; otherwise it is 0.Source: §B.2.2 · p. 13

B.2.3. CONSTRAINTS SPECIFIED

  • QuizGen suggestions enforce JSON formatting, correct-answer inclusion, and plausible distractors through Python checks and a DSPy assessment program.Source: §B.2.3 · pp. 13–14

B.3.1. TASK

  • TweetGen generates answers from retrieved context and questions, targeting correct, engaging, context-faithful tweets within the task’s constraints.Source: §B.3.1 · p. 14, Figure 5 · p. 14

B.3.2. METRICS

TweetGen is evaluated with intrinsic constraints covering format, answer inclusion, length, engagement, and faithfulness, plus a composite extrinsic score. These constraints are operationalized through LM Suggestions checked by Python functions or LM-based evaluators.

B.3.2. METRICS

  • TweetGen measures five intrinsic properties: no hashtags, correct answer inclusion, length compliance, engagement, and faithfulness.Source: §B.3.2 · pp. 14–15
  • Its extrinsic metric averages all five intrinsic scores only when the tweet includes the correct answer and satisfies the character limit; otherwise, it returns 0.Source: §B.3.2 · p. 15

B.3.3. CONSTRAINTS SPECIFIED

  • Suggestions enforce the no-hashtag, 280-character, and correct-answer constraints through Python checks such as has_no_hashtags, is_within_length_limit, and has_correct_answer.Source: §B.3.3 · p. 15
  • Engagement and faithfulness are assessed by DSPy programs that ask whether the tweet is engaging and grounded in its retrieved context.Source: §B.3.3 · p. 15, §B.3.2 · p. 15

C. Impact on Using Different LLM Instructions

The appendix compares primitive and complete instructions for TweetGen and QuizGen, showing that generic instructions poorly expose constraints while assertion-driven pipelines remain comparatively instruction-insensitive. In QuizGen, the primitive baseline achieves only 2.8% correct JSON formatting and 2.6% correct-answer inclusion.

C. Impact on Using Different LLM Instructions

C.1. TweetGen

  • TweetGen’s primitive instruction asks for a tweet that effectively answers a question, whereas its complete version additionally specifies engagement, faithfulness, length, and hashtag constraints.Source: §C.1 · p. 15

C.2. QuizGen

  • QuizGen’s primitive instruction requests answer choices, while its complete version requires JSON output containing the correct answer and plausible distractors.Source: §C.2 · p. 15
  • Figure 6 evaluates TweetGen and QuizGen under primitive instructions using the same LM pipeline program except for LM Assertions, with extrinsic metrics highlighted.Source: Figure 6 · p. 16

C.3. Discussion

  • 2.8% of primitive-instruction QuizGen outputs have correct JSON formatting, while 2.6% contain the correct answer under the Vanilla baseline.Source: §C.3 · p. 16
  • Assertion-optimized pipelines perform comparably with primitive and complete instructions, indicating less sensitivity to manual prompt tuning.Source: §C.3 · p. 16

D. Interesting Examples of Correlated Constraints

The examples illustrate conflicts among TweetGen constraints: outputs can satisfy answer correctness and engagement while violating hashtag requirements, or satisfy hashtag requirements while failing engagement. These cases show that assertion configurations expose different trade-offs rather than uniformly satisfying all constraints.

Loading 2312.13382v2…