PaperlayerView exact revision on arXiv

Source-linked AI summary

Guiding Large Language Models via Directional Stimulus Prompting

Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, Xifeng Yan

arXiv:2302.11520v4cs.CL

TL;DR

Black-box LLMs can struggle with fine-grained, instance-specific alignment, while directly tuning them is often impractical. DSP uses a small policy model to generate directional stimuli and trains it with supervised fine-tuning and reinforcement learning; experiments report improved performance across summarization, dialogue, and reasoning tasks. The method’s supervised initialization relies on pseudo-stimuli that may not be optimal, motivating reinforcement learning.

  • Problem

    Black-box LLMs may fail to produce outputs with desired instance-specific behaviors, and directly optimizing them is inefficient or infeasible for many users.

  • Method

    DSP trains a small tunable policy model to generate input-specific directional stimuli, using supervised fine-tuning and reinforcement learning while leaving the black-box LLM untuned.

  • Results

    DSP improves black-box LLM performance across summarization, dialogue response generation, and chain-of-thought reasoning tasks using small labeled datasets.

  • Takeaways & Limitations

    Directional stimuli provide fine-grained guidance and enable optimization of a tunable policy model rather than direct optimization of black-box LLMs.

  • Takeaways & Limitations

    Heuristically selected or annotated pseudo-stimuli may not be optimal for guiding LLMs toward desired outputs.

Abstract

from arXiv · show

We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model (e.g., T5) to generate an auxiliary directional stimulus prompt for each input instance. These directional stimulus prompts act as nuanced, instance-specific hints and clues to guide LLMs in generating desired outcomes, such as including specific keywords in the generated summary. Our approach sidesteps the challenges of direct LLM tuning by optimizing the policy model to explore directional stimulus prompts that align LLMs with desired behaviors. The policy model can be optimized through 1) supervised fine-tuning using labeled data and 2) reinforcement learning from offline or online rewards based on the LLM's output. We assess our method across summarization, dialogue response generation, and chain-of-thought reasoning tasks. Our experiments demonstrate that the framework consistently improves LLMs' (e.g., ChatGPT, Codex, InstructGPT) performance on these supervised tasks using minimal labeled data. Notably, using just 80 dialogues on the MultiWOZ dataset, our approach enhances ChatGPT's performance by an impressive 41.4%, matching or surpassing some fully supervised start-of-the-art models. Additionally, the instance-specific chain-of-thought prompt generated by our approach improves InstructGPT's reasoning accuracy compared to human-crafted or automatically generated prompts. The code and data are publicly available at \url{https://github.com/Leezekun/Directional-Stimulus-Prompting}.

1 Introduction

Directional Stimulus Prompting addresses the difficulty of steering black-box LLMs toward fine-grained desired outputs by adding instance-specific hints generated by a small policy model. The policy model can be trained with supervised fine-tuning and reinforcement learning, and the framework is evaluated across multiple tasks and LLMs.

  • LLMs can perform strongly yet still struggle to generate outputs that fully align with desired behaviors on specific tasks.
  • Directly optimizing LLMs is often inefficient or infeasible, while prompt engineering still struggles to steer outputs and exploit labeled data effectively.
  • DSP adds input-specific directional stimuli, such as keywords, that guide LLMs toward outputs more closely aligned with desired reference summaries.
  • A small tunable policy model generates each directional stimulus, shifting optimization from the black-box LLM to the policy model.
  • The policy model is trained with supervised fine-tuning and can be further optimized with reinforcement learning using downstream performance or other output-alignment rewards.
  • Experiments evaluate DSP on summarization, dialogue response generation, and chain-of-thought reasoning with ChatGPT, Codex, and InstructGPT.

2 Directional stimulus prompting

DSP inserts discrete, input-dependent directional stimuli into prompts while keeping the black-box LLM fixed. Its policy model is first trained from pseudo-stimulus data and can then be optimized with reward-driven reinforcement learning, constrained implementation choices, and KL regularization.

  • 2 Directional stimulus prompting: DSP introduces discrete tokens z as directional stimuli, combines them with input x, and samples LLM output y conditioned on both.The black-box LLM parameters remain inaccessible and untunable.
  • 2.1 Supervised fine-tuning: For summarization, pseudo-stimuli are keywords from reference summaries; analogous input-stimulus data can support policy-model supervised fine-tuning.The resulting dataset contains input-stimulus pairs.
  • 2.1 Supervised fine-tuning: Pseudo-stimuli may be non-optimal, so supervised fine-tuning alone may not generate the most preferred stimulus for the desired LLM output.
  • 2.2 Reinforcement learning: The objective is to maximize an alignment measure such as task performance or human preference, while optimizing the policy model because the LLM itself cannot be tuned.
  • 2.2 Reinforcement learning: The policy-model objective recasts LLM optimization as maximizing the LLM’s performance under generated stimulus, using the LLM as an evaluation function.
  • 2.2 Reinforcement learning: PPO treats stimulus generation as a Markov decision process and updates a policy initialized from the supervised policy model.
  • 2.2 Reinforcement learning: The reward includes a KL-divergence penalty to keep the policy from moving too far from its initial policy, with the coefficient dynamically adapted during training.
  • 2.2 Reinforcement learning: NLPO-PPO masks less relevant vocabulary tokens through top-p sampling, using p=0.9 in the experiments, and initializes policy and value networks from the supervised model.

3 Experiments

DSP is evaluated on summarization, task-oriented dialogue, and prompt-generation settings using black-box LLMs and small labeled subsets. Across experiments, it improves target-oriented generation, with reinforcement learning providing additional gains in several settings.

  • Experiments: Experiments evaluate DSP for summarization, dialogue response generation, and automatic prompt generation using ChatGPT, Codex, and InstructGPT.The framework is tested with pre-trained T5 or Flan-T5 policy models guiding black-box LLMs.
  • Summarization: DSP improves ChatGPT’s ROUGE, BLEU, and Meteor scores by 1-2 points using only 1,000 to 4,000 CNN/Daily Mail training samples.The comparison includes standard prompting, SFT-trained DSP, and SFT+RL-trained DSP.
  • Summarization: Reinforcement learning further improves summarization performance after supervised fine-tuning, while larger training subsets make the improvement more significant.The training curve relates validation ROUGE-1 performance closely to the reward during training.
  • Dialogue response generation: DSP significantly improves Codex and ChatGPT Success and Inform rates on MultiWOZ, although corpus-level BLEU does not improve.The comparison includes models trained on the full 8,438-dialogue dataset, whereas DSP achieves notable results with only 80 dialogues.
  • Dialogue response generation: Increasing supervised fine-tuning data does not guarantee dialogue gains, whereas reinforcement learning consistently provides performance improvements.The authors attribute this pattern to RL exploring model-preferred stimuli beyond pseudo-labeled dialogue acts.
  • Prompt generation: Instance-specific prompt generation targets the weakness of general task prompts, which may not be optimal for every input.The approach generates prompts tailored to individual instances rather than relying only on task-level instructions.

4 Related work

Related work addresses black-box LLM capabilities, prompt optimization, controllable generation, and reinforcement learning for NLP. DSP is positioned as a method for improving alignment without directly fine-tuning or differentiably optimizing the target LLM.

  • Black-box large language models: Black-box LLMs exhibit emergent abilities but can still fail to align outputs with desired downstream behaviors.Their scale also makes local execution and fine-tuning costly for many users and researchers.
  • Prompt optimization and engineering: Prompt optimization studies manual, automatic, and soft-prompt methods, but black-box APIs limit approaches requiring gradients or continuous prompts.These constraints motivate methods that optimize prompts without passing gradients through the target model.
  • Controllable text generation: Controllable generation has used attribute-conditioned models, attribute predictors, and generative discriminators, while direct language-model training remains costly.These methods control outputs through additional models, codes, or distributions.
  • Reinforcement learning for NLP: Reinforcement learning has been applied to language generation tasks including parsing, translation, summarization, and conversational systems.The literature formulates token generation as sequential action selection under a reward.

5 Conclusions and future work

The paper concludes that DSP provides fine-grained, instance-specific guidance for black-box LLMs while shifting optimization to a tunable policy model. It also identifies machine-language stimuli and non-textual directional stimuli as future directions.

  • Conclusions: DSP guides black-box LLMs toward desired outputs by using a tunable policy model to generate fine-grained, instance-specific directional stimuli.This converts optimization of the black-box LLM into optimization of the policy model.
  • Conclusions: The authors report that DSP improves control and guidance while effectively using labeled data.Generated stimuli also provide insights into LLM behavior.
  • Future work: Future work will explore machine-language communication between policy models and LLMs and directional stimuli beyond text.The proposed machine language may not be intuitively preferred by humans but could convey guidance information more effectively.

A.1 Summarization

The summarization experiments use CNN/Daily Mail data with small training subsets and keyword-based pseudo-stimuli, while dialogue experiments use MultiWOZ annotations and dialogue acts as stimuli.

  • Summarization: CNN/Daily Mail contains 287,113 training examples, 13,368 validation examples, and 11,490 test examples.The experiments use subsets of 1,000, 2,000, and 4,000 training examples, plus 500 validation and 500 test examples.
  • Summarization: Summarization pseudo-stimuli are keywords automatically extracted from articles and reference summaries with TextRank, retaining keywords appearing in the reference summary.These keywords are converted into stimulus text for policy-model training.
  • Dialogue response generation: MultiWOZ provides dialogue contexts, system responses, belief states, database results, and dialogue-act annotations for task-oriented dialogue.The dataset contains 8,438 training dialogues, 1,000 validation dialogues, and 1,000 test dialogues.
  • Dialogue response generation: The policy model is trained to convert dialogue context into verbalized dialogue acts used as directional stimuli.Structured domain, act, and slot triplets are serialized into bracketed text.

A.3 Chain of Thought reasoning

The study evaluates instance-specific chain-of-thought trigger prompts on MultiArith and AQuA, comparing them with human-crafted and automatically designed prompts.

  • The evaluation uses the MultiArith and AQuA arithmetic reasoning datasets.These datasets are used to assess generated chain-of-thought trigger prompts.
  • The study compares its prompts with 14 human-crafted chain-of-thought prompts evaluated in prior work.
  • The study also compares against a prompt automatically designed by the APE approach.

B.1 Summarization

The summarization analysis examines keyword guidance, generated-hint properties, GPT-4 preferences, and robustness under zero-shot prompting.

  • The training analysis tracks generated keywords, keyword precision, and summary ROUGE-1 over 4,000 samples.
  • Increasing keyword precision aligns with higher ROUGE-1, but too few keywords can prevent performance improvement even when precision is high.
  • Generated hints most frequently contain nouns and proper nouns, while common named entities include persons, geopolitical entities, dates, organizations, and numerals.
  • GPT-4 favored DSP-generated summaries in 255 of 500 cases (51.0%), versus 222 standard-prompting summaries (44.4%) and 23 ties (4.6%).
  • Zero-shot training and evaluation produce performance improvements over standard prompting comparable to few-shot prompting.
  • The approach remains robust when training and evaluation use different numbers of prompt examples, with few-shot training outperforming standard prompting under zero-shot testing.

B.2 Dialogue response generation

The low-resource MultiWOZ evaluation compares systems using limited dialogue data and reports especially weak performance for most baseline methods.

  • Most compared methods struggle to achieve acceptable performance with only 80 dialogues, so their results at that setting are generally not reported.
  • Among methods reporting results with 80 dialogues, performance is significantly worse than Codex and ChatGPT guided by the policy model.
  • Even with around 800 dialogues, baseline Inform and Success rates remain much lower than those achieved by ChatGPT and Codex.

B.3 Chain-of-Thought reasoning

The paper illustrates Directional Stimulus Prompting across summarization, dialogue generation, and chain-of-thought reasoning by adding task-specific, instance-level guidance to prompts.

  • Chain-of-Thought reasoning: For chain-of-thought reasoning, reinforcement learning yields newly discovered trigger prompts that differ from those used during supervised fine-tuning.
  • Chain-of-Thought reasoning: The examples include modified or combined prompts and newly generated wording such as “Let’s think step by step using proven methods.”
  • Examples: The paper provides running examples comparing standard prompting, DSP with supervised fine-tuning, and DSP with supervised fine-tuning plus reinforcement learning.
  • Summarization: For CNN/Daily Mail summarization, DSP adds keywords to the prompt so ChatGPT generates summaries conditioned on policy-model guidance.
  • Dialogue response generation: For MultiWOZ, DSP adds dialogue acts for each system turn and explanations of those acts to guide response generation.
  • Prompt comparison: Standard prompts provide task instructions and demonstrations, whereas DSP augments them with directional stimulus information.
Loading 2302.11520v4…