PaperlayerView exact revision on arXiv

Source-linked AI summary

Black-Box Prompt Optimization: Aligning Large Language Models without Model Training

Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, Minlie Huang

arXiv:2311.04155v3cs.CL

TL;DR

LLMs can struggle to follow human intents, while conventional alignment often requires costly training that users cannot apply to closed models. The paper introduces BPO, which optimizes prompts from preference feedback without changing model parameters, and reports improved alignment across models, including gains over PPO and DPO. BPO is also described as complementary to existing alignment pipelines, though its optimizer is trained on a limited set of 14K prompt pairs.

  • Problem

    LLMs are not always well aligned with human intents, while training-based alignment is costly and unavailable for many closed-source API models.

  • Method

    BPO learns to rewrite user prompts using preference comparisons so unchanged LLMs can produce responses better aligned with human preferences.

  • Results

    BPO improves alignment across API-based and open-sourced LLMs, with reported win-rate increases of 8.8% to 22.0%.

  • Takeaways & Limitations

    BPO reaches or surpasses mainstream alignment techniques on Vicuna models and further improves alignment methods when used alongside them.

  • Takeaways & Limitations

    The released optimizer is trained on only 14K prompt pairs from a limited range of feedback datasets and scenarios.

Abstract

from arXiv · show

Large language models (LLMs) have shown impressive success in various applications. However, these models are often not well aligned with human intents, which calls for additional treatments on them; that is, the alignment problem. To make LLMs better follow user instructions, existing alignment methods primarily focus on further training them. However, the extra training of LLMs is usually expensive in terms of GPU computing; even worse, some LLMs are not accessible for user-demanded training, such as GPTs. In this work, we take a different perspective -- Black-Box Prompt Optimization (BPO) -- to perform alignments. The idea is to optimize user prompts to suit LLMs' input understanding, so as to best realize users' intents without updating LLMs' parameters. BPO leverages human preferences to optimize prompts, thus making it superior to LLM (e.g., ChatGPT) as a prompt engineer. Moreover, BPO is model-agnostic, and the empirical results demonstrate that the BPO-aligned ChatGPT yields a 22% increase in the win rate against its original version and 10% for GPT-4. Notably, the BPO-aligned LLMs can outperform the same models aligned by PPO and DPO, and it also brings additional performance gains when combining BPO with PPO or DPO. Code and datasets are released at https://github.com/thu-coai/BPO.

1 Introduction

The paper identifies a gap between human intents and LLM understanding, and proposes Black-Box Prompt Optimization (BPO) to align models by rewriting prompts rather than modifying model parameters. BPO uses preference data to improve responses without training target LLMs and reports gains across models and alignment methods.

  • LLM alignment remains difficult because human prompts and model understanding can diverge, despite strong instruction-following and preference-understanding abilities.
  • Training-based alignment is constrained by rising training costs, limited access to closed-source API models, and poor interpretability of preference improvements.
  • BPO rewrites ambiguous or disorganized human prompts into instructions better suited to LLM understanding, without modifying the underlying models.
  • BPO learns prompt transformations from preference comparisons, using paired favorable and unfavorable responses to construct 14K original-versus-optimized instruction pairs.
  • 8.8% to 22.0% win-rate increases were reported across API-based and open-sourced models, including gpt-3.5-turbo, gpt-4, claude-2, llama-2-chat, and vicuna.
  • BPO outperformed PPO and DPO in experiments and provided additional gains when combined with existing reinforcement-learning-based alignment.

2 Related Work

Prior alignment and prompt-optimization methods either train models or optimize prompts for particular tasks and models. The paper positions BPO as a model-agnostic alignment approach that learns to transform user inputs using preference feedback.

  • Existing alignment methods commonly use supervised fine-tuning and reinforcement learning from human feedback, but they depend on substantial data, human effort, or unstable training procedures.
  • BPO is presented as applicable to various LLMs rather than tied to a single model or task.
  • BPO constructs optimized prompts from paired good and bad responses, then trains a sequence-to-sequence prompt preference optimizer.
  • Hard prompt engineering often requires extensive manual effort, while automated prompt methods have historically focused on specific tasks rather than general alignment.
  • Prior prompt-optimization methods may require searching separately for each task and optimizing for an individual model, limiting their general applicability.
  • BPO reframes alignment as optimizing the input prompt while leaving the LLM unchanged, supporting use with models that are difficult to train or access directly.

3 Black-Box Prompt Optimization

Black-Box Prompt Optimization (BPO) aligns LLM outputs with human preferences by optimizing user prompts rather than model parameters. It constructs preference-based instruction pairs, trains a small sequence-to-sequence optimizer, and compares BPO with existing alignment methods.

  • 3.1 Task Definition: BPO optimizes input prompts to improve alignment between model outputs and human preferences without modifying LLM parameters.The approach targets the input side of alignment and is applicable to API-based models.
  • 3.1 Task Definition: BPO uses favorable and unfavorable responses to identify preference-relevant differences and refine the original user instruction.An LLM maps (Xuser, Ygood, Ybad) to an optimized prompt Xopt.
  • 3.2 Training Data Construction: The training data combines four preference-annotated instruction-tuning datasets, quality filtering, and diversity filtering to produce about 14k samples.The samples use the format (Xuser, Ygood, Ybad), and the work focuses on single-turn response generation.
  • 3.2 Training Data Construction: ChatGPT refines the instructions, after which rule-based filtering removes incorrect optimizations and yields about 14k original-versus-optimized instruction pairs.The resulting dataset is reported as highly diverse according to its overall distinct score.
  • 3.3 Model Training: A small sequence-to-sequence model is trained to generate Xopt conditioned on Xuser.The optimizer uses llama2-7b-chat as its backbone, while model-scaling exploration is left for future work.
  • 3.3 Model Training: The model-training formulation generates the optimized prompt token by token from the original user input.N denotes the length of Xopt and xt denotes its t-th token.
  • 3.4 Comparison with Existing Methods: BPO is presented as a complementary alignment approach that can add gains beyond conventional RLHF-based alignment.The paper positions BPO as an additional approach rather than a replacement for existing alignment methods.
  • 3.4 Comparison with Existing Methods: Compared with PPO, DPO, and OPRO, BPO avoids modifying LLM parameters and does not require training reward or policy models.This design makes BPO applicable to API-based models and agnostic to LLMs or application tasks.

4 Experiments

The experiments evaluate BPO across black-box and open-source models, alignment methods, data construction, scaling, iteration, and feedback. Results show consistent gains across models and tasks, including improvements over PPO and DPO and benefits from combining BPO with them.

  • Experimental Setup: The evaluation uses multiple instruction datasets and pairwise scoring by GPT-4 and Claude to assess alignment quality.The datasets include Dolly Eval, Vicuna Eval, Self-Instruct Eval, and BPO-test Eval.
  • Black-Box Alignment Results: BPO improves win rates across models and datasets, with average gains of about 20% on gpt-3.5-turbo and text-bison and over 10% for several models including gpt-4.Gains occur across API-based and open-source models with varying capabilities.
  • Scaling Results: BPO boosts smaller models beyond larger baselines: BPO-aligned llama2-7b-chat can match or outperform llama2-70b-chat, while llama2-13b-chat can substantially surpass it.The comparison uses LLaMA2-chat models of varying sizes against the original llama2-70b-chat model.
  • RLHF Results: BPO outperforms PPO and DPO in the reported comparisons, and combining BPO with DPO yields around 30% win-rate increases for vicuna-7b and vicuna-13b.PPO, DPO, and BPO all improve vicuna models, while BPO adds gains on top of preference-training methods.
  • BPO for Data Augmentation: BPO-optimized prompts improve Alpaca-derived training data, producing substantial gains over models trained on the original Alpaca dataset.The approach optimizes instructions before generating responses used to train llama-7b and llama-13b.
  • Iterative Prompt Optimization: Iterative optimization improves ΔWR through four iterations, with a small decline on the fifth, while retaining good prompts with high probability.Retention helps avoid forcing unreasonable changes to the user’s original intent.
  • Feedback Ablation: Feedback-trained BPO improves beyond direct GPT-3.5-turbo prompt optimization, indicating that preference feedback makes prompt refinement more effective.The ablation compares BPO with direct optimization without feedback.

5 Interpretability of BPO

The paper interprets BPO by examining how optimized prompts change and grouping recurring strategies into four non-exclusive categories. These strategies target reasoning, instruction clarity, answer organization, and safety.

  • Optimization Strategies: An analysis of 500 samples identifies four recurring, non-mutually-exclusive BPO strategies.The categories are Explanation Generation, Prompt Elaboration, Providing Hint, and Safety Enhancement.
  • Explanation Generation: Explanation Generation asks models to produce reasoning steps or detailed explanations for more logical and understandable responses.This strategy explicitly structures the requested response process.
  • Prompt Elaboration: Prompt Elaboration clarifies unclear, overly concise, or erroneous user instructions to improve models’ understanding of intent.The strategy targets underspecified or flawed prompts.
  • Providing Hint: Providing Hint adds key points or relevant knowledge so models can organize answers more effectively.The hints specify content to address or background information to use.
  • Safety Enhancement: Safety Enhancement refines potentially unsafe requests toward harmless advice while preserving helpfulness and making the intervention interpretable.The paper presents these four categories as typical examples rather than an exhaustive taxonomy.

6 Conclusion

BPO improves LLM alignment by automatically optimizing user inputs without modifying model parameters. The method shows strong results, but its current optimizer remains limited by training-data coverage and requires further research.

  • BPO automatically optimizes user inputs to improve LLM responses without further adjusting the models.
  • BPO can reach or surpass mainstream alignment techniques on Vicuna models and further improve models already aligned with PPO or DPO.
  • Limitations: The prompt preference optimizer is trained on only 14k optimized-prompt pairs from a limited set of academic feedback datasets.
  • Limitations: Limited topic and length coverage, including few long-context prompts, constrains the optimizer’s expected performance for general usage.

Ethical Considerations

BPO training uses four preference-annotated datasets with different sources and formats. The construction process selects favorable and unfavorable responses and uses separate prompts depending on whether context is available.

  • The source datasets include human or AI preferences and contain some instructions with security issues, while optimized pairs provide safety enhancements.
  • The BPO training data combines OASST1, HH-RLHF, Chatbot Arena Conversations, and Alpaca-GPT4 preference datasets.
  • OASST1 uses the highest- and lowest-rated responses as good and bad responses, while HH-RLHF records helpfulness and harmfulness preferences.
  • Alpaca-GPT4 contributes comparison data with GPT-4-generated preferences, retaining samples where GPT-4 outperforms text-davinci-003.
  • Two data-construction prompts accommodate format differences: one without context for three datasets and one with context for Alpaca-GPT4.

C Implementation Details

The experiments train BPO on a Llama-2-7b-chat backbone and evaluate models with GPT-4 and Claude-v1.3. BPO-aligned smaller models can approach or surpass much larger models.

  • BPO uses Llama-2-7b-chat-hf as its backbone, training for three epochs with AdamW, a 2e-5 learning rate, and linear decay.
  • The implementation uses batch size 4 per GPU with Huggingface Transformers, DeepSpeed Zero-2, and 8×80GB NVIDIA A800 GPUs across experiments.
  • PPO baselines use DeepSpeed-Chat, one epoch for reward-model learning and PPO optimization, and a reward model achieving 80% in-distribution accuracy.
  • GPT-4 and Claude-v1.3 serve as evaluators using prompts adapted from MT-Bench and Alpaca Eval, respectively.
  • BPO-aligned Llama-2-13B-chat outperforms the 70B version, indicating potential for smaller models to surpass much larger ones.

F Experimental Results of Claude Evaluation

Claude-v1.3 evaluations corroborate GPT-4 results: BPO improves every tested model comparison, can approach or exceed much larger models, and outperforms or complements PPO and DPO. Feedback is central to these gains, including in supervised fine-tuning.

  • Claude-v1.3 results are consistent with GPT-4, and every model performs better with BPO alignment than without it.
  • BPO-aligned Llama-2-7B-chat nearly matches Llama-2-70B-chat, while the aligned 13B version surpasses the 70B model.
  • BPO outperforms both PPO and DPO and adds further gains when combined with either method; BPO with DPO exceeds 20% win-rate increases for Vicuna-7B and Vicuna-13B.
  • Fine-tuning on a BPO-reproduced Alpaca dataset produces more than a 40% win-rate increase on Llama-13B.
  • Removing preference feedback may reduce performance on some datasets, whereas BPO achieves gains across every test set.
  • An example optimization makes the prompt more specific and complete by covering more possible scenarios.

H OPRO Experiments

BPO is compared with OPRO as an automated prompt-optimization method, with results indicating stronger applicability to new tasks and more stable performance across categories. The analysis attributes OPRO’s declines partly to task-wide prompts that are poorly tailored to individual samples.

  • OPRO setup: OPRO requires task-specific training data and searches for improved prompts using sampled Dolly examples and GPT-4-generated responses.The comparison uses 2,000 sampled Dolly instances and runs OPRO over 200 samples per category.
  • Comparison: BPO improves applicability to new tasks without requiring the specific searching used by OPRO.
  • Results: BPO achieves stable improvements across most categories, while OPRO declines on more than half the tasks with a negative average improvement.BPO also shows noticeable gains on General QA, whereas OPRO largely declines there.
  • Error analysis: OPRO’s large performance drop is linked to applying the same prompt to every sample within a task.In summarization, a prompt converging on a specific topic caused performance loss on many samples.

I Error Analysis

BPO’s interpretability supports rapid error analysis and iterative prompt improvement, while Figure 11 identifies over-specification, inconsistency, and neglected context as common error patterns. Figure 10 reports more consistent GPT-4 gains for BPO than OPRO across nearly all tasks.

  • Error Analysis: BPO’s interpretability enables rapid error analysis and iterative improvement from optimization failures.Figure 11 presents illustrative examples of common errors.
  • Evaluation Setup: The reference-based evaluation prompt asks GPT-4 to judge an assistant response against a high-quality reference answer using multiple quality factors.The prompt considers helpfulness, relevance, accuracy, depth, creativity, and detail.
  • Comparative Results: BPO demonstrates consistent GPT-4 score gains across nearly all tasks, whereas OPRO declines on over half of the tasks with an average negative improvement.Both methods were run three times and evaluated using average scores relative to the original.
  • Error Analysis: Over-specification occurs when BPO makes a general instruction more specific, limiting the LLM’s output too much.The optimized prompt turns general topics into more specific ones.
  • Error Analysis: Inconsistent optimized instructions can trace back to low-quality training data that favors responses violating the original constraints.Such responses were nevertheless annotated as the favored ones.
  • Error Analysis: BPO can neglect additional context, making the optimized instruction under-specified.This is identified as a third error case in Figure 11.
Loading 2311.04155v3…