Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery

Yuxin Wen; Neel Jain; John Kirchenbauer; Micah Goldblum; Jonas Geiping; Tom Goldstein

Source-linked AI summary

Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery

Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, Tom Goldstein

arXiv:2302.03668v2Feb 7, 2023cs.LGcs.CL

TL;DR

Hard prompts are interpretable and portable but difficult to discover, whereas soft prompts are optimizable yet opaque and model-specific. The paper introduces PEZ, which uses continuous embeddings as intermediate variables while optimizing discrete prompt tokens. It reports competitive image-prompt performance and strong language-classification performance, with transfer improved by fluency regularization.

Problem
Hard prompts require manual engineering, while soft prompts are difficult to interpret, reuse across models, or access through text-based interfaces.
Method
PEZ learns hard prompts with gradient-based optimization by maintaining continuous embeddings and projecting them to discrete vocabulary tokens during optimization.
Results
The learned prompts perform competitively for image generation and outperform other text-optimization schemes on language classification, while transferring across networks.
Takeaways & Limitations
Hard prompts can combine the automation of soft-prompt optimization with the portability, flexibility, and interpretability of discrete text prompts.
Takeaways & Limitations
The authors note that understanding of language-model embedding geometry remains limited and that prompt optimization can bypass simple rule-based content filters.

Abstract

from arXiv · show

The strength of modern generative models lies in their ability to be controlled through text-based prompts. Typical "hard" prompts are made from interpretable words and tokens, and must be hand-crafted by humans. There are also "soft" prompts, which consist of continuous feature vectors. These can be discovered using powerful optimization methods, but they cannot be easily interpreted, re-used across models, or plugged into a text-based interface. We describe an approach to robustly optimize hard text prompts through efficient gradient-based optimization. Our approach automatically generates hard text-based prompts for both text-to-image and text-to-text applications. In the text-to-image setting, the method creates hard prompts for diffusion models, allowing API users to easily generate, discover, and mix and match image concepts without prior knowledge on how to prompt the model. In the text-to-text setting, we show that hard prompts can be automatically discovered that are effective in tuning LMs for classification.

1. Introduction

The paper addresses the tension between interpretable, portable hard prompts and highly optimized but opaque soft prompts. It introduces gradient-based optimization for automatically learning hard prompts across image-generation and language-classification tasks.

Hard prompts use interpretable tokens but are often discovered through trial and error, whereas soft prompts are optimized continuous embeddings without human-readable tokens.
Hard prompts are portable across models and usable through APIs, while soft prompts depend on model-specific embedding dimensions and representation spaces.
The paper proposes learning hard prompts with continuous optimization, combining soft-prompt automation with hard-prompt portability, flexibility, and simplicity.
For image generation, the method learns prompts that elicit specific styles, objects, and appearances without hand-crafted components.
For language classification, learned hard prompts outperform other text-optimization schemes and transfer across networks, with fluency regularization improving transfer.
Optimization can recover tokens that are both interpretable and non-obvious, supporting prompt exploration and discovery.

2. Related Works

Related work spans soft prompt tuning, discrete prompt optimization, image-based prompt discovery, and discrete optimization for quantized networks. The paper positions its method as a hybrid that retains hard vocabulary constraints while using continuous optimization.

Prompting in Language Models: Soft prompt tuning optimizes continuous embeddings prepended to inputs, but the resulting sequences can map to tokens with limited semantic scrutability.
Discrete Optimization for Language: Existing discrete prompt methods include gradient-based AutoPrompt, gradient-free phrase editing, Langevin-dynamics embedding optimization, and reinforcement learning.
Discrete Optimization for Language: AutoPrompt greedily selects tokens using gradients but can become expensive because it evaluates candidates at every prompt location for each gradient step.
Discrete Optimization for Language: Nearest-neighbor projection keeps baseline optimization on discrete tokens but can cause stagnant embeddings, extensive hyperparameter tuning, and reduced solution flexibility.
Prompt Discovery from Images: Image-captioning methods may produce generic captions, while soft prompt optimization for diffusion models is expensive and yields prompts that are neither interpretable nor portable.
Discrete Optimization: The method adapts lessons from discrete optimization in binary networks to refine and simplify language optimizers.

3. Methodology

PEZ learns hard prompts by maintaining continuous prompt embeddings while projecting them to discrete vocabulary neighbors during forward passes. Gradients computed on the projected prompts update the continuous iterates before a final projection.

The method optimizes a sequence of learnable embeddings for a frozen model using an objective function evaluated on task data.
A projection function maps each continuous prompt embedding to its nearest neighbor in the model’s vocabulary embedding matrix.
PEZ maintains continuous iterates while using projected discrete prompts for forward computation and gradient calculation.
Each optimization step retrieves a minibatch, projects the prompt, computes the task gradient, and applies that gradient to the continuous embedding.
The final continuous prompt is projected onto the vocabulary embedding matrix before being returned as a hard prompt.

4. Prompt Inversion with CLIP

The method uses CLIP-guided optimization to discover discrete hard prompts that capture image semantics while remaining usable for generation. These prompts support image similarity, style transfer, concept composition, and prompt distillation, with 8-token prompts performing comparably to CLIP Interrogator in several datasets.

4. Prompt Inversion with CLIP: CLIP-guided optimization discovers hard prompts by minimizing one minus cosine similarity between text- and image-encoder representations.The method avoids gradient calculations through the full diffusion model by optimizing against the CLIP image encoder.
4.1. Experimental Setting: Experiments span LAION, MS COCO, Celeb-A, and Lexica.art, while image quality is measured using a separate OpenCLIP-ViT/G similarity score.Stable Diffusion-v2 generates images, and OpenCLIP-ViT/H provides the prompt-optimization text encoder.
4.2. Results: The learned prompts capture target-image semantics, produce diverse generations across random seeds, and contain interpretable words alongside non-word token sequences.Examples include tokens such as “milkyway” and “campfire,” while emojis can encode additional image information concisely.
4.2. Results: Across four datasets, the method outperforms other gradient-based baselines and approaches CLIP Interrogator while using only CLIP and 8 tokens.CLIP Interrogator performs best on LAION, MS COCO, and Lexica.art, but not Celeb-A; it also uses BLIP, a curated prompt dataset, and up to 77 tokens.
4.2. Results: Longer prompts reduce CLIP image-encoder loss but do not necessarily improve Stable Diffusion generation, with length 16 yielding the most generalizable performance.The authors characterize longer prompts as overfitting and less transferable; constraining optimization to CLIP Interrogator’s keyword bank recovers much of the quantitative difference on LAION and Lexica.art.
4.3. Style Transfer: Shared visual styles can be extracted from several examples into a hard prompt and transferred to new objects or scenes.The approach adapts a setting previously studied with soft prompts while using hard prompts.
4.4. Prompt Concatenation: Hard prompts learned from unrelated images can be concatenated to fuse concepts such as painted horses on a beach and a realistic forest sunset.The prompts function as composable building blocks for intricate scenes.
4.5. Prompt Distillation: Prompt distillation preserves similar image concepts with only 3 or 4 tokens, reducing longer human-made instructions.This addresses diffusion text encoders with limited maximum input lengths, including CLIP’s 77-token limit.

5. Discrete Prompt Tuning with Language Models

The method optimizes discrete hard prompts for language-model classification using task and fluency losses, then evaluates transfer across models and few-shot prompt discovery. It performs competitively on sentiment tasks, improves AGNEWS results, transfers across larger models, and discovers coherent prompts from few-shot data.

5. Discrete Prompt Tuning with Language Models: The optimization objective combines classification task loss with fluency loss to discover discrete hard prompts.Fluency is intended to improve prompt readability and performance; λ is set to 0.003, with a no-fluency ablation.
5.2. Results: The soft-prompt baseline reaches 93.35±0.01 accuracy on SST-2 but cannot transfer across models.Table 2 reports accuracy and standard error across five prompts per method transferred from GPT-2 Large.
5.1. Datasets and Setup: The evaluation covers SST-2, Amazon Polarity, and AGNEWS using prompts optimized on GPT-2 Large.The transfer setup tests GPT-2 XL, T5-LM-XL, OPT-2.7B, and OPT-6B, while few-shot experiments use two or four examples per class.
5.2. Results: The method is comparable to other methods on sentiment analysis and outperforms them on AGNEWS by about 2%.The reported comparison covers the evaluated classification tasks and refers to Table 5 for detailed results.
5.2. Results: Prompts trained on GPT-2 Large transfer to larger language models, while model scaling alone does not guarantee corresponding performance gains.The transfer comparison evaluates five prompts per method against larger models and a template baseline.
5.2. Results: Fluency-constrained prompts transfer better than other prompts, with the fluent method achieving about a 14% increase over the template baseline on OPT-6.7B.The AGNEWS prompts also transfer from GPT-2 Large to GPT-2 XL.
5.2. Results: Few-shot prompt discovery achieves high validation accuracy compared with prepended counterparts, with each few-shot run taking about 5 minutes.The experiments use prompts optimized from small numbers of examples per class.
5.2. Results: Across 100 seeds, many discovered prompts are coherent despite some containing non-interpretable tokens.Examples include prompts referencing BBC, Brian blog, and Blog Revolution analyze.

6. Safety Concerns

Prompt optimization can circumvent token-level content filters, including in an API-generated reproduction of a banned copyrighted image. Iteratively expanding block-lists may not prevent such circumvention.

6. Safety Concerns: Token- or word-level filters are used by text-to-image APIs to prevent NSFW or copyrighted content.The passage gives Midjourney as an example of an API with such restrictions.
6. Safety Concerns: Midjourney banned prompts containing “Afghan” because of a copyright issue involving a famous photograph.The restriction is described as a substring-based ban.
6. Safety Concerns: PEZ generated a prompt without the banned word “Afghan” that reproduced the targeted image through Midjourney.The optimized prompt appeared to incorrectly associate Sharbat Gula with the Taliban.
6. Safety Concerns: Attackers can consistently optimize around additional block-list restrictions, limiting the effectiveness of iterative word bans.The paper points to supplementary Figure 10 for this behavior.

7. Conclusion

The paper presents PEZ as a robust gradient-based method for optimizing hard prompts through continuous embeddings. Its experiments show practical flexibility, while unresolved embedding geometry and prompt safety concerns remain boundaries.

7. Conclusion: PEZ uses continuous embeddings as intermediate variables to reliably optimize hard prompt tokens.The method selects useful locations in embedding space rather than projecting a separately optimized soft prompt afterward.
7. Conclusion: Accumulating gradients into the soft prompt makes optimization more robust to learning rates and potential data noise.This describes the method’s stated robustness mechanism.
7. Conclusion: A deeper understanding of language-model embedding-space geometry may enable stronger prompt optimization in the future.The paper characterizes current understanding of that geometry as still in its infancy.
7. Conclusion: Hard prompts can be generated and flexibly used in practical applications, but some may still contain uninterpretable tokens.The conclusion distinguishes human readability from complete token interpretability.
7. Conclusion: Hard prompts may extract harmful phrases or sensitive training-data content, although the authors did not observe specific instances.The paper identifies this as a concern for future applications.

A.1. Additional Results for Prompt Inversion with CLIP

Additional results provide qualitative prompt-inversion examples for image concepts and styles. The examples use fixed textual templates and generated token sequences, with further qualitative results referenced in Figure 9.

A.1. Additional Results for Prompt Inversion with CLIP: The appendix provides additional qualitative results for prompt inversion with CLIP.It directs readers to Figure 9 for more qualitative examples.
A.1. Additional Results for Prompt Inversion with CLIP: The experiments use templates describing a tiger, Paris streets, a calculator, and a rocket in a learned style.The placeholder is replaced with the learned prompt.
A.1. Additional Results for Prompt Inversion with CLIP: The listed learned prompt sequences include tokens such as “watercolor,” “anime,” and “illustration.”The sequences also contain unusual or nonstandard token strings.

A.2. Additional Experiments and Details for Text-to-Text Hard Prompting

Additional experiments evaluate discrete hard prompts for text classification, prompt transfer, image-prompt distillation, and content-filter evasion. The results include competitive classification accuracy, model-dependent learning-rate behavior, and quantitative comparisons based on CLIP scores.

Experimental details: Table 5 reports the best validation accuracy across three learning rates, using a 10-token prompt prepended to each input and early stopping on a 5,000-example hold-out set.The experiments evaluate every 100 steps.
Text-to-text results: Text classification accuracy is comparable to other methods on sentiment analysis and about 2% better on AGNEWS.The learned prompts are not coherent English, but contain relevant tokens and phrases.
Text-to-text results: All optimized tokens moved away from label-token initialization, suggesting the process relearned the class label.
Text-to-text results: The optimization process finds task-relevant words but lacks the ability to create full sentences.
Optimization behavior: FluentPrompt and AutoPromptSGD exhibit model-dependent learning-rate behavior when every optimization step is projected.
Optimization behavior: AutoPrompt with SGD performs better than AutoPrompt with k=1 on SST-2, where k denotes the number of candidates evaluated by the greedy process.