PaperlayerView exact revision on arXiv

Source-linked AI summary

RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning

Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, Zhiting Hu

arXiv:2205.12548v3cs.CLcs.LG

TL;DR

Existing prompt optimization methods struggle to provide interpretable, reusable discrete prompts and often require gradients, while discrete search itself is difficult. RLPrompt trains a parameter-efficient policy to generate discrete prompts and stabilizes rewards; experiments show improvements across few-shot classification and unsupervised style transfer, with gibberish prompts transferring between LMs.

  • Problem

    Discrete prompts are interpretable and reusable but difficult to optimize, while soft prompts lack interpretability and cross-LM compatibility and may require unavailable gradients.

  • Method

    RLPrompt formulates discrete prompt optimization as reinforcement learning with a parameter-efficient policy network and reward stabilization.

  • Results

    RLPrompt improves over a wide range of fine-tuning and prompting methods on few-shot classification and unsupervised text style transfer, while optimized prompts are often gibberish yet transferable between LMs.

  • Takeaways & Limitations

    Strong prompting behavior can be encoded in incoherent discrete prompts that retain significant performance across different LMs.

  • Takeaways & Limitations

    The method was not evaluated on newer huge models such as GPT-3, and reward-function design may require domain expertise.

Abstract

from arXiv · show

Prompting has shown impressive success in enabling large pretrained language models (LMs) to perform diverse NLP tasks, especially when only few downstream data are available. Automatically finding the optimal prompt for each task, however, is challenging. Most existing work resorts to tuning soft prompt (e.g., embeddings) which falls short of interpretability, reusability across LMs, and applicability when gradients are not accessible. Discrete prompt, on the other hand, is difficult to optimize, and is often created by "enumeration (e.g., paraphrasing)-then-selection" heuristics that do not explore the prompt space systematically. This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL). RLPrompt formulates a parameter-efficient policy network that generates the desired discrete prompt after training with reward. To overcome the complexity and stochasticity of reward signals by the large LM environment, we incorporate effective reward stabilization that substantially enhances the training efficiency. RLPrompt is flexibly applicable to different types of LMs, such as masked (e.g., BERT) and left-to-right models (e.g., GPTs), for both classification and generation tasks. Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods. Interestingly, the resulting optimized prompts are often ungrammatical gibberish text; and surprisingly, those gibberish prompts are transferrable between different LMs to retain significant performance, indicating LM prompting may not follow human language patterns.

1 Introduction

RLPrompt addresses the difficulty of optimizing interpretable, reusable discrete prompts by training a reinforcement-learning policy network rather than tuning embeddings or enumerating candidates. It improves over broad fine-tuning and prompting baselines, while revealing that effective prompts can be gibberish and transferable across language models.

  • Motivation: Soft prompts are difficult to interpret, incompatible across LMs, and dependent on gradients, while discrete prompts are hard to optimize.Discrete prompt search is challenging because tokens are not directly amenable to gradient-based optimization.
  • Approach: RLPrompt trains a small policy network to generate discrete prompts using reward signals instead of directly editing prompt tokens.The policy is implemented as an MLP inserted into a frozen compact model.
  • Approach: Reward stabilization addresses the instability caused by the large LM’s complex, black-box environment.The paper proposes two methods to stabilize rewards and improve optimization efficiency.
  • Results: Experiments on few-shot classification and unsupervised text style transfer improve over a wide range of fine-tuning and prompting methods.The approach is also reported as robust to different modeling choices, including classification verbalizers.
  • Results: Optimized prompts often contain unintelligible text yet can retain significant performance when transferred between different LMs.This suggests that different pretrained LMs may share structures for prompting that do not follow human language patterns.

2 Discrete Prompt Optimization with RL

RLPrompt formulates discrete prompt optimization as reinforcement learning: a policy samples prompt tokens, receives task rewards from a frozen LM, and is trained through a parameter-efficient architecture. Reward normalization and piecewise objectives address unstable or exploitable signals across classification and generation settings.

  • 2.1 Discrete Prompt Optimization Problem: The framework supports masked-LM classification and left-to-right generation without requiring fine-tuning of the pretrained task LM.These usages are illustrated for classification labels and generated text.
  • 2.1 Discrete Prompt Optimization Problem: The optimization target is a fixed-length vocabulary prompt maximizing a downstream performance measure, but brute-force search has exponential complexity O(|V|^T).Prior methods approximate gradients through embeddings or modify human-written prompts heuristically.
  • 2.2 The Reinforcement Learning Formulation: RLPrompt represents discrete prompt learning as an RL problem in which a policy selects tokens sequentially and receives a task reward after completing the prompt.The policy explores a vocabulary-defined prompt space while the task LM produces the evaluated output.
  • 2.2 The Reinforcement Learning Formulation: Treating the task LM as a black box removes the need for gradient access and guides exploration with reward signals rather than enumeration.After training, greedy token selection produces a deterministic prompt.
  • 2.3 Efficient Parameterization of Policy: A task-specific MLP inserted into a frozen compact policy LM provides a parameter-efficient policy whose policy LM need not equal the task LM.Contextual embeddings of partial prompts are adapted before the original LM head generates token probabilities.
  • 2.4 Reward Engineering and Stabilization: Input-specific z-score normalization reduces bias from different reward scales by standardizing rewards across sampled prompts for each input.The policy is conditioned on inputs so z-scores from different inputs can be distinguished within a batch.
  • 2.4 Reward Engineering and Stabilization: Piecewise rewards address misspecified objectives by combining smooth and disjoint components that better express task priorities.The motivation includes preventing classification policies from exploiting high probability for one class on arbitrary inputs.

3 Experiments

Experiments evaluate RLPROMPT on few-shot classification and unsupervised text style transfer, comparing it with prompting, fine-tuning, and black-box methods. Results show strong task performance, training efficiency, robustness, and transferable prompts, while fluency constraints reduce task performance.

  • Experiment setup: RLPROMPT is evaluated on few-shot classification and unsupervised text style transfer across diverse language models and task settings.The classification experiments use 16 examples per class and compare representative training and prompting baselines.
  • Few-Shot Text Classification: On few-shot classification, the 5-token method outperforms Manual Prompt and Instructions on every dataset and beats Fine-Tuning and In-Context Demonstration on nearly all datasets.It also outperforms prior discrete prompt optimization methods on all benchmarks.
  • Few-Shot Text Classification: RLPROMPT achieves higher average accuracy with lower standard deviations than Prompt Tuning and substantially outperforms BB Tuning, including when BB Tuning uses 50 soft tokens.The lower variance indicates reduced sensitivity to training factors in the reported few-shot experiments.
  • Training Efficiency: The RL-based method converges in a similar number of steps to BB Tuning while achieving superior performance and comparable worst-prompt accuracy after convergence.The comparison uses mean, minimum, and maximum test accuracy over five trials.
  • Unsupervised Text Style Transfer: On Yelp style transfer, RLPROMPT has markedly better fluency than expensive training baselines, producing higher or competitive joint score J(·) and geometric mean GM(·).Its frozen language-model parameters may preserve fluent generation capability, while trained prompts outperform manual prompts on average with lower variance.
  • Analysis: Fluency-constrained prompts have lower perplexity but much lower downstream task performance than unconstrained optimized prompts.The trade-off is illustrated with fluent and gibberish prompts in the style-transfer analysis.
  • Analysis: Gibberish prompts transfer between different language models, with prompts learned from smaller models sometimes matching or exceeding performance on larger models.Transfer success depends on the source and target models, while prompts learned from larger models can decline sharply on smaller models.
  • Few-Shot Text Classification: The method discovers well-performing classification prompts across a wide variety of verbalizers, addressing sensitivity to verbalizer choices.This robustness is reported on few-shot classification experiments including SST-2.

4 Related Work

Prior work contrasts expensive fine-tuning with manual and learned prompting, including soft prompts optimized by gradient descent. These approaches motivate discrete prompt optimization that is more interpretable and broadly usable.

  • Fine-tuning updates all model parameters expensively and has limited success with small datasets.
  • Manual prompts can steer large language models without any training.
  • Soft prompts use continuous embedding vectors optimized with gradient descent.

5 Conclusion

The paper presents an efficient and flexible RL approach for discrete prompt optimization and reports improvements across few-shot classification and unsupervised text style transfer. Its analysis finds that strong prompts can be incoherent yet transferable between language models.

  • RLPrompt improves over a wide range of fine-tuning and prompting methods on few-shot classification and unsupervised text style transfer.
  • Strong optimized prompts are often incoherent but transferable between different language models.
  • Prompt transfer could support learning prompts cheaply from smaller models and inference with larger models.

6 Limitations

The paper’s evaluation covers regular-sized language models such as RoBERTa and GPT-2, while leaving huge models, reward design, and learned-prompt patterns for future study.

  • The method was evaluated on regular-sized LMs like RoBERTa and GPT-2, not recent huge models like GPT-3.
  • Designing reward functions may require domain expertise, although inverse reinforcement learning is suggested as a possible remedy.
  • The paper does not closely examine patterns in learned prompts or the LMs’ so-called secret language.

Ethics Statement

The authors state that they follow established research and professional ethics codes while acknowledging potential malicious uses of large pretrained language models. They also express hope that RLPrompt can help control unethical LM behavior.

  • The research follows the ACL Code of Ethics and the ACM Code of Ethics and Professional Conduct.
  • Large pretrained language models may be used to generate fake, toxic, or offensive content, with or without prompting.
  • The authors hope their prompting technique can help harness and control language models’ unethical behaviors.

A Experiment Details

RLPrompt uses a compact distilGPT-2 policy language model with a one-hidden-layer MLP, adding only 3.1M trainable parameters.

  • The policy uses distilGPT-2 with 82M parameters and a one-hidden-layer MLP containing 2048 hidden states.
  • 3.1M added parameters represent 3.8% of the policy language model’s parameters.

A.2 Few-Shot Text Classification

The few-shot classification experiments evaluate RLPrompt with standardized training and evaluation procedures, including reward scaling and normalization. Additional results and ablations examine dataset size effects and reward-function stability.

  • Reward function: The reward averages performance over all few-shot training examples, with balancing weights λ1 = 180 and λ2 = 200 tuned on validation data.
  • Baselines: The experiments compare RLPrompt against manual prompts, instructions, in-context demonstrations, and prompt tuning.
  • Training procedure: Training applies reward scaling and z-score normalization, uses top-256 policy sampling, and evaluates three highest-validation prompts.The policy is trained for 6K steps with two prompt tokens and 12K steps with five prompt tokens.
  • Results: The method outperforms prompting baselines on average across additional few-shot classification datasets.
  • Results: Fine-Tuning achieves the highest average accuracy on larger multi-way datasets with more training examples.Yahoo has 256 training examples in the 16-class setting, whereas SST-2 has 32 examples in the 2-class setting.
  • Reward ablation: The piecewise reward function improves training stability and average performance across random seeds and datasets.The ablation uses SST-2 and AG’s News with five prompt tokens and distilRoBERTa-base.

A.3 Text Style Transfer

RLPrompt is evaluated for few-shot text style transfer using content, style, fluency, and standard generation metrics. On Shakespeare, it performs competitively with models trained on the full dataset while using only 100 examples per style and no model update.

  • Reward design: Content preservation uses the CTC metric, which measures bidirectional information alignment between input and output.
  • Evaluation: Yelp evaluation compares outputs with references using BLEU and BERTScore, and reports GPT-2 perplexity with standard deviations across three prompt sets.
  • Evaluation: The evaluation reports Content, Style, and Fluency alongside BLEU, BERTScore, and perplexity.Content uses CTC; Style uses BERT classifiers; Fluency uses a grammaticality classifier.
  • Experimental setup: The style-transfer experiments compare RLPrompt with Deep Latent, STRAP, and prompting baselines on Shakespeare.Deep Latent and STRAP are trained on the full dataset, while RLPrompt is evaluated in the few-shot setting.
  • Few-shot results: With only 100 training examples and no model update, RLPrompt outperforms or approaches Deep Latent and STRAP trained on the full Shakespeare dataset.
  • Few-shot results: RLPrompt accommodates a wider range of styles than STRAP and shows greater robustness to randomly drawn training sets.The lower standard deviations occur for both Content and Style.

B Additional Analysis

Additional analyses constrain prompts toward fluent text, test transfer across language models, and inspect the learned discrete tokens. They find both semantically meaningful “strong words” and high-performing gibberish prompts, while composition effects depend on prompt construction.

  • Fluent prompts: RLPrompt can optimize fluent prompts by restricting each action to the top-20 GPT-2 token probabilities conditioned on preceding prompt tokens.
  • Prompt transfer: Transferred prompts are evaluated across GPT-2 model sizes using a heatmap whose rows are generation models and columns are prompt-learning models.Brighter colors indicate better joint score J(·); Manual and Random are comparison baselines.
  • Qualitative analysis: Learned prompts frequently contain semantically similar task-relevant tokens called “strong words.”Examples include opinion-related words for sentiment analysis and “News” for news-topic classification.
  • Qualitative analysis: A gibberish prompt, “imentariesariesaryary,” reaches 80% accuracy with RoBERTa-large on AG’s News.
  • Qualitative analysis: Composing more strong words can improve performance, but success is sensitive to factors such as word order.
  • Related approaches: The analysis compares manual prompting, fine-tuning, instructions, demonstrations, discrete-prompt enumeration, AutoPrompt, and soft prompt tuning.The related methods differ in whether they use human-written prompts, demonstrations, heuristics, gradients, or continuous embeddings.

C.2 Controllable Text Generation

This section situates controllable text generation among prompt-based methods and compares manually written, fluent learned, and RL-optimized prompts, including Shakespeare style transfer.

  • Prompting steers language models toward properties such as topic and toxicity, and conditions generation on modalities including images and structured data.
  • Manual instructions and dataset-specific templates learned by RLPROMPT are included in the baseline comparisons.
  • Table 15 compares manual, fluency-constrained learned, and RL-optimized prompts for text style transfer using GPT-2-xl and metrics averaged over five runs.
  • Shakespeare style-transfer results are reported separately with the same metrics and format used for the broader prompt comparison.
Loading 2205.12548v3…