Source-linked AI summary
Black-Box Tuning for Language-Model-as-a-Service
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, Xipeng Qiu
TL;DR
LMaaS makes large PTMs accessible through inference APIs, but unavailable gradients make conventional prompt optimization difficult, especially in high-dimensional spaces. BBT addresses this by optimizing continuous prompts with derivative-free search in a random low-dimensional subspace, and experiments report performance exceeding manual, in-context, prompt-tuning, and full-model-tuning approaches. The authors note that hand-crafted templates and label words leave reported performance as a lower bound, while random subspaces may be sub-optimal.
Problem
LMaaS lacks PTM gradients, making direct optimization of continuous prompts with tens of thousands of parameters intractable for derivative-free methods.
Method
BBT iteratively queries the PTM inference API and optimizes a prepended continuous prompt within a smaller random linear-projection subspace using derivative-free optimization.
Results
BBT with RoBERTa on a few labeled samples outperforms manual prompting, in-context learning, prompt tuning, and full model tuning.
Takeaways & Limitations
Black-box tuning enables users to optimize task-specific prompts locally on resource-limited devices without access to PTM parameters, gradients, or GPUs.
Takeaways & Limitations
The reported performance is a lower bound because templates and label words are hand-crafted, while random-projection subspaces can be sub-optimal.
Abstract
from arXiv · showhide
Extremely large pre-trained language models (PTMs) such as GPT-3 are usually released as a service. It allows users to design task-specific prompts to query the PTMs through some black-box APIs. In such a scenario, which we call Language-Model-as-a-Service (LMaaS), the gradients of PTMs are usually unavailable. Can we optimize the task prompts by only accessing the model inference APIs? This paper proposes the black-box tuning framework to optimize the continuous prompt prepended to the input text via derivative-free optimization. Instead of optimizing in the original high-dimensional prompt space, which is intractable for traditional derivative-free optimization, we perform optimization in a randomly generated subspace due to the low intrinsic dimensionality of large PTMs. The experimental results show that the black-box tuning with RoBERTa on a few labeled samples not only significantly outperforms manual prompt and GPT-3's in-context learning, but also surpasses the gradient-based counterparts, i.e., prompt tuning and full model tuning.
1. Introduction
LMaaS exposes powerful but parameter- and gradient-inaccessible PTMs through black-box APIs, motivating prompt optimization using only inference queries. The paper proposes Black-Box Tuning, which searches a low-dimensional random subspace with derivative-free optimization and reports results surpassing manual, in-context, prompt-tuning, and full-model-tuning baselines.
- LMaaS releases large PTMs through black-box APIs because running them locally is expensive or infeasible and parameters may remain closed.
- Hand-crafted prompts and in-context learning can solve simple tasks but cannot fully exploit labeled data.
- Continuous prompt tuning is deployment-efficient, but prior methods learn prompts through backpropagation unavailable in LMaaS.
- Derivative-free optimization is required without gradients, yet direct search over prompts with tens of thousands of parameters is intractable because convergence slows in high dimensions.
- BBT iteratively queries the inference API and optimizes a continuous prompt in a smaller random subspace rather than the original high-dimensional space.
- BBT with a few labeled samples outperforms manual prompting, in-context learning, prompt tuning, and full model tuning.
2. Background
The background frames large PTMs as remotely served models and connects BBT to intrinsic dimensionality, prompt-based learning, and derivative-free optimization. These concepts explain why continuous prompts and low-dimensional search are relevant when model parameters and gradients are unavailable.
- Large-Scale PTMs as APIs: Large-scale PTMs are commonly served through APIs because inference can be accelerated, models may remain closed, and local execution is expensive or infeasible.
- Intrinsic Dimensionality of PTMs: Intrinsic dimensionality is the minimum parameter count needed to obtain satisfactory solutions, representing an effective low-dimensional reparameterization.
- Intrinsic Dimensionality of PTMs: Randomly projected subspaces can retain substantial tuning performance, while multi-task supervision can compress task subspaces below 100 dimensions.
- Prompt-Based Learning: Prompt-based learning formulates downstream tasks as masked language modeling and can optimize continuous prompts while keeping PTM parameters unchanged.
- Derivative-Free Optimization: Derivative-free optimization updates sampled solutions using function values rather than gradients and includes evolutionary algorithms and Bayesian optimization.
3. Approach
The approach formulates language understanding as black-box prompt optimization: continuous prompts are searched through API evaluations, using a low-dimensional projected space and derivative-free optimization.
- Task formulation: Modified texts combine task templates and label words, while the PTM API returns masked-position logits used to evaluate prompt quality.The objective minimizes a loss between API predictions and golden labels over a batch.
- Black-box optimization: Derivative-free optimization is used because LMaaS inference APIs expose function evaluations but not gradients, while direct optimization in the high-dimensional prompt space is intractable.Large PTM prompt embeddings can have tens of thousands of dimensions.
- Prompt parameterization: The continuous prompt is optimized through an increment Az added to an initial prompt p0, rather than directly searching the full prompt space.The initialization may use randomly sampled vocabulary-token embeddings, or an NLI-pretrained prompt for sentence-pair tasks.
- Loss functions: Cross entropy and hinge loss are considered alongside negative accuracy because accuracy rewards can be sparse and less informative with limited training data.The hinge-loss margin is set to γ = 2, and the three losses are compared in Figure 3.
- Derivative-free optimizer: CMA-ES optimizes the projected variables by sampling query populations from a multivariate normal search distribution and updating its mean, scale, and covariance from successful steps.The covariance matrix controls the distribution's shape, while the standard deviation controls step length.
4. Experiments
Experiments evaluate black-box tuning with RoBERTaLARGE across few-shot language-understanding tasks, comparing it with gradient-based, gradient-free, manual-prompt, in-context, and feature-based methods. Ablations examine loss functions, subspace dimensionality, and prompt length.
- Experimental setup: Experiments cover sentiment analysis, topic classification, natural language inference, and paraphrase across seven datasets.The datasets are SST-2, Yelp polarity, AG’s News, DBPedia, SNLI, RTE, and MRPC.
- Experimental setup: All overall-comparison results use pre-trained RoBERTaLARGE with 16 labeled samples per class and report mean and standard deviation over three splits.The few-shot protocol randomly selects k samples per class for training and another k for development.
- Overall comparison: Black-box tuning significantly outperforms four gradient-free methods and surpasses prompt tuning, p-tuning v2, and model tuning on average across seven tasks.The paper attributes the possible advantage over gradient-based optimization to DFO’s exploration mechanism, while presenting this as a suspicion rather than an established cause.
- Deployment comparison: Among the considered methods, only black-box tuning combines satisfactory performance with reasonable training time, memory footprint, and network load for LMaaS.Model tuning requires a full model copy per user, while gradient-based methods require gradients and cannot make the PTM serve as a service.
- Ablation study: Adam converges faster than CMA-ES on SST-2 and AG’s News, but CMA-ES achieves slightly better SST-2 development performance and better average test performance across seven tasks.On AG’s News development performance, Adam performs better than CMA-ES.
- Ablation study: Cross entropy and hinge loss significantly outperform negative accuracy, with cross entropy slightly better than hinge loss on SST-2 and AG’s News.The paper explains that accuracy rewards are sparse and provide less informative optimization directions in few-shot settings.
- Ablation study: The best subspace dimensionality differs by task: d = 200 performs best on SST-2, whereas d = 500 performs best on AG’s News development sets.Very small subspaces may miss good solutions, while large subspaces may generalize poorly.
- Ablation study: Prompt length L = 50 achieves the best development accuracy on both SST-2 and AG’s News, while shorter prompts converge faster on training sets.Prompt length determines the original parameter-space dimensionality as D = L × 1024.
5. Discussion and Future Work
The discussion situates black-box tuning within derivative-free optimization and prompt-based learning, while identifying projection design and prompt engineering as directions for improvement.
- Derivative-Free Optimization: Black-box tuning uses random embedding and CMA-ES to optimize in a lower-dimensional subspace without backpropagation.The method differs from prior approaches through uniform random projection and CMA-ES optimization.
- Derivative-Free Optimization: The randomly generated subspace can be sub-optimal, motivating sequential embeddings and more advanced projection-matrix construction.The paper also notes that multi-task supervision can produce better and smaller subspaces.
- Prompt-Based Learning: Black-box tuning shares prompt tuning’s efficient serving and mixed-task inference because it tunes only the continuous prompt.It additionally inserts hand-crafted hard prompt tokens.
- Prompt-Based Learning: The reported performance is a lower bound because templates and label words are hand-crafted without trial-and-error.Prompt engineering, label-word engineering, prompt pre-training, and prompt ensembling are left for future work.
A. Additional Experimental Results
Additional experiments examine random-projection distributions, subspace and prompt dimensions, and optimizer behavior in low-dimensional subspaces.
- Random Projection: Uniform random projection achieves faster convergence than normal random projection on SST-2 and AG’s News.Both projection choices achieve considerable cross-entropy loss within reasonable budgets.
- Random Projection: The random projection matrix A determines whether and how difficult it is to find a good solution in the generated subspace.The study compares normal sampling N(0, 1/d) with a uniform distribution for initializing A.
- Ablation Study: The 16-shot ablation evaluates subspace dimensionality and prompt length while matching the experimental setting of Table 3.The corresponding figure isolates these two hyperparameters in the 16-shot setting.
- Optimizer Comparison: CMA-ES is more efficient and stable than Adam in low-dimensional subspaces.The comparison contrasts Adam in the original prompt space with CMA-ES in the generated subspace.
B. Parallel Evaluation
Parallel evaluation improves training efficiency by evaluating an optimization population simultaneously, especially with smaller training data or larger server batches.
- Parallel Evaluation: Parallel evaluation assesses the entire population in parallel to improve black-box tuning efficiency.Asynchronous queries can simulate parallel evaluation when each API call has a limited batch size.
- Parallel Evaluation: 100% accuracy on the SST-2 training set is achieved with population sizes 20 and 25 in 300 API-call iterations.This result is reported for parallel evaluation.
C. Estimation of Uploaded/Downloaded Data Size
The data-size analysis separates uploads of training inputs and continuous prompts from downloads of model outputs, while noting that feature-based estimates are upper bounds without compression.
- Data Transfer: Black-box tuning uploads training samples and a continuous prompt, then downloads candidate-word output logits.Training samples contain input IDs and attention masks, while the prompt contains hundreds of floating-point values.
- Data Transfer: For the SST-2 16-shot example, input IDs require approximately 2.9KB and attention masks approximately 1.5KB.These estimates use a 32 × 47 input shape with unsigned shorts for IDs and booleans for masks.
- Data Transfer: A 500-dimensional prompt adds approximately 2KB of upload data in the SST-2 16-shot example.The prompt is represented with 4-byte floating-point values.
- Feature-Based Methods: Feature-based data-transfer estimates are upper bounds because they assume no data compression.Feature-MLP downloads the [CLS] representation, whereas Feature-BiLSTM downloads representations for all tokens.