PaperlayerView exact revision on arXiv

Source-linked AI summary

Black-Box Tuning for Language-Model-as-a-Service

Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, Xipeng Qiu

arXiv:2201.03514v4cs.CLcs.AI

TL;DR

LMaaS makes large PTMs accessible through inference APIs, but unavailable gradients make conventional prompt optimization difficult, especially in high-dimensional spaces. BBT addresses this by optimizing continuous prompts with derivative-free search in a random low-dimensional subspace, and experiments report performance exceeding manual, in-context, prompt-tuning, and full-model-tuning approaches. The authors note that hand-crafted templates and label words leave reported performance as a lower bound, while random subspaces may be sub-optimal.

  • Problem

    LMaaS lacks PTM gradients, making direct optimization of continuous prompts with tens of thousands of parameters intractable for derivative-free methods.

    Source: §1 · p. 2

  • Method

    BBT iteratively queries the PTM inference API and optimizes a prepended continuous prompt within a smaller random linear-projection subspace using derivative-free optimization.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    BBT with RoBERTa on a few labeled samples outperforms manual prompting, in-context learning, prompt tuning, and full model tuning.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Black-box tuning enables users to optimize task-specific prompts locally on resource-limited devices without access to PTM parameters, gradients, or GPUs.

    Source: §1 · p. 2

    The reported performance is a lower bound because templates and label words are hand-crafted, while random-projection subspaces can be sub-optimal.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

Extremely large pre-trained language models (PTMs) such as GPT-3 are usually released as a service. It allows users to design task-specific prompts to query the PTMs through some black-box APIs. In such a scenario, which we call Language-Model-as-a-Service (LMaaS), the gradients of PTMs are usually unavailable. Can we optimize the task prompts by only accessing the model inference APIs? This paper proposes the black-box tuning framework to optimize the continuous prompt prepended to the input text via derivative-free optimization. Instead of optimizing in the original high-dimensional prompt space, which is intractable for traditional derivative-free optimization, we perform optimization in a randomly generated subspace due to the low intrinsic dimensionality of large PTMs. The experimental results show that the black-box tuning with RoBERTa on a few labeled samples not only significantly outperforms manual prompt and GPT-3's in-context learning, but also surpasses the gradient-based counterparts, i.e., prompt tuning and full model tuning.

1. Introduction

LMaaS exposes powerful but parameter- and gradient-inaccessible PTMs through black-box APIs, motivating prompt optimization using only inference queries. The paper proposes Black-Box Tuning, which searches a low-dimensional random subspace with derivative-free optimization and reports results surpassing manual, in-context, prompt-tuning, and full-model-tuning baselines.

  • LMaaS releases large PTMs through black-box APIs because running them locally is expensive or infeasible and parameters may remain closed.Source: §1 · p. 1
  • Hand-crafted prompts and in-context learning can solve simple tasks but cannot fully exploit labeled data.Source: §1 · p. 1
  • Continuous prompt tuning is deployment-efficient, but prior methods learn prompts through backpropagation unavailable in LMaaS.Source: §1 · p. 2
  • Derivative-free optimization is required without gradients, yet direct search over prompts with tens of thousands of parameters is intractable because convergence slows in high dimensions.Source: §1 · p. 2
  • BBT iteratively queries the inference API and optimizes a continuous prompt in a smaller random subspace rather than the original high-dimensional space.Source: Figure 1 · p. 1, §1 · p. 2
  • BBT with a few labeled samples outperforms manual prompting, in-context learning, prompt tuning, and full model tuning.Source: §1 · p. 2

2. Background

The background frames large PTMs as remotely served models and connects BBT to intrinsic dimensionality, prompt-based learning, and derivative-free optimization. These concepts explain why continuous prompts and low-dimensional search are relevant when model parameters and gradients are unavailable.

Large-Scale PTMs as APIs

  • Large-scale PTMs are commonly served through APIs because inference can be accelerated, models may remain closed, and local execution is expensive or infeasible.Source: §2 · p. 2

Intrinsic Dimensionality of PTMs

  • Intrinsic dimensionality is the minimum parameter count needed to obtain satisfactory solutions, representing an effective low-dimensional reparameterization.Source: §2 · p. 2
  • Randomly projected subspaces can retain substantial tuning performance, while multi-task supervision can compress task subspaces below 100 dimensions.Source: §2 · p. 3

Prompt-Based Learning

  • Prompt-based learning formulates downstream tasks as masked language modeling and can optimize continuous prompts while keeping PTM parameters unchanged.Source: §2 · p. 3

Derivative-Free Optimization

  • Derivative-free optimization updates sampled solutions using function values rather than gradients and includes evolutionary algorithms and Bayesian optimization.Source: §2 · p. 3

3. Approach

The approach formulates language understanding as black-box prompt optimization: continuous prompts are searched through API evaluations, using a low-dimensional projected space and derivative-free optimization.

Task formulation

  • Modified texts combine task templates and label words, while the PTM API returns masked-position logits used to evaluate prompt quality.The objective minimizes a loss between API predictions and golden labels over a batch.Source: §3.1 · p. 3

Black-box optimization

  • Derivative-free optimization is used because LMaaS inference APIs expose function evaluations but not gradients, while direct optimization in the high-dimensional prompt space is intractable.Large PTM prompt embeddings can have tens of thousands of dimensions.Source: §3.2 · pp. 3–4, §3.1 · p. 3

Prompt parameterization

  • The continuous prompt is optimized through an increment Az added to an initial prompt p0, rather than directly searching the full prompt space.The initialization may use randomly sampled vocabulary-token embeddings, or an NLI-pretrained prompt for sentence-pair tasks.Source: §3.2 · p. 3, §3.4 · p. 4

Loss functions

  • Cross entropy and hinge loss are considered alongside negative accuracy because accuracy rewards can be sparse and less informative with limited training data.The hinge-loss margin is set to γ = 2, and the three losses are compared in Figure 3.Source: §3.2 · p. 4

Derivative-free optimizer

  • CMA-ES optimizes the projected variables by sampling query populations from a multivariate normal search distribution and updating its mean, scale, and covariance from successful steps.The covariance matrix controls the distribution's shape, while the standard deviation controls step length.Source: §3.3 · p. 4

4. Experiments

Experiments evaluate black-box tuning with RoBERTaLARGE across few-shot language-understanding tasks, comparing it with gradient-based, gradient-free, manual-prompt, in-context, and feature-based methods. Ablations examine loss functions, subspace dimensionality, and prompt length.

Experimental setup

  • Experiments cover sentiment analysis, topic classification, natural language inference, and paraphrase across seven datasets.The datasets are SST-2, Yelp polarity, AG’s News, DBPedia, SNLI, RTE, and MRPC.Source: §4.1 · pp. 4–5
  • All overall-comparison results use pre-trained RoBERTaLARGE with 16 labeled samples per class and report mean and standard deviation over three splits.The few-shot protocol randomly selects k samples per class for training and another k for development.Source: §4.1 · p. 5

Overall comparison

  • Black-box tuning significantly outperforms four gradient-free methods and surpasses prompt tuning, p-tuning v2, and model tuning on average across seven tasks.The paper attributes the possible advantage over gradient-based optimization to DFO’s exploration mechanism, while presenting this as a suspicion rather than an established cause.Source: §4.2 · p. 6

Deployment comparison

  • Among the considered methods, only black-box tuning combines satisfactory performance with reasonable training time, memory footprint, and network load for LMaaS.Model tuning requires a full model copy per user, while gradient-based methods require gradients and cannot make the PTM serve as a service.Source: §4.2 · p. 6

Ablation study

  • Adam converges faster than CMA-ES on SST-2 and AG’s News, but CMA-ES achieves slightly better SST-2 development performance and better average test performance across seven tasks.On AG’s News development performance, Adam performs better than CMA-ES.Source: §4.3 · p. 6
  • Cross entropy and hinge loss significantly outperform negative accuracy, with cross entropy slightly better than hinge loss on SST-2 and AG’s News.The paper explains that accuracy rewards are sparse and provide less informative optimization directions in few-shot settings.Source: §4.3 · p. 7
  • The best subspace dimensionality differs by task: d = 200 performs best on SST-2, whereas d = 500 performs best on AG’s News development sets.Very small subspaces may miss good solutions, while large subspaces may generalize poorly.Source: §4.3 · p. 7
  • Prompt length L = 50 achieves the best development accuracy on both SST-2 and AG’s News, while shorter prompts converge faster on training sets.Prompt length determines the original parameter-space dimensionality as D = L × 1024.Source: §4.3 · p. 7

5. Discussion and Future Work

The discussion situates black-box tuning within derivative-free optimization and prompt-based learning, while identifying projection design and prompt engineering as directions for improvement.

Derivative-Free Optimization

  • Black-box tuning uses random embedding and CMA-ES to optimize in a lower-dimensional subspace without backpropagation.The method differs from prior approaches through uniform random projection and CMA-ES optimization.Source: §5 · pp. 8–9
  • The randomly generated subspace can be sub-optimal, motivating sequential embeddings and more advanced projection-matrix construction.The paper also notes that multi-task supervision can produce better and smaller subspaces.Source: §5 · p. 8

Prompt-Based Learning

  • Black-box tuning shares prompt tuning’s efficient serving and mixed-task inference because it tunes only the continuous prompt.It additionally inserts hand-crafted hard prompt tokens.Source: §5 · p. 8
  • The reported performance is a lower bound because templates and label words are hand-crafted without trial-and-error.Prompt engineering, label-word engineering, prompt pre-training, and prompt ensembling are left for future work.Source: §5 · p. 9

A. Additional Experimental Results

Additional experiments examine random-projection distributions, subspace and prompt dimensions, and optimizer behavior in low-dimensional subspaces.

Random Projection

  • Uniform random projection achieves faster convergence than normal random projection on SST-2 and AG’s News.Both projection choices achieve considerable cross-entropy loss within reasonable budgets.Source: A. Additional Experimental Results · p. 13
  • The random projection matrix A determines whether and how difficult it is to find a good solution in the generated subspace.The study compares normal sampling N(0, 1/d) with a uniform distribution for initializing A.Source: A. Additional Experimental Results · p. 13

Ablation Study

Optimizer Comparison

B. Parallel Evaluation

Parallel evaluation improves training efficiency by evaluating an optimization population simultaneously, especially with smaller training data or larger server batches.

Parallel Evaluation

  • Parallel evaluation assesses the entire population in parallel to improve black-box tuning efficiency.Asynchronous queries can simulate parallel evaluation when each API call has a limited batch size.Source: B. Parallel Evaluation · p. 14
  • 100% accuracy on the SST-2 training set is achieved with population sizes 20 and 25 in 300 API-call iterations.This result is reported for parallel evaluation.Source: B. Parallel Evaluation · p. 14

C. Estimation of Uploaded/Downloaded Data Size

The data-size analysis separates uploads of training inputs and continuous prompts from downloads of model outputs, while noting that feature-based estimates are upper bounds without compression.

Data Transfer

  • Black-box tuning uploads training samples and a continuous prompt, then downloads candidate-word output logits.Training samples contain input IDs and attention masks, while the prompt contains hundreds of floating-point values.Source: C. Estimation of Uploaded/Downloaded… · pp. 14–15
  • For the SST-2 16-shot example, input IDs require approximately 2.9KB and attention masks approximately 1.5KB.These estimates use a 32 × 47 input shape with unsigned shorts for IDs and booleans for masks.Source: C. Estimation of Uploaded/Downloaded… · p. 15
  • A 500-dimensional prompt adds approximately 2KB of upload data in the SST-2 16-shot example.The prompt is represented with 4-byte floating-point values.Source: C. Estimation of Uploaded/Downloaded… · p. 15

Feature-Based Methods

  • Feature-based data-transfer estimates are upper bounds because they assume no data compression.Feature-MLP downloads the [CLS] representation, whereas Feature-BiLSTM downloads representations for all tokens.Source: C. Estimation of Uploaded/Downloaded… · p. 15

Related papers

Loading 2201.03514v4…