PaperlayerView exact revision on arXiv

Source-linked AI summary

Noisy Channel Language Model Prompting for Few-Shot Text Classification

Sewon Min, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer

arXiv:2108.04106v3cs.CLcs.AI

TL;DR

Few-shot language-model prompting can be unstable, motivating an alternative to direct label prediction. The paper applies noisy channel models through demonstrations and prompt tuning, finding stronger and more stable performance, especially with imbalance or unseen labels. It also identifies scope limitations for non-classification tasks and masked language models.

  • Problem

    Few-shot prompting methods that directly predict label tokens can show high variance across verbalizers and seeds, with poor worst-case performance.

  • Method

    The paper computes the conditional probability of the input given the label and applies noisy channel models to in-context demonstrations and continuous prompt tuning.

  • Results

    Across eleven datasets, channel models significantly outperform direct counterparts, with lower variance and higher worst-case accuracy; channel prompt tuning is especially effective under imbalance or unseen-label generalization.

  • Takeaways & Limitations

    Channel prompt tuning is preferred when training data is small or imbalanced, or when the model must generalize to unseen labels.

  • Takeaways & Limitations

    The approach is harder to use for non-classification tasks where modeling prior distributions is non-trivial, and adapting it to masked language models remains an open question.

Abstract

from arXiv · show

We introduce a noisy channel approach for language model prompting in few-shot text classification. Instead of computing the likelihood of the label given the input (referred as direct models), channel models compute the conditional probability of the input given the label, and are thereby required to explain every word in the input. We use channel models for recently proposed few-shot learning methods with no or very limited updates to the language model parameters, via either in-context demonstration or prompt tuning. Our experiments show that, for both methods, channel models significantly outperform their direct counterparts, which we attribute to their stability, i.e., lower variance and higher worst-case accuracy. We also present extensive ablations that provide recommendations for when to use channel prompt tuning instead of other competitive methods (e.g., direct head tuning): channel prompt tuning is preferred when the number of training examples is small, labels in the training data are imbalanced, or generalization to unseen labels is required.

1 Introduction

The paper introduces noisy channel models for few-shot language-model prompting, contrasting them with direct models and evaluating them across demonstration and prompt-tuning methods. Channel models are reported as more stable and particularly effective under imbalance and unseen-label generalization.

  • Few-shot prompting can be unstable across verbalizers and random seeds, with worst-case performance often near random.
  • Channel models compute the input likelihood given the output, requiring them to explain every input word rather than predict only the label token.This approach is inspired by noisy channel models in machine translation.
  • The paper attributes channel models’ stronger performance to lower variance and higher worst-case accuracy across verbalizers and seeds.
  • Channel prompt tuning significantly outperforms direct baselines when training data is imbalanced or unseen-label generalization is required.
  • Channel models significantly outperform direct counterparts for both demonstration-based methods and prompt tuning.

2 Related Work

The paper places its noisy channel prompting approach within prior work on probabilistic language models, few-shot learning, and prompting. It distinguishes its setup from related label-conditioning and zero-shot channel-like methods.

  • Direct models compute P(y|x), whereas noisy channel models maximize P(x|y)P(y).
  • The paper explores channel models with a large language model across a wide range of prompt-based few-shot text-classification tasks.
  • Prior few-shot learning approaches include semi-supervised data augmentation, consistency training, meta-learning, and prompting with concatenated training examples.
  • Unlike this work, Tam et al. use a label-conditioning objective for masked language models but retain a discriminative objective and direct-model inference.
  • The paper presents its approach as the first noisy channel model for few-shot language-model prompting for classification.

3 Formulation

The formulation compares direct and channel probabilities for text classification, using verbalizers to map labels into natural-language expressions. It also defines Direct++ as a calibrated direct variant for demonstration methods.

  • The task maps natural-language inputs x from X to labels in C through a task function f : X −→ C.
  • Direct models compute label distributions P(ci|x), while channel models reparameterize them as P(x|ci)P(ci) using Bayes’ rule.
  • Direct++ computes P(ci|x)P(ci|NULL) instead of P(ci|x) to address language-model calibration and competition among equivalent label strings.
  • Channel models assume a uniform label prior P(ci) = 1/|C| and therefore compute only P(x|ci).

4 Method

The method applies direct and noisy-channel language-model objectives to few-shot classification, using demonstrations and limited-parameter alternatives. Channel models score the input conditioned on labels, while a proposed ensemble-style method processes each training example separately.

  • Parameter-efficient methods: The method studies settings with no trainable parameters or very small parameter updates, typically below 0.01% of the language model.This reduces the cost of updating and storing task-specific parameters.
  • Demonstration methods: Direct models estimate label-token probabilities given the input, whereas channel models estimate input probabilities given the label.The channel objective requires the model to explain every input word.
  • Demonstration methods: Few-shot demonstrations prepend K labeled training examples so the language model can infer the task setup.The framework supports both direct and channel prediction objectives.
  • Ensemble-based demonstrations: The proposed alternative runs the language model K times, conditioning on one training example at a time, and multiplies the resulting probabilities.Compared with concatenation, it uses O(K) rather than O(K^2) memory and removes dependence on example ordering.

4.2 Tuning methods

The paper compares head, transformation, and prompt tuning as limited-parameter alternatives to full language-model fine-tuning. Prompt tuning freezes the language model and learns only continuous prompt embeddings for direct or channel scoring.

  • Overview: The tuning methods update a very limited number of parameters compared with typical fine-tuning.The methods include direct head and transformation tuning, plus direct and channel prompt tuning.
  • Head tuning: Head tuning updates the language-model head while freezing all other parameters.The head maps final hidden representations to vocabulary logits.
  • Transformation tuning: Transformation tuning learns a matrix U applied to the frozen language-model head, with U initialized as the identity matrix.Other language-model parameters, including the original head, remain frozen.
  • Prompt tuning: Prompt tuning prepends learned continuous prompt embeddings while freezing the language-model parameters.Direct models compute P(ci|x), whereas channel models compute P(x|ci) using the learned prompts.

5 Experimental Setup

The experiments evaluate parameter-efficient few-shot classification across eleven datasets with varied class counts, realistic class imbalance, and multiple verbalizer and random-seed settings. Accuracy is reported using both average and worst-case performance.

  • Datasets: Eleven text classification datasets cover tasks ranging from 2 to 14 classes.The benchmark includes SST-2, SST-5, MR, CR, Amazon, Yelp, TREC, AGNews, Yahoo, DBPedia, and Subj.
  • Few-shot setting: Few-shot experiments primarily use K = 16 examples, with ablations at K = 4, 16, 64, and Full.Examples are sampled uniformly from the true training distribution without assuming equal examples per label.
  • Protocol: The experimental protocol follows prior-work hyperparameters and omits a held-out validation set because the training data are very limited.The implementation is reported as reproducible from the authors' GitHub repository.
  • Models and metrics: The evaluation uses GPT-2, primarily GPT-2 Large, while ablations vary model size from Small through X-Large.The experiments use accuracy for every dataset.
  • Evaluation: Results aggregate four verbalizers, five training-data sampling seeds, and four training seeds.The paper reports average accuracy and worst-case accuracy to capture few-shot variability.

6 Experimental Results

Across demonstration and tuning methods, channel models generally outperform direct models, with the strongest advantages under instability, limited data, imbalance, and unseen-label generalization. Ablations identify conditions favoring channel prompt tuning versus head tuning or full finetuning.

  • Demonstration methods: Direct++ significantly outperforms the naive direct model across all evaluated setups.The comparison uses P(c_i|x) / P(c_i|NULL) rather than only P(c_i|x).
  • Demonstration methods: 3.1% and 7.2% absolute gains distinguish the strongest channel model from the strongest direct model in average and worst-case accuracy, respectively.
  • Demonstration methods: Channel models’ strong performance is associated with lower variance and higher worst-case accuracy across verbalizers and seeds, despite direct++ often achieving the highest best-case accuracy.
  • Tuning methods: 13.3% and 23.5% absolute gains separate channel from direct models under prompt tuning in average and worst-case accuracy, respectively.Channel models consistently outperform direct models across all datasets in this setting.
  • Tuning methods: Head tuning significantly outperforms direct prompt tuning in all cases and exceeds channel prompt tuning particularly on TREC and Subj.Channel prompt tuning nevertheless outperforms direct head tuning on most datasets, with the largest gains on Yahoo and DBPedia.
  • Ablations: With K ≤16, channel prompt tuning performs strongly, whereas head tuning surpasses channel head tuning at K = 64 and direct prompt tuning and head tuning surpass it at K = Full.Channel models are more competitive with smaller K and less competitive with larger K.
  • Ablations: At K = Full, direct prompt tuning matches all finetuning while using fewer trainable parameters, but in few-shot settings all finetuning significantly outperforms other methods.The result contrasts with analyses favoring fewer trainable parameters when data is scarce.
  • Ablations: When p− is small, channel prompt tuning is insensitive to imbalance, significantly outperforms direct models, and exceeds all finetuning when p− < 0.25.Near p− = 0.5, direct head tuning matches or outperforms channel prompt tuning.

7 Discussion & Conclusion

The discussion identifies conditions favoring channel prompt tuning and highlights limitations involving task type and language-model architecture. Channel prompt tuning is preferred with few examples, imbalanced data, or unseen-label generalization, while masked-LM use remains open.

  • K is small: Channel prompt tuning is more competitive when training examples are fewer, partly because channel models are more stable and provide word-by-word training signals.The discussion links these advantages to lower variance, higher worst-case accuracy, and richer input-level signals in the low-data regime.
  • Data is imbalanced or |C| is large: When training data is even slightly imbalanced, no direct models are competitive, whereas channel prompt tuning is less sensitive because labels are only conditioning variables.The authors identify label-prior reliance in the LM head as a possible reason for this difference.
  • Generalization to unseen labels is required: All direct models cannot predict unseen labels, while channel models can, likely because channel models indirectly model the label space.The discussion connects this pattern with prior findings that channel models are more competitive under distribution shift.
  • Task is closer to language modeling: Head tuning outperforms prompt tuning when tasks differ substantially from language modeling, such as TREC and Subj.The authors suggest this may reflect the benefit of directly updating language-model parameters for such tasks.
  • Limitations and future work: Channel models are harder to use for non-classification tasks because modeling prior distributions is non-trivial.A proposed direction is obtaining the prior with a separate model and incorporating it into the conditional language model.
  • Limitations and future work: Using channel models with masked language models remains an open question because their pretraining makes generation of long sentences difficult without parameter updates.The discussion mentions label-conditioning objectives as one channel-like approach for existing masked language models.

A Samples & Verbalizers

The appendix provides dataset samples and documents the verbalizers used to express labels in the experiments.

  • Samples & Verbalizers: Table 10 provides samples from each dataset, while Table 6 lists four verbalizers for every dataset.The verbalizers mainly come from Gao et al. (2021) and label words in the original data.

B Implementation Details

The implementation appendix records software, sequence-length, optimization, learning-rate, verbalizer, sample, and evaluation-reporting details used across the experiments.

  • Implementation Details: The experiments use PyTorch and Hugging Face Transformers, with batch size and sequence length varying between short- and long-text datasets.Concat-based demonstrations multiply sequence length by the number of training examples, capped at GPT-2’s 1024-token limit.
  • Implementation Details: Finetuning runs for 100 global steps with Adam, no weight decay, and no warmup, while learning rates are selected from {0.1, 0.01, 0.001} using average training loss.The same selection procedure is applied to head, transformation, and prompt tuning.
  • Implementation Details: Tables 7, 8, and 9 report learning rates for the tuning models in Table 4, Figure 6, and Figure 3, respectively.
  • Implementation Details: Table 6 lists four verbalizers per dataset, and Table 10 provides dataset samples with c indicating the label.
  • Implementation Details: Tables 11–13 report average, standard deviation, best-case, and worst-case accuracy for demonstration and tuning methods.Table 13 distinguishes head, transformation, and prompt tuning.
  • Implementation Details: Figure 6 reports average and worst-case accuracy across GPT-2 Small through GPT-2 X-Large, with each model run 20 times.The runs cover four verbalizers and five data seeds; trends are consistent across model sizes.

C Additional Results

Additional results show that channel models are more stable across evaluation conditions, while performance trends remain consistent across language-model sizes.

  • Additional Results: Channel models consistently achieve significantly lower variance and higher worst-case accuracy, although direct models often achieve the best-case accuracy.Channel models nevertheless outperform direct models on average.
  • Additional Results: Average and worst-case accuracy trends are fairly consistent across varying language-model sizes, regardless of whether channel prompt tuning or direct head tuning performs best.
Loading 2108.04106v3…