PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Prompt-Based Models Really Understand the Meaning of their Prompts?

Albert Webson, Ellie Pavlick

arXiv:2109.01247v2cs.CL

TL;DR

This study tests whether prompt-based models understand prompts as meaningful task instructions by evaluating over 30 manually written NLI templates and 13 sets of language-model targets. Models often learn equally fast with irrelevant or misleading templates, while target-word choices can override prompt semantics, including in large and instruction-tuned models.

  • Problem

    The paper examines whether models understand prompts as meaningful task instructions in ways analogous to humans, rather than merely benefiting from prompt-based learning.

  • Method

    The authors train prompt-based models on NLI using over 30 manually written templates and 13 sets of language-model targets, evaluating prompt effects without self-labeling or ensembling augmentations.

  • Results

    Models often learn equally fast with misleading and irrelevant templates as with instructive ones, while target-word choices override prompt meaning; this pattern appears across models and datasets in few-shot settings.

  • Takeaways & Limitations

    The results contradict the common assumption that prompts function as semantically meaningful task instructions and that high-performing prompts require domain expertise.

  • Takeaways & Limitations

    The study does not resolve how much NLI measures language understanding, and the interaction between prompt semantics and models’ inductive biases remains a future-work question.

Abstract

from arXiv · show

Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models. It is commonly argued that prompts help models to learn faster in the same way that humans learn faster when provided with task instructions expressed in natural language. In this study, we experiment with over 30 prompt templates manually written for natural language inference (NLI). We find that models learn just as fast with many prompts that are intentionally irrelevant or even pathologically misleading as they do with instructively "good" prompts. Further, such patterns hold even for models as large as 175 billion parameters (Brown et al., 2020) as well as the recently proposed instruction-tuned models which are trained on hundreds of prompts (Sanh et al., 2022). That is, instruction-tuned models often produce good predictions with irrelevant and misleading prompts even at zero shots. In sum, notwithstanding prompt-based models' impressive improvement, we find evidence of serious limitations that question the degree to which such improvement is derived from models understanding task instructions in ways analogous to humans' use of task instructions.

1 Introduction

The paper tests the assumption that prompts function as meaningful task instructions by evaluating diverse NLI prompts across models and shot settings. It finds that models often learn equally quickly from irrelevant or misleading templates, challenging instruction-based explanations of prompt gains.

  • Prompt-based reformulations have improved zero-shot and few-shot NLP performance, motivating the hypothesis that prompt text provides semantically meaningful task instructions.
  • Expert-written prompts often outperform automatically searched prompts across SuperGLUE tasks, reinforcing the belief that meaningful instructions drive performance.
  • Over 30 manually written NLI templates and 13 target-word sets produce more than 390 prompt combinations evaluated in zero-shot and few-shot settings.
  • Models usually learn equally fast with irrelevant or misleading templates as with instructively good templates.
  • Models from 235 million to 175 billion parameters, including instruction-tuned models, exhibit this insensitivity to template meaning.
  • Models are more sensitive to LM target-word choices than to instruction-template meaning, providing limited evidence for human-like understanding of task instructions.

2 Related Work

Related work distinguishes discrete, priming, and continuous prompts and examines how examples, instructions, and prompt optimization affect few-shot learning. This paper focuses on manually controlled discrete prompts to isolate the role of wording and semantics.

  • 2.1 Prompt-Based Models: Prompting can refer to discrete templates, priming with labeled examples, or continuous prompts whose learned tokens need not correspond to real vocabulary words.
  • 2.1 Prompt-Based Models: Discrete prompts convert masked target words into class predictions through a predefined mapping, and their few-shot performance can exceed that of much larger language models.
  • 2.1 Prompt-Based Models: Priming prepends k labeled examples without updating model parameters, but prior work found strong performance mainly for the largest GPT-3 model.
  • 2.1 Prompt-Based Models: The study focuses on discrete prompts because their wording and semantics can be manually written and controlled across k = {0, 4, 8, 16, 32, 64, 128, 256} shots.
  • 2.2 Related Findings: Prior work suggests priming examples help models more than task definitions, while models may remain indifferent to whether instructions describe the task correctly.
  • 2.2 Related Findings: Concurrent work likewise questions whether meaningful instructions are necessary, reporting that models can learn with incorrect as well as correct priming labels.

3 Overall Setup

The study evaluates manually controlled discrete prompts for NLI across baseline, instruction-tuned, and very large models, using few-shot trajectories and reproducible statistical comparisons. RTE is the primary benchmark, with additional datasets used to assess qualitative consistency.

  • 3 Overall Setup: The implementation measures prompt effects through rank classification of LM target words while omitting self-labeling and prompt ensembling augmentations.
  • 3 Overall Setup: ALBERT is used as the baseline because preliminary comparisons found it consistently yielded the best performance among the tested models.
  • 3 Overall Setup: At 32 shots, the baseline achieved 70.22% median RTE validation accuracy, comparable to the previously reported 69.8%.
  • 3 Overall Setup: The study evaluates T0 models of 3B and 11B parameters against non-instruction-tuned T5 LM-Adapted, plus GPT-3 175B through priming.
  • 3 Overall Setup: NLI asks whether a premise entails a hypothesis, and the selected setup holds out all NLI prompts and datasets from T0 training for fair comparison.
  • 3 Overall Setup: Experiments use RTE validation accuracy, four shared random seeds, recorded example indices, and corrected parametric and nonparametric significance tests.
  • 3 Overall Setup: Additional ANLI, HANS, and WSC experiments show no qualitative difference from the main RTE results, although ANLI requires many more shots to exceed random accuracy.

4 Effect of Templates

The study tests whether prompt templates function as meaningful task instructions by comparing instructive, irrelevant, misleading, and null templates across NLI learning trajectories. Many pathological prompts perform similarly to instructive prompts, although effects vary by model, template, and shot count.

  • 4.1 Method: The authors evaluate more than 30 manually written NLI templates spanning instructive, misleading-moderate, misleading-extreme, irrelevant, and null categories.Prompts combine templates with predefined target words; reported template comparisons use “yes”/“no” targets.
  • 4.2 Result: Irrelevant templates produce no practical performance difference from instructive templates at any shot count across all tested models and datasets, including GPT-3.The comparison is based on small absolute effect sizes rather than statistical significance alone.
  • 4.2 Result: Misleading-moderate and misleading-extreme templates have no consistent performance ordering across models, but instructive templates outperform at least one misleading category for each model.T0 favors moderately misleading prompts, ALBERT and T5 3B favor extremely misleading prompts, and T5 11B and GPT-3 are comparable across the two categories.
  • 4.2 Result: Null templates perform much worse overall, although some ALBERT null templates learn nearly as fast as instructive templates after 32 shots.Other null-template permutations remain substantially worse.
  • 4.2 Result: At zero shots, non-instruction-tuned models perform only marginally above random, whereas T0 variants often show no practical difference between instructive and misleading prompts.T0++ is the exception with statistically different performance across all prompt categories, though pathological prompts can still perform well.
  • 4.3 Discussion: The discussion concludes that scaling does not resolve prompt-semantic insensitivity, while instruction tuning improves robustness but can make models less sensitive to distinctions between proper and pathological prompts.T0++ improves semantic sensitivity at zero shots, yet numerous pathological prompts still match proper prompts; T0 also improves performance and variance robustness relative to T5.

5 Effect of Target Words

Models are substantially more sensitive to the target words than to the semantic content of prompt templates. Yes/no targets support faster learning, while arbitrary or reversed targets can impede performance even with instructive templates.

  • Target-word categories: Four target categories were tested: yes-no, yes-no-like, arbitrary, and reversed word pairs.Arbitrary targets have no semantic relation to entailment, while reversed targets invert the intuitive label mapping.
  • Learning effects: Models trained with yes-no targets learn faster than those trained with yes-no-like, arbitrary, or reversed targets.This pattern holds for both ALBERT and T0.
  • Learning effects: 22.2% separates the median accuracies of “yes”/“no” and “no”/“yes” targets at 32 shots.This difference is larger than the effect of varying template categories.
  • Template-target interaction: Across template-target combinations, target-word choice matters much more than template meaning.Irrelevant or misleading templates paired with yes-no targets often outperform instructive templates paired with arbitrary targets.
  • Template-target interaction: Yes-no-like targets remain behind yes-no targets until 256 shots, and appending answer hints often reduces performance.Misleading templates with yes-no targets can substantially outperform instructive templates with arbitrary targets.

6 General Discussion

The paper tests whether models treat prompts as meaningful task instructions analogous to human instruction use. Across alternative interpretations, the results indicate prompt sensitivity without human-like instruction understanding.

  • Most models are insensitive to whether templates provide instructive, misleading, irrelevant, or no task guidance.This pattern motivates the paper’s central question about whether prompt-based performance reflects meaningful instruction understanding.
  • Target-word choices can override template semantics, with irrelevant prompts paired with “yes”/“no” outperforming instructive prompts paired with arbitrary targets.
  • Lack of Competence: The zero-shot results of non-instruction-tuned models are ambiguous because near-random accuracy may reflect insufficient NLI competence rather than misunderstanding instructions.The study therefore emphasizes few-shot settings, where models achieve substantially better accuracy.
  • Alternative Interpretations: Preliminary transfer to HANS makes it unlikely that models are entirely incompetent at entailment reasoning or rely solely on NLI heuristics.
  • Lack of Compliance: Models cannot simply be ignoring prompts, because instructive and irrelevant templates consistently outperform misleading and null templates in learning speed.
  • Alternative Interpretations: Differences across prompts may arise from interactions between prompt features and spurious data features rather than human-like interpretation of instructions.Punctuation affects irrelevant templates strongly, while instructive templates may suppress that effect, leaving prompt semantics as a possible influence on inductive biases.

7 Conclusion

The conclusion reports that prompt-based models often succeed with irrelevant or misleading templates, challenging the view that prompts function as semantically meaningful task instructions. It nevertheless treats instruction learning as an important direction for future research.

  • Over 30 templates and 13 target-word sets show that models often learn equally fast with misleading or irrelevant templates as with instructive ones.
  • Target-word choices override the meaning of overall prompts across the models and datasets tested in few-shot settings.
  • These results contradict the assumption that prompts are semantically meaningful instructions and that high-performing prompts require domain expertise.
  • The authors conclude that current models are far from fully understanding prompt meaning while identifying instruction learning as an important research direction.

Ethical Considerations

The paper warns that prompt-based models can create an illusion of instruction following, with implications for interpretability and increasingly commercial use. This interface may mislead nonexpert users.

  • Even the largest language models can appear to follow instructions without actually following them.
  • Prompt-based interfaces may make models more accessible to lay users while making misleading instruction-following behavior more pernicious.

A Effect of Punctuation

The appendix examines how punctuation and API or priming choices affect prompt-based NLI results. Effects vary by model and template, showing that punctuation alone does not explain performance differences.

  • Effect of Punctuation: Quotation and question marks substantially affect irrelevant-template performance for ALBERT and T0.Removing these marks causes substantial performance drops in both models.
  • Effect of Punctuation: For T5 LM-Adapted (3B), irrelevant templates show no significant qmark difference, while instructive templates without qmarks outperform instructive templates at 32 and 64 shots.
  • Effect of Punctuation: Punctuation alone cannot explain all differences because misleading prompt categories lack a consistent pattern and punctuation interacts with template semantics and data features.
  • Effect of Punctuation: Null-template performance depends on premise–hypothesis ordering: one arrangement performs second-best, while swapping the order is worst among null templates.
  • GPT and Priming Setup: The GPT experiments use OpenAI’s davinci model, corresponding to an estimated 175 billion parameters, with priming rather than fine-tuning.
  • GPT and Priming Setup: The Instruct Series was excluded because its training data and prompts were undisclosed, could include NLI, and might change while in beta.
  • GPT and Priming Setup: Priming reproduces GPT-3’s reported zero-shot RTE performance only with its exact prompt; other prompts perform at random at zero shots.The reported 32-shot result could not be reproduced because the API’s 2049-token context limit is insufficient for RTE.
  • GPT and Priming Setup: GPT priming results are not directly comparable to gradient-updated few-shot models because each inference example uses a different set of priming examples.

I Zero-Shot Results (Figure 5)

Zero-shot performance remains substantial across instructive, irrelevant, misleading, and null prompts, including for T0 models. The reported results therefore show that prompt category alone does not determine prediction quality.

  • T0 (3B) achieves accuracies from 0.5523 to 0.7148 across instructive, irrelevant, and null templates.The highest listed value is 0.7148 for the instructive MNLI_YN template, while the lowest is 0.5523 for an irrelevant template.
  • T0 (11B) reaches 0.8051 on multiple instructive templates while also scoring 0.7184 on a misleading-moderate template.The listed accuracies vary across templates within the same model category.
  • ALBERT’s best irrelevant, misleading, and null prompts use semantically unrelated or minimally informative wording rather than explicit NLI instructions.The examples include a zoning statement, a question about whether a paragraph starts with “the,” and a bare mask between premise and hypothesis.

K Preliminary Results on HANS

On HANS, few-shot RTE-trained ALBERT performs similarly with instructive and irrelevant templates, whereas misleading templates perform worse, particularly for lexical non-entailment.

  • Instructive and irrelevant templates generally produce similar performance on HANS.
  • Misleading templates fare worse than instructive and irrelevant templates, especially for lexical non-entailment.The figure distinguishes lexical, subsequence, and constituency cases, and entailment versus non-entailment labels.
  • The HANS evaluation uses few-shot RTE-trained ALBERT in a zero-shot transfer setting.The figure reports zero-shot performance on HANS after RTE training.

L Preliminary Results on Winograd

Winograd results show no statistically significant difference between instructive and misleading-extreme templates for T0. The listed templates illustrate that misleading prompts can preserve similar zero-shot accuracy.

  • The instructive templates include several alternative yes-or-no formulations for resolving a pronoun–referent relation.Examples ask whether the pronoun is the same as, refers to, means, or is equivalent to the referent.
  • The misleading-extreme templates replace the core relation with unrelated actions or social relations, such as eating cakes, marrying, or being friends.
  • T0 shows no statistically significant difference between instructive and misleading-extreme templates on the Winograd Schema Challenge.
Loading 2109.01247v2…