PaperlayerView exact revision on arXiv

Source-linked AI summary

The language of prompting: What linguistic properties make a prompt successful?

Alina Leidinger, Robert van Rooij, Ekaterina Shutova

arXiv:2311.01967v1cs.CLcs.AIcs.LG

TL;DR

The paper asks how linguistic properties of prompts relate to LLM performance, an issue not systematically understood despite sensitivity to prompt choice. It tests semantically equivalent prompts with controlled grammatical and lexical variation, finding substantial performance differences not generally explained by common prompt measures or eliminated by instruction-tuning and scale.

  • Problem

    The relationship between linguistic properties of prompts and task performance is not systematically understood, despite observed sensitivity to prompt choice.

    Source: §1 · p. 1

  • Method

    The study evaluates five pretrained and instruction-tuned LLMs on manually constructed, semantically equivalent prompts varying grammatical properties and synonyms.

    Source: §1 · p. 1, §4.3 · p. 4

  • Results

    There is no favored sentence structure across models and tasks; prompt performance varies considerably, transfers poorly across datasets and models, and is not explained by perplexity, ambiguity, or word frequency.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Single fixed prompts and best-case reporting have limitations, motivating a more robust and comprehensive evaluation framework for prompting research.

    Source: §1 · p. 2

    The experiments cover zero-shot LLMs up to 30b parameters and exclude larger open-source models and GPT-3 partly because of computational cost.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The latest generation of LLMs can be prompted to achieve impressive zero-shot or few-shot performance in many NLP tasks. However, since performance is highly sensitive to the choice of prompts, considerable effort has been devoted to crowd-sourcing prompts or designing methods for prompt optimisation. Yet, we still lack a systematic understanding of how linguistic properties of prompts correlate with task performance. In this work, we investigate how LLMs of different sizes, pre-trained and instruction-tuned, perform on prompts that are semantically equivalent, but vary in linguistic structure. We investigate both grammatical properties such as mood, tense, aspect and modality, as well as lexico-semantic variation through the use of synonyms. Our findings contradict the common assumption that LLMs achieve optimal performance on lower perplexity prompts that reflect language use in pretraining or instruction-tuning data. Prompts transfer poorly between datasets or models, and performance cannot generally be explained by perplexity, word frequency, ambiguity or prompt length. Based on our results, we put forward a proposal for a more robust and comprehensive evaluation standard for prompting research.

1 Introduction

The introduction asks whether prompt performance reflects linguistic properties associated with training data, given substantial variability across prompting strategies. The paper studies semantically equivalent prompts with controlled linguistic variation and finds that performance is not generally explained by common linguistic or prompt-level measures.

1 Introduction

  • Although increasing model scale often improves average downstream performance, performance variability across samples and instructions raises questions about how models respond to language encountered during training.The introduction asks whether simpler, frequent, less ambiguous, lower-perplexity prompts perform better.Source: §1 · p. 1
  • The study manually constructs parallel prompts varying in grammatical and lexical properties and evaluates five pretrained or instruction-tuned LLMs across six datasets and three tasks.The study uses zero-shot evaluation focused on instruction understanding.Source: §1 · p. 1
  • Prompt structure produces large performance variation, including for instruction-tuned models on seen tasks, while perplexity, length, ambiguity, and word frequency do not generally explain performance.More complex structures and rarer synonyms sometimes outperform simpler formulations.Source: §1 · p. 2
  • The authors argue that evaluating models on single fixed prompts and reporting best-case performance has limitations, and propose a more comprehensive prompting evaluation framework.They note that prompts transfer poorly between datasets and models, and that instruction-tuning or model scaling does not eliminate considerable variation.Source: §1 · p. 2

2 Related work

Prior work documents sensitivity to prompt formulations and proposes ways to generate or calibrate prompts, but robust evaluation across formulations remains limited. This study instead uses manually crafted prompts to isolate linguistic variation.

2 Related work

3 Tasks and datasets

The study uses paired datasets across sentiment classification, natural language inference, and question answering, selected to support above-chance zero-shot performance and varied evaluation settings.

3 Tasks and datasets

  • Six datasets cover sentiment classification, natural language inference, and question answering; they include both cross-dataset comparisons and held-out or supervised settings for OPT-IML.The sets are SST-2 and IMDB; RTE and CommitmentBank; and BoolQ and ARC-E.Source: §3 · p. 3

4 Method

The method compares pretrained and instruction-tuned LLMs using zero-shot accuracy on controlled prompt variants. Linguistic features are changed systematically, and statistical tests and correlations assess performance differences.

4.1 Models

  • The study compares pretrained LLaMA and OPT models with instruction-tuned OPT-IML models, including 1.3b and 30b variants.The models are evaluated using the HuggingFace transformers library.Source: §4.1 · p. 3

4.2 Prompting setup

  • In zero-shot evaluation, prompts map classification labels to yes/no or yes/no/maybe and question-answer choices to letters, with a postamble added to each prompt.Each prompt is evaluated on 500 random samples per dataset; accuracy is the metric, and prediction uses the highest log probability among target words.Source: §4.2 · pp. 3–4

4.3 Linguistic variation in prompts

  • The researchers manually create parallel prompts that vary one grammatical property at a time, including mood, aspect, tense, modality, and synonym choice.They use nonparametric tests for prompt-set differences and correlation coefficients to assess relationships between accuracy and prompt length, perplexity, ambiguity, and frequency.Source: §4.3 · p. 4

5 Results

Prompt performance varies with linguistic structure, and prompts that work well for one dataset or model often transfer poorly. Neither instruction-tuning nor larger model size consistently removes that variability.

5.1 Performance variability

  • Questions and orders generally outperformed statements, but no mood consistently won; active voice lacked a cross-dataset advantage, except for OPT-IML 1.3b.Passive prompts generally did better on BoolQ for other models; 30b models favored active prompts only on RTE and ARC-E.Source: §5.1 · p. 4, Table 2 · p. 5, Table 3 · p. 5
  • Present tense was confirmed as best only for CB; on SST and IMDB no tense consistently led, and past or future prompts sometimes outperformed present.For OPT-IML 30b on IMDB, past/future prompts exceeded 96% accuracy versus 89% for present; BoolQ and ARC-E differences stayed under 2.5 percentage points.Source: §5.1 · p. 4, Table 2 · p. 5, Table 3 · p. 5
  • Modal-verb changes shifted SST accuracy by up to 10 points for LLaMA and 17 points for OPT 30b; QA and NLI prompt changes also produced drops up to 5 percentage points.The SST comparisons were ‘must’ versus ‘would’ for LLaMA and ‘may’ versus ‘might’ for OPT 30b.Source: §5.1 · pp. 4–5, Table 2 · p. 5, Table 3 · p. 5
  • Non-standard synonyms did not generally hurt performance and sometimes improved it; ‘appraisal’ and ‘commentary’ outperformed ‘review’ for SST and IMDB, while ‘appropriate’ often worked better on BoolQ and ARC-E.The ‘review’ result also held for OPT-IML 30b, despite its training on instructions that often used that word.Source: §5.1 · pp. 5–6, Table 2 · p. 5, Table 3 · p. 5

5.2 Prompt transfer

  • Prompts that were optimal for one model and dataset often transferred poorly; drops exceeded 20 percentage points on SST and IMDB or 5 points on BoolQ and CB.For example, OPT-IML 1.3b’s best IMDB prompt reached 97.6% accuracy but barely exceeded chance on SST.Source: §5.2 · p. 6

5.3 The relation between robustness and instruction-tuning

  • Instruction-tuning made seen-task performance more reliable than pre-training alone, but substantial variation remained, including 85–92% accuracy on IMDB for OPT-IML 30b.ARC-E, which was not part of OPT-IML 30b’s instruction-tuning tasks, varied by 5 percentage points.Source: §5.3 · p. 6

5.4 The relation between robustness and model size

  • Increasing model size did not consistently improve stability: with OPT-IML, the best-to-worst prompt gap widened from 5 to 12 percentage points on CB.Across comparisons of OPT-IML with OPT at 1.3b and 30b, the gap closed only on RTE, stayed stable on BoolQ, ARC-E, and SST, and widened on IMDB and CB.Source: §5.4 · pp. 6–7

6 Analysis

The analysis tests whether prompt accuracy tracks perplexity, synonym frequency or ambiguity, and prompt length. None provides a consistent performance explanation across models and tasks.

Prompt perplexity

  • Lower prompt perplexity did not significantly correlate with higher accuracy across models or datasets; for LLaMA 30b and OPT-IML 30b, higher-perplexity prompts often performed better.The authors conclude that the success of high- or low-perplexity prompts depends on the dataset–model combination.Source: §6 · p. 7

Frequency of synonyms

  • Frequent synonyms did not generally improve performance: frequency correlations varied across tasks and models, and were often near zero for BoolQ, ARC-E, and LLaMA.More frequent synonyms did better on RTE and CB except for LLaMA and OPT 1.3b on RTE.Source: §6 · p. 7

Ambiguity of synonyms

  • Word-sense ambiguity generally had little correlation with accuracy on BoolQ and ARC-E, while its association differed by task and model on CB, SST, and IMDB.OPT-IML 30b did better with less ambiguous prompts on SST and IMDB, whereas the opposite held for OPT-IML 1.3b.Source: §6 · p. 7

Prompt length

  • No model consistently performed better with longer or shorter prompts across tasks; length associations varied by model and dataset.For example, longer prompts significantly improved accuracy only on IMDB and BoolQ for LLaMA 30b and OPT 1.3b.Source: §6 · p. 7

7 Lessons learnt and way forward

The study finds substantial prompt-related performance instability, including weak transfer across datasets and models, and recommends evaluating broader, more systematically varied prompt sets.

7.1 Implications of our research

  • The findings challenge the assumption that lower-perplexity prompts with simple, frequent words and grammatical structures are reliably best.The authors call for further investigation of how model behaviour relates to language distributions in pretraining and instruction-tuning.Source: §7.1 · p. 7
  • Instruction-tuned models may still vary by over 5pp on seen tasks, despite overall gains in performance and robustness.Source: §7.1 · p. 7
  • Single-prompt or small-set benchmarking can be unreliable: prompt performance varies substantially, transfers poorly across datasets and models, and instruction-tuning does not resolve these problems.The authors also note that limited prompt transparency makes evaluations hard to reproduce.Source: §7.1 · pp. 7–8

7.2 Recommendations

  • Collect prompts representative of linguistic variability, using controlled paraphrasing to vary grammatical constructions systematically.Source: §7.2 · p. 8
  • Replace content words with synonyms to diversify vocabulary and enable controlled study of ambiguity, frequency, and model behaviour.Source: §7.2 · p. 8
  • Use sufficiently large prompt sets to estimate performance reliably; observed differences between prompts reached up to 17 pp.Report performance means and variances based on a large set of prompts.Source: §7.2 · p. 8
  • When selecting one evaluation prompt, treat it as a hyperparameter and choose it per model and dataset using held-out development data.Source: §7.2 · p. 8
  • Characterize prompt sets with standardized metrics, analyse their correlations with performance, and use mixed-effects models to study prompt effects by dataset and sample.Suggested metrics include perplexity, ambiguity, and their distributions.Source: §7.2 · p. 8

8 Conclusion

The conclusion describes a study of five pretrained and instruction-tuned LLMs using systematically varied prompts. It reports that prompt effects are substantial, transfer is poor, and common prompt characteristics do not generally explain performance.

  • The study evaluates five pretrained and instruction-tuned LLMs on parallel prompt sets varying in grammatical features and synonym use.Source: §8 · p. 9
  • No sentence structure performs best across models and tasks, and prompts generally transfer poorly across datasets and models.Source: §8 · p. 9
  • Performance variation can persist for larger instruction-tuned models on seen tasks and is not explained by perplexity, word-sense ambiguity, or word frequency.Source: §8 · p. 9

Limitations

The limitations section bounds the study to zero-shot models of up to 30 billion parameters and notes excluded models, settings, architectures, and languages.

  • The experiments cover zero-shot LLMs up to 30b parameters; larger open-source models and GPT-3 were excluded, partly because of computational cost and API log-probability limits.Source: Limitations · p. 9
  • Smaller OPT (IML) and LLaMA variants were omitted because initial zero-shot experiments found they did not perform significantly above chance across all tasks.Source: Limitations · p. 9
  • The study focuses on zero-shot prompting without demonstrations, excluding in-context learning and priming to isolate linguistic variation.Source: Limitations · p. 9
  • Other architectures, multilingual models, and languages with richer morphology are identified as directions for future experiments.Source: Limitations · p. 9
  • Vicuna and Alpaca were excluded because their OpenAI-generated or user-interaction training differed from OPT-IML's fine-tuning on NLP tasks, risking a skewed comparison.Source: Limitations · p. 9

Ethics & broader impact

The section recommends broader prompt evaluation and releases 550 prompts, while noting that prompt-engineering recommendations are localized to particular models or datasets. It also records supplementary evaluation details, including illustrative encoder-decoder results and per-prompt accuracy listings.

Ethics & broader impact

  • Evaluation reports should measure variability across prompts, seeds, and demonstration examples; the released 550-prompt set supports further research, while engineering recommendations remain localized to a model or dataset.The study also cautions that optimal prompts should not be assumed to transfer across datasets or domains.Source: Ethics & broader impact · p. 9

A Example prompt

  • The appendix introduces an example sentiment-classification prompt using a random SST-2 input sample.Source: A Example prompt · p. 14

B Average runtime

  • Evaluating 30b LLaMA, OPT, or OPT-IML across one dataset’s prompts took about two hours on four NVIDIA A100 GPUs; 1.3b OPT or OPT-IML took around 15 minutes on one GPU.Source: B Average runtime · p. 14

C Performance variability in encoder-decoder models

D Performance per prompt

  • Per-prompt accuracy is listed across models for sentiment analysis, NLI, and question answering, with prompt counts differing by linguistic property.Some prompts cannot be phrased in every property category and are omitted where inapplicable.Source: D Performance per prompt · p. 14

E Perplexity per category

The appendix lists perplexity scores for prompts across six datasets and provides per-prompt accuracy tables for sentiment classification, question answering, and natural language inference.

E Perplexity per category

Loading 2311.01967v1…