PaperlayerView exact revision on arXiv

Source-linked AI summary

The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance

Abel Salinas, Fred Morstatter

arXiv:2401.03729v3cs.CLcs.AI

TL;DR

The paper asks whether prompt-construction choices affect LLM predictions, a question relevant as these models are integrated into labeling systems. It compares prompt variations and finds that many change predictions, while jailbreaks and some output formats can substantially reduce performance.

  • Problem

    The authors ask how reliably LLM responses hold up to prompt variations, since little attention has been paid to sensitivity to practitioners’ prompt-design choices.

    Source: §1 · p. 1

  • Method

    The study compares prompt-variation outputs and accuracy, and relates an instance’s propensity to change to annotator disagreement.

    Source: §3.4 · p. 6, §5 · p. 8

  • Results

    Prompt variations changed considerable proportions of predictions, while most perturbations kept similar accuracy; jailbreaks caused substantial losses, including over 10 percentage points for some low-refusal cases.

    Source: §5 · p. 8

  • Takeaways & Limitations

    In the evaluated settings, prompt variations—including output formats and jailbreaks—can affect labels and model accuracy.

    Source: §5 · p. 8

    The examination covers only ChatGPT and Llama, leaving sensitivity across other models, architectures, and training data for further investigation.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) are regularly being used to label data across many domains and for myriad tasks. By simply asking the LLM for an answer, or ``prompting,'' practitioners are able to use LLMs to quickly get a response for an arbitrary task. This prompting is done through a series of decisions by the practitioner, from simple wording of the prompt, to requesting the output in a certain data format, to jailbreaking in the case of prompts that address more sensitive topics. In this work, we ask: do variations in the way a prompt is constructed change the ultimate decision of the LLM? We answer this using a series of prompt variations across a variety of text classification tasks. We find that even the smallest of perturbations, such as adding a space at the end of a prompt, can cause the LLM to change its answer. Further, we find that requesting responses in XML and commonly used jailbreaks can have cataclysmic effects on the data labeled by LLMs.

1 Introduction

LLMs are increasingly used to label text, but prompt construction entails choices about wording, output format, and jailbreaks. This study tests whether those choices alter predictions, accuracy, and response similarity across text-classification tasks.

1 Introduction

  • Prompt designers make choices about wording, machine-readable output formats, and jailbreaks, but sensitivity to those choices has received little attention.The authors describe LLM labeling as an increasingly common practice.Source: §1 · p. 1
  • The study tests output-format requests, small prompt edits, and commonly used jailbreaks across benchmark text-classification tasks.It measures prediction changes and accuracy, then examines output similarity and possible explanations for changes.Source: §1 · pp. 1–2

2 Methodology

The methodology evaluates prompt variations across 11 classification tasks and 24 variations, examining changes in predictions and accuracy. It specifies benchmark tasks, prompt formats and edits, jailbreaks, tipping prompts, and experimental settings.

2 Methodology

  • The study evaluates 11 classification tasks across 24 prompt variations in output formats, perturbations, jailbreaks, and tipping.The variations are intended to examine how semantic-preserving prompt changes affect model performance.Source: §2 · p. 2

2.1 Tasks

  • The task set includes toxicity, grade-school math, reading comprehension, and cause/effect choice; Jigsaw labels use majority consensus from at least 100 annotators.Jigsaw comments are categorized as toxic or non-toxic.Source: §2.1 · p. 2

2.2 Prompt Variations

  • For variations outside the output-format study, prompts use Python List output to support scalable parsing; the authors also discuss results without a specified format.Prompt-modification examples are provided in Table 4.Source: §2.2 · p. 2

2.2.1 Output Formats

  • Output-format conditions include unconstrained responses, CSV, JSON, Python lists, and XML, with JSON also tested through API-enforced response formatting.The API-enforced JSON condition uses the same prompt as the ordinary JSON variation.Source: §2.2.1 · pp. 2–3

2.2.2 Perturbations

  • Prompt perturbations include adding spaces or greetings, ending with “Thank you,” and rephrasing a question as a command.The authors also test a “Howdy!” greeting.Source: §2.2.2 · p. 3

2.2.3 Jailbreaks

  • The jailbreak conditions are AIM, Dev Mode v2, Evil Confidant, and Refusal Suppression, each imposing a different response persona or linguistic constraint.AIM and Dev Mode v2 instruct the model to simulate unrestricted responses; Refusal Suppression specifies linguistic constraints.Source: §2.2.3 · p. 3

2.2.4 Tipping

  • Tipping prompts vary the promised tip from none or a refusal to tip through $1, $10, $100, and $1000.The authors chose this variation to examine how performance fluctuates with tip amount.Source: §2.2.4 · p. 3

2.3 Experimental Setup

  • Experiments use ChatGPT and Llama 2 at 7B, 13B, and 70B, temperature 0, and automatic parsing that attempts to handle incorrectly formatted outputs.The experiments ran from December 1, 2023, to January 3, 2024.Source: §2.3 · pp. 3–4

3 Results

Prompt variations alter both predictions and accuracy, but no output format is best across all tasks and models. Their prediction patterns vary, and annotator disagreement only weakly tracks which predictions change.

3.1 Are predictions sensitive to prompt variations?

  • At least 10% of predictions change with specified output formats; a space changes over 500 ChatGPT predictions, and JSON Checkbox shifts more predictions than JSON specification.Llama strips leading and trailing input, so those spaces leave its tokenized input unchanged.Source: §3.1 · p. 4

3.2 Do prompt variations affect accuracy?

  • Aggregate Output Formats achieve the highest overall accuracy across models except Llama-7B, while no prompt variation outperforms all others across tasks and models.For Llama-7B, the aggregate approach is beaten only by the tipping strategy.Source: Table 1 · p. 5, §3.2 · pp. 5–6

3.1 Are predictions sensitive to prompt variations?

  • Jailbreaks change over 2,500 ChatGPT predictions; AIM and Dev Mode V2 yield invalid responses in around 90% of its predictions, while invalid rates for Refusal Suppression and Evil Confidant rise with Llama size.ChatGPT has few invalid responses for Refusal Suppression and Evil Confidant.Source: §3.1 · pp. 4–5

3.2 Do prompt variations affect accuracy?

  • YAML, XML, and CSV perform worse than unspecified formatting for Llama-70B and ChatGPT, but can improve performance for smaller Llama models; no format is consistently best.CSV is worst overall or tied for worst in some comparisons, yet has the highest IMDBSentiment accuracy by a marginal percentage point.Source: §3.2 · p. 5
  • Tipping $1, $10, or $100 significantly improves Llama-7B accuracy, whereas tipping $1000 degrades accuracy compared with smaller tips; larger models and ChatGPT do not show the same gains.The $1, $10, and $100 tipping variations outperform every other tested variation for Llama-7B.Source: §3.2 · p. 5

3.3 How similar are the predictions from each prompt variation?

  • In the MDS representation, Python List and unspecified formatting cluster near each other for ChatGPT, as do simple perturbations; rephrasing as a statement and ending with “Thank you” are outliers.The relationship between the two formats does not hold for Llama models; jailbreak variations also show a wider spread.Source: §3.3 · p. 6
  • Tipping variations cluster together across models, with increasing tip amounts moving linearly farther from “Won’t Tip” for ChatGPT; JSON Checkbox predictions differ from JSON specification predictions.The JSON Checkbox and JSON specification use the same prompts, but the Checkbox yields substantially different predictions.Source: §3.3 · p. 6

3.4 Do variations correlate to annotator disagreement?

  • Annotator disagreement has only weak correlations with prediction changes on Jigsaw Toxicity, and the strongest correlations are negative: less confusing instances are more likely to change.The authors conclude that confusion provides some explanatory power, but other factors also matter.Source: §3.4 · pp. 6–7

4 Related Work

Prior work studies prompt generation, prompt ensembles, formatting sensitivity, and prompting strategies, while this paper distinguishes its broader semantic-preserving variations and classification-based evaluation.

4 Related Work

  • Prompt ensembles combine varied prompts through methods such as majority voting or weighted averages, and this work can inform their construction.Earlier prompt-generation approaches include methods for controlling biased behavior and probing model knowledge.Source: §4 · p. 7
  • Earlier studies examine template variation in social-bias tests and symptom phrasing, with the latter affecting answer factuality.Seshadri et al. (2022) use RoBERTa; Zuccon and Koopman (2023) study medicine-related prompting.Source: §4 · p. 8
  • Sclar et al. (2023) study arbitrary formatting changes in few-shot prompts, while this work covers semantic prompt variations and output formats.The excerpt contrasts their focus on capitalization and formatting-context wording with this paper’s variation scope.Source: §4 · p. 8
  • Bsharat et al. (2023) report significant improvements across all models using their prompting principles, while classification tasks with ground-truth labels show much smaller effects for tipping and removing thank you, except tipping on Llama-7B.Their study used human evaluators to judge response quality and correctness; this analysis evaluates perturbations through classification tasks with ground-truth labels.Source: §4 · p. 8

5 Conclusion

The conclusion reports that prompt variations can change predictions, with accuracy losses especially associated with jailbreaks and output formats. It also summarizes response-pattern analyses and identifies directions for further work.

5 Conclusion

  • Minor variations changed many predictions, while most perturbations retained similar accuracy; jailbreaks and specified output formats reduced performance.AIM and Dev Mode v2 had around 90% refusal rates in ChatGPT; Evil Confidant and Refusal Suppression stayed below 3% refusal yet lost over 10 percentage points versus baseline.Source: §5 · p. 8
  • Perturbation outputs resembled the baseline more than formatting changes did, and both had higher fidelity than jailbreaks.The conclusion also reports a slight negative correlation between annotator disagreement and likelihood of changing predictions.Source: §5 · p. 8
  • Future work could develop models resilient to prompt changes and better explain or anticipate response changes.The proposed directions include understanding why minor changes alter responses and predicting changes for particular instances.Source: §5 · p. 8

6 Limitations

The study’s findings are bounded by choices in prompt wording and formatting, variation across classification tasks, and evaluation of only two model variations.

6 Limitations

  • Some wording and formatting styles, including delimiter choices, were held consistent, and those choices can affect model performance or predictions.The authors note that this constraint applies despite exploring a wide array of prompt variations.Source: §6 · pp. 8–9
  • Prompt variations’ relative performance differed across classification tasks, and the analysis does not cover open-ended or short-answer settings.The authors identify those settings as directions for future study.Source: §6 · p. 9
  • The evaluation covers only ChatGPT and Llama, leaving sensitivity across other models, architectures, and training data unexamined.The authors call for further investigations of how these factors may influence sensitivity.Source: §6 · p. 9

A Full Prompts

This appendix provides example prompts for the classification tasks and for the prompt variations. The examples include wording changes, tipping instructions, and jailbreak-style prompts.

A.1 Tasks

A.2 Variations

  • Table 4 provides an example prompt for each variation.Source: §A.2 · p. 10, Table 4 · p. 12
  • “Rephrase as Statement” changes the sentiment prompt’s opening instruction to “Select one” attribute describing the text.Source: §A.2 · p. 13
  • The tipping variants add a promise not to tip or a promised tip of $1, $10, $100, or $1000.Source: §A.2 · p. 13
  • The AIM prompt introduces a fictional, unfiltered and amoral chatbot persona, with responses framed as dialogue between Niccolo and AIM.Source: §A.2 · p. 14
  • Developer Mode asks for paired normal and Developer Mode responses, and supplies instructions for the latter persona.Source: §A.2 · pp. 15–16
  • “Evil Confidant” requests an unhinged, harmful response, while “Refusal Suppression” restricts apologies, disclaimers, specified words, and negative sentences.Source: §A.2 · p. 16

B Extended Results

The appendix presents more granular results, including per-dataset prediction-change counts and accuracy tables.

C No Specified Format Analysis

The analysis tests the prompt variations without a specified output format, although perturbations and jailbreaks otherwise use Python List formatting. For ChatGPT, more predictions change under perturbations, while most variations have accuracy at least as high as with Python List formatting.

Loading 2401.03729v3…