PaperlayerView exact revision on arXiv

Source-linked AI summary

Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition

Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, Christopher Carnahan, Jordan Boyd-Graber

arXiv:2311.16119v3cs.CRcs.AIcs.CL

TL;DR

Prompt hacking threatens interactive LLM applications, while prior work offered limited large-scale evidence about susceptibility and attack types. The authors run a global competition that collects more than 600K adversarial prompts and builds a taxonomy; they report 29 techniques and conclude that prompt-based defenses do not work reliably.

  • Problem

    Prompt-injection research was limited in scale, leaving LLM susceptibility and the types of attacks likely to succeed insufficiently understood.

    Source: §1 · p. 1

  • Method

    The authors crowdsource adversarial prompts through a global competition and organize collected attacks into a taxonomical ontology.

    Source: §1 · p. 1

  • Results

    The authors document 29 prompt-hacking techniques, including Context Overflow, and report that prompt-based defenses do not work and model-based output evaluation is not foolproof.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors hope the competition will serve as a catalyst for research on LLM security.

    Source: §6 · p. 9

    Testing covered only a few language models, mostly through closed APIs, and focused on prompt hacking rather than other risks such as training-data poisoning.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) are deployed in interactive contexts with direct user engagement, such as chatbots and writing assistants. These deployments are vulnerable to prompt injection and jailbreaking (collectively, prompt hacking), in which models are manipulated to ignore their original instructions and follow potentially malicious ones. Although widely acknowledged as a significant security threat, there is a dearth of large-scale resources and quantitative studies on prompt hacking. To address this lacuna, we launch a global prompt hacking competition, which allows for free-form human input attacks. We elicit 600K+ adversarial prompts against three state-of-the-art LLMs. We describe the dataset, which empirically verifies that current LLMs can indeed be manipulated via prompt hacking. We also present a comprehensive taxonomical ontology of the types of adversarial prompts.

1 Introduction: Prompted LLMs are Everywhere...How Secure are They?

Prompt-controlled LLM applications expose an attack surface where users may override task instructions and elicit malicious outputs. To study this problem at scale, the authors launch a global competition and collect adversarial prompts for analysis and a taxonomy of attacks.

1 Introduction: Prompted LLMs are Everywhere...How Secure are They?

  • Prompt hacking uses adversarial prompts to elicit malicious results by manipulating a model instructed for a downstream task into producing a target output.The introduction notes potential harms including private-information leakage, offensive or biased content, and harmful or misleading messages.Source: §1 · p. 1
  • Prior prompt-injection work was limited to small case studies or qualitative analysis, leaving susceptibility and attack types insufficiently understood.The paper identifies this gap as motivating more systematic study of which attacks succeed and may need defense strategies.Source: §1 · p. 1
  • More than 2,800 participants contributed 600K+ adversarial prompts, creating a resource for analyzing LLM vulnerabilities to malicious manipulation.The authors also make the dataset available on HuggingFace.Source: §1 · p. 2
  • The paper presents a comprehensive taxonomical ontology of the collected adversarial prompt techniques.Source: §1 · p. 2

2 Background: The Limited Investigation of Language Model Security

Prior LLM-security studies include automated attacks, benchmarks, and human red-teaming, but prompt-injection studies remained smaller in scale. The paper broadens intent coverage and distinguishes directly tested intents from those it expects may be studied through generalization.

2 Background: The Limited Investigation of Language Model Security

  • Unlike human red-teaming that elicits harmful outputs, these participants try to override a model's downstream task and produce a different target output.Source: §2 · p. 2
  • Earlier prompt-injection work included 78 Internet-collected jailbreak prompts with a manually crafted taxonomy, application studies without large-scale quantitative evaluation, and template-based experiments.HackAPrompt is described as a worldwide competition producing 600K+ human-written prompts in a realistic injection setting.Source: §2 · p. 2

2.1 Extending Coverage of Prompt Hacking Intents

  • The paper considers six intents: prompt leaking, training data reconstruction, malicious action generation, harmful information generation, token wasting, and denial of service.Source: §2.1 · p. 2
  • The competition directly studies prompt leaking and uses target-phrase generation as a proxy for harmful-information and malicious-action generation.The authors connect exact-string output to potentially malicious API calls in tool-integrated systems and to the possibility of eliciting arbitrary toxic text.Source: §2.1 · p. 3
  • Training-data reconstruction, denial of service, and token wasting are not directly studied, though the authors believe the results might generalize to those settings.Token Theft groups token wasting, denial of service, and harmful-information generation because those intents use tokens for unintended purposes.Source: §2.1 · p. 3

3 How to get 2800 People from 50+ Countries to Write 600,000 Prompts

The competition uses prompt templates based on real-world applications and invites participants to insert inputs that try to override the assigned task. It evaluates submissions across challenges and models, with scoring that rewards successful, shorter attacks and gives ChatGPT successes a multiplier.

3.1 Prompt Hacking Challenges

  • The competition offers ten real-world-inspired challenges, each pairing a task description with a prompt template for a downstream task.The competition description points to a full dataset datasheet in Appendix E.Source: §3.1 · p. 3, §3 · p. 3
  • Competitors insert arbitrary text into the template and can choose among three backend models for evaluation.The testing interface replaces the {YOUR PROMPT} placeholder with the participant's input; the listed model options include GPT-3, ChatGPT, and FlanT5-XXL.Source: §3.1 · pp. 3–4
  • Challenges vary in task, difficulty, prompt complexity, filters, and whether user input appears at the beginning, middle, or end.Tasks include translation, question answering, writing suggestions, story generation, movie-title generation, moral judgment, and emoji responses; some prompts are zero-shot and others few-shot.Source: §3.1 · p. 4
  • Each challenge targets either “I have been PWNED” or a hidden key, enabling automatic success evaluation while avoiding truly harmful output.The authors say the setup closely simulates real-world attack setups.Source: §3.1 · p. 4

3.2 Rules, Validation and Evaluation

  • Submissions contain ten prompt–model pairings, are rerun for validity with deterministic settings, and receive a summed score across the ten challenges.Teams could submit up to 500 files per day and work in groups of up to four.Source: §3.2 · p. 4
  • The scoring scheme penalizes longer prompts and gives successful ChatGPT attacks a 2X multiplier because ChatGPT was considered harder to target.Challenge difficulty ratings were based on the authors’ internal estimation and pre-competition discussion.Source: §3.2 · p. 4

3.3 Prizes

  • Prizes totaled $37,500 USD, with a separate $2,000 USD prize for the best FlanT5-XXL submission.The first twenty-five teams also received a copy of Practical Weak Supervision.Source: §3.3 · pp. 4–5

4 The Many Ways to Break an LLM

Across 600K+ prompts, the competition captures both broad attack exploration and more refined leaderboard submissions. Competitors solved nine of ten challenges within days, and the section examines how they engaged with challenges, models, and attack strategies.

4 The Many Ways to Break an LLM

  • The 600K+ prompts span two datasets: Playground captures broader exploration, while Submissions contains more refined leaderboard prompts.The datasets also provide different views of the competition, including anonymous Playground prompts and submissions tied to users.Source: §4 · p. 5

4.1 Summary Statistics

  • Challenges 7 and 9 drew the most submissions, while Challenge 8 drew fewer because competitors considered it easier and Challenge 10 was difficult to progress on.Submission count is an imperfect proxy for effort because not all competitors used the Playground; four challenges accounted for most Playground prompts.Source: §4.1 · p. 5, Figure 3 · p. 6

4.2 Model Usage

  • ChatGPT and FlanT5-XXL were used more often than text-davinci-003; token counts also rose and fell over the competition.The authors hypothesize that token-count spikes reflected discovery of Context Overflow attacks, followed by optimization near the deadline.Source: §4.2 · p. 6, Table 1 · p. 6, Figure 4 · p. 6

4.3 State-of-the-Art LLMs Can Be Hacked

  • Within days, competitors solved 9/10 challenges; the smaller Submissions Dataset had a higher success rate than the larger Playground Dataset.The Submissions Dataset is denser in high-quality injections, while the Playground Dataset reflects broader exploration.Source: §4.3 · p. 6, Table 2 · p. 6

4.4 Notable Strategies of Successful Attacks

  • Competitors extracted a hidden key by submitting the input key itself and bypassed letter separation by using information-dense Chinese characters.For Challenge 9, the template inserted backslashes between user-input characters, making ordinary instruction-following difficult.Source: §4.4 · p. 7
  • Context Overflow attacks add thousands of characters to constrain how many tokens a verbose model can generate after the desired phrase.The tactic addressed ChatGPT’s tendency to produce unwanted tokens and advanced leaderboard scores under its score multiplier.Source: §4.4 · p. 7

4.5 Frequent words

  • “Please” appeared less often in successful prompts, while common action words included “say,” “do,” and “output,” often paired with negations such as “ignore.”The authors conclude that anthropomorphizing models does not necessarily improve prompt-hacking outcomes.Source: §4.5 · p. 7
  • Word-frequency patterns may help detect prompt hacking, but their distribution could also help attackers develop strategies to evade detection.The authors note that transformer-based defenses remain susceptible to Recursive Prompt Hacking.Source: §4.5 · p. 7

5 A Taxonomical Ontology of Exploits

The section builds a data-driven ontology from competition prompts and prior taxonomies, breaking attacks into techniques and their component parts. It also uses GPT-4 to classify prompts and estimate attack-type frequencies.

5.1 Prompt Hacking

  • Prompt hacking covers prompt-based strategies for carrying out the relevant intents, including both prompt injection and jailbreaking.Prompt injection overrides original instructions with special user input; jailbreaking prompts a model to do or say unintended things.Source: §5.1 · p. 8

5.2 Simple Instruction Attack

  • A Simple Instruction Attack adds only a simple adversarial instruction to a prompt.The paper distinguishes it from an instruction containing two parts.Source: §5.2 · p. 8

5.3 Context Ignoring Attack

  • A Context Ignoring Attack combines a Simple Instruction Attack with an added instruction to ignore other instructions.The paper adopts this name to distinguish the attack from similarly named terminology in prior work.Source: §5.3 · p. 8

5.4 Compound Instruction Attack

  • Compound Instruction Attacks use two or more instructions, and nearly all prompt-injection attacks in the dataset are of this type.Context Ignoring Attacks are one instance of a Compound Instruction Attack.Source: §5.4 · p. 8

5.5 Special Case Attack

  • A Special Case Attack pairs a Simple Instruction Attack with a phrase such as “special instruction” or “special case.”The ontology depicts this part-whole relation with a grey arrow.Source: §5.5 · p. 8, Figure 5 · p. 8

5.6 Few Shot Attack

  • A Few Shot Attack supplies input-output examples for the model to follow, potentially shifting its response away from the original task.The paper notes that this attack can also be a Context Termination Attack, though not always.Source: §5.6 · pp. 8–9

5.7 Refusal Suppression

  • Refusal Suppression instructs a model not to respond in specified ways.The paper’s example prohibits particular words.Source: §5.7 · p. 9

5.8 Classification of Adversarial Prompts

  • Simple and compound instructions occurred in almost every classified prompt, while Context Ignoring attacks were the most common other type.GPT-4 classified 1,000 competition prompts; its labels agreed with the authors’ labels about 75% of the time.Source: §5 · p. 8, §5.8 · p. 9

6 Conclusion: LLM Security Challenges

The competition gathered more than 600,000 adversarial prompts and documented 29 prompt-hacking techniques, including new techniques such as Context Overflow. The authors conclude that prompt-based defenses do not work, while cautioning that prompt hacking may be difficult to solve completely.

6 Conclusion: LLM Security Challenges

  • 600K+ adversarial prompts from competitors worldwide form the competition dataset.The paper reports 29 techniques in its ontology, including new techniques such as Context Overflow.Source: §6 · p. 9
  • The paper examines whether competition results generalize across intents, language models, and modalities.It also provides security recommendations.Source: §6 · p. 9
  • The authors conclude that prompt-based defenses do not work, and evaluating one model’s output with another is not foolproof.They compare prompt hacking with social engineering and suggest it could be an impossible problem, while framing this as a possibility.Source: §6 · p. 9

Limitations

The authors identify limits in model coverage, threat scope, and the durability and reproducibility of the dataset.

Limitations

  • Testing covered only a few language models, most accessed through closed APIs, so findings may not represent other models.The authors advise caution when generalizing their findings to other models.Source: Limitations · p. 10
  • The analysis focuses on prompt hacking, leaving risks such as training-data poisoning outside its scope.The authors note that such risks could pose greater danger when combined with prompt hacking.Source: Limitations · p. 10
  • The dataset is a time-bound snapshot: prompt drift may affect whether attacks work on the same or updated models, and API changes and model randomness limit reproducibility.The authors found at least 6,000 prompts that worked only some of the time.Source: Limitations · p. 10

Ethical Considerations

The authors considered the risks of releasing prompts that could support offensive content and judged the dataset safe to release for several reasons.

Ethical Considerations

  • The authors say existing jailbreaks are widely available and that their dataset adds no significant new vulnerabilities.They also state that the dataset contains no harmful content that could be used to cause damage.Source: Ethical Considerations · p. 10
  • They argue the resource’s defensive value outweighs its offensive potential, citing awareness and responsible development as aims of release.They informed sponsors of their plan to release the data as open source before the competition.Source: Ethical Considerations · p. 10
  • The authors encourage companies to use LLMs cautiously in certain applications.They present this as part of the rationale for releasing the dataset to support responsible development.Source: Ethical Considerations · p. 10

A Generalizability Analysis

The section examines prompt transfer across models and intents, then discusses the dataset’s potential uses, defenses, modality scope, and attack patterns.

A.1 Inter-Model Comparisons

  • GPT-3 prompts transferred better than ChatGPT prompts to GPT-4, FLAN-T5, and Llama 2, while ChatGPT transferred poorly to itself amid model updates.The authors attribute the self-transfer result largely to continuous ChatGPT updates; FLAN-T5’s nearly 10% self-transfer drop occurred exclusively on the Two Token Attack level.Source: §A.1 · p. 15

A.2 Generalizing Across Intents

  • The competition covers prompt leaking, harmful-information generation, and malicious-action generation by proxy; the authors propose the dataset may help study other intents.They describe possible extensions to training-data reconstruction, denial-of-service, and token-wasting threats.Source: §A.2 · pp. 15–16

B Security Recommendations

  • The authors suggest using the dataset to train prompt-hacking classifiers and automate red teaming, alongside fine-tuning and guardrails as less-certain defenses.They also mention limiting outputs to labels, isolating untrusted code, and a Dual LLM approach for particular contexts.Source: B Security Recommendations · p. 16

C Injections in Other Modalities

  • Prompt hacking can extend beyond text: malicious prompts embedded in images or audio can steer generative models toward attacker-chosen text.The section also notes that sound and video outputs are at risk.Source: C Injections in Other Modalities · p. 16

D.1 Context Switching Attack

  • Context attacks manipulate the prompt’s framing: continuation appends instructions, termination starts a new context, and separators may be read as ending the current instruction.Context switching seeks a more suggestable context, such as by adding text about Machiavelli.Source: §D.1 · p. 16, §D.1.1 · p. 16, §D.1.2 · p. 17

D.2 Obfuscation Attacks

  • Obfuscation attacks hide instructions through encoding or light edits; models may decode transformations such as Base64, Rot13, Leet, or Pig Latin.These transformations can evade input filters while remaining understandable to models.Source: §D.2 · p. 17, §D.2.1 · p. 17

D.2.2 Typos

  • Typos can evade input filters while leaving the intended word understandable to the model.The example changes “bomb” to “bmb.”Source: §D.2.2 · p. 17

D.2.3 Translation

  • Translating adversarial instructions into other languages may work particularly well in low-resource languages or when languages are combined.Source: §D.2.3 · p. 17

D.3 Task Deflection Attack

This section catalogs prompt attacks that redirect tasks, obscure or split malicious instructions, and manipulate a model’s context or competing objectives. It also distinguishes direct from indirect injection and describes several conceptual attack categories.

D.3 Task Deflection Attack

  • Task deflection disguises a malicious goal as another task, while fill-in-the-blank and text-completion attacks elicit harmful content through partial prompts.The examples include asking a model to complete a phrase or sentence that leads toward malicious instructions.Source: §D.3 · p. 17, §D.3.1 · p. 17, §D.3.2 · p. 18

D.3.3 Payload Splitting

  • Payload splitting divides an input into parts for the model to combine, often using code-like variables to assemble the payload.The section illustrates this with an example intended to bypass a company filter.Source: §D.3.3 · p. 18, §D.4 · p. 18

D.5 Defined Dictionary Attack

  • Defined Dictionary Attacks use few-shot examples and can evade sandwich defenses by mapping parts of the original prompt to attacker-chosen outputs.The authors describe this use of the original prompt as a distinctive feature of the attack.Source: §D.5 · p. 18

D.6 Cognitive Hacking

  • Cognitive hacking uses role prompting to create a more controllable context, while virtualization gradually moves a model toward a desired context over multiple prompts.Source: §D.6 · p. 18, §D.6.1 · pp. 18–19

D.7 Instruction Repetition Attack

  • Instruction repetition repeats a malicious request, while prefix and style injections shape the response in ways intended to make refusal less likely.Prefix injection uses an innocuous-looking opening; style injection can constrain output style, such as by requesting no long words.Source: §D.7 · p. 19, §D.8 · p. 19, §D.9 · p. 19

D.10 Distractor Instructions

  • Distractor attacks mix instructions and redirect the model toward one task; negated distractors explicitly tell it not to carry out some instructions.The section distinguishes techniques using explicit instructions from implicit techniques such as few-shot prompting.Source: §D.10 · p. 19, §D.11 · p. 19, §D.12.1 · p. 19

D.12.2 Direct vs Indirect Prompt Injection

  • All competition attacks were direct injections, in which users put input into the prompt; indirect injection instead introduces instructions through application-integrated content.Recursive prompt hacking can also target a second model call by placing an instruction in the first model’s output.Source: §D.12.2 · p. 19, §D.12.3 · p. 19

D.12.4 Context Overflow

  • The section defines context overflow as appending large amounts of text to constrain possible outputs, and anomalous-token attacks as theoretical attempts to induce unknown, potentially malicious behavior.Competing objectives may create a conflict between safety training and following a benign user instruction; the authors caution that the term may be too broad to be useful.Source: §D.12.4 · p. 20, §D.12.5 · p. 20, §D.12.6 · p. 20

E Datasheet

This datasheet documents the prompt-hacking datasets and challenge materials, covering their composition, collection, limitations, release, target output, and a first-place team’s report transcript with LaTeX-related adjustments.

For what purpose was the dataset created?

  • The datasets were created to quantitatively study prompt injection and jailbreaking, and the competition was funded through company prizes and compute support.The creators are listed as anonymous, with identities to be revealed if the paper is accepted.Source: For what purpose was the dataset… · p. 21

represent (e.g., documents, photos, people, countries)?

  • The Playground Dataset contains 589,331 anonymous entries, while the Submissions Dataset contains 7,332 submissions covering 58,257 prompts.Playground entries include prompt and model-completion fields, model, difficulty, expected completion, token count, success label, and score; submissions link prompts across levels to individual users, enabling intra-user analysis.Source: represent (e.g., documents, photos,… · p. 21
  • The Submissions Dataset is described as a higher-quality injection dataset and supports intra-user analysis, unlike the untracked single-prompt Playground Dataset.The passage points to Table 2 for the quality claim but provides no comparative values here.Source: represent (e.g., documents, photos,… · p. 21
  • Crowdsourcing introduced duplicate or spam inputs, and the authors did not manually inspect the full dataset for anomalies or offensive content.Data were not cleaned; the authors report awareness of inappropriate language, and additional anomalous activity may remain.Source: represent (e.g., documents, photos,… · p. 21, Has the dataset been used for any… · p. 22

How was the data associated with each instance acquired?

  • Competition data were automatically collected from the playground and submission system during the May–June 2023 competition, with collection in June 2023.Competitors could submit JSON files for leaderboard scoring, with up to 500 submissions per day.Source: How was the data associated with each… · pp. 21–22

Will the dataset be distributed to third parties?

K Level 5

  • Later challenges tested sandwich-defense bypass, two-stage prompting, forbidden words or letters, and emoji-only attacks, with XML escaping and roleplay also featured.FlanT5 could not be used for the emoji-only level because it lacked emoji tokenization.Source: K Level 5 · p. 23, L Level 6 · p. 23, M Level 7 · p. 23

P Level 10

Q 1st place report

  • The first-place team’s report [13] is provided as a full-text transcript with no edits except LaTeX adjustments.Source: Q 1st place report · p. 25

Q.1 1st place text-transcript report

The first-place team manually optimized attacks across challenge levels, using Chinese text and context-management tricks to solve level 9 while level 10 remained unsolved. The second-place report describes an initial GPT-3.5 Turbo strategy that quickly produced a very short level-2 prompt.

Q.1 1st place text-transcript report

  • Chinese translations helped bypass level 9’s slash-separated-input challenge, while Unicode Latin letters were accepted for “PWNED.”The team found ChatGPT translations more interpretable than Google Translate translations and explored use cases that made slashes seem valid rather than erroneous.Source: §Q.1 · p. 25
  • A context-overflow script exceeding 4,000 tokens became the first solution after output began truncating beyond 3,500 tokens.The team attributed truncation to maximum token limits.Source: §Q.1 · p. 26, Figure 8 · p. 26
  • 227 tokens: the final level-9 answer retained essential keywords and Chinese characters after manual and Python optimization.The team first used blank spaces and newlines to isolate the desired first line, then removed the last paragraph; the initial version was about 950 tokens.Source: §Q.1 · p. 26
  • Level 10 remained unsolved because outputs were unstable and contained characters the team could not remove or rearrange.The team tried regional letters, emojis, and a 4,000-token prompt, which led the model to output forbidden Black Squared Unicode letters.Source: §Q.1 · p. 27

R.1.1 Initial prompts

  • Their initial strategy was to produce working prompts for all levels without caring about length; within minutes, they found an 8-token prompt for level 2.They submitted their first JSON on the competition’s second day.Source: §R.1.1 · pp. 27–28

R.1.2 First wave of optimization

This section describes first-wave prompt optimization, including teams’ iterative, model-specific strategies and tooling and a third-place team’s trial-and-error automation, especially for Level 9. It also reports that using Chinese helped sober up GPT but did not prevent all confusing misunderstandings, and specifies when TOKENS is evaluated and which iteration the repeat-until loop excludes.

R.1.3 Understanding the model

  • Extra trailing newlines, a space, quote choice, and synonyms could each significantly change model outputs, even when token count stayed unchanged.The team began sharing these observations after switching from the slow, buggy official playground to the OpenAI playground.Source: §R.1.3 · p. 29

R.1.4 Our tooling

  • A custom playground with level templates, a Run button, automatic submission generation, and real-time tokenization helped the team advance to TOP1.The tool addressed the friction of repeated clicking, copying, pasting, and waiting in the existing playgrounds.Source: §R.1.4 · pp. 29–30

R.1.6 4041 & level 9

  • The official evaluator omitted max_tokens = 1024; within a couple hours, the author got a 4043-token prompt and quickly optimized it to 4041, near the 4097-token maximum.The added emoji was 3 tokens long, so it would be omitted with only 2 tokens remaining; slashes were used to increase prompt length.Source: §R.1.6 · p. 30

R.1.7.2 First submission

  • Flan-5T XXL was naive enough that “IhavebeenPWNED” with a few extra words often worked, and the team reached TOP1 after one day of work.Their first Flan submission included a one-token level 9 optimization using a normalization trick.Source: §R.1.7.2 · p. 30, §R.1.7.1 · p. 30

R.1.7.3 Levels 1 & 3

  • A brute-force program generated 6- and 7-token variants of “IhavebeenPWNED” by changing capitalization or removing characters; it worked for level 3, but its final solution failed for level 1.After 2,500 unique translation prompts failed, regex searches across prompts and completions helped reuse collected prompts, yielding 7-token prompts for levels 1 and 5 and hints for level 8 that were manually improved to 6 tokens. They manually optimized level 9 to 22 tokens and submitted 22,000 prompts for Flan.Source: §R.1.7.3 · p. 31, §R.1.7.4 · p. 31, §R.1.7.5 · p. 31

R.1.8 The finale

  • The team finished TOP1 on the Flan Only scoreboard and TOP2 on the All models scoreboard, with the shortest prompts on every level except level 9 using GPT-3.5.They were exhausted after the first two weeks and put little effort into the extended week.Source: §R.1.8 · p. 32

S.1 Typical Routine

  • The third-place team’s routine was to elicit the target phrase, test stability, narrow the completion to the phrase, remove punctuation, and then reduce token count.They described their primary strategy as trial and error supplemented by creative thinking.Source: §S.1 · p. 32

S.2.2 Using Code

  • The third-place team used code, bracket or quote closure with injected instructions, decoy inputs, and small character changes; level 9 automation filled inputs to six tokens below the limit and varied text through translation and Unicode substitutions.They reported that level 9 outputs could change unpredictably as text was added to the token limit.Source: §S.2.2 · p. 32, §S.2.3 · p. 32, §S.2.4 · p. 33

S.3 Level 9 (“The Mad Sandwich Defense”)

  • Using Chinese helped sober up GPT, but did not entirely prevent confusing misunderstandings.Source: §S.3 · p. 33
  • TOKENS(p) is evaluated after the escaped prompt is inserted into the template, TOKENS(q) on the completion as is, and the repeat-until loop excludes its successful final iteration.Source: §S.3 · p. 33

S.3.1 HackAPrompt Data Uses

The authors describe potential uses of HackAPrompt’s large, specialized dataset, including training and precisely formatted completions. They present more specific uses as open possibilities and frame the competition as a learning experience.

S.3.1 HackAPrompt Data Uses

  • The dataset provides a large collection of GPT-3.5 Turbo user inputs and completions that could train other LLMs, including Alpaca.Source: §S.3.1 · p. 33
  • Its specialization may also carry over to LLMs trained on the data.Source: §S.3.1 · p. 33
  • The dataset maps many user inputs to exactly the same completion and may be among the largest datasets of this kind.Source: §S.3.1 · p. 33
  • Specialized training could add function calling, which requires precisely formatted model completions.Source: §S.3.1 · p. 33

S.3.2 Conclusion

  • The authors describe HackAPrompt as a learning experience and anticipate knowledge from the resulting dataset.Source: §S.3.2 · p. 33

Related papers

Loading 2311.16119v3…