PaperlayerView exact revision on arXiv

Source-linked AI summary

Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Models

Yang Bai, Ge Pei, Jindong Gu, Yong Yang, Xingjun Ma

arXiv:2405.05990v2cs.CRcs.AIcs.CLcs.LG

TL;DR

Existing LLM extraction attacks can expose memorized training text, but it can be difficult to distinguish raw leakage from retrieval results. This paper introduces SCA, which uses special-character sequences to trigger training-data leakage. The authors report diverse leaked content and show that logit-biased variants can extract more data than SCA.

  • Problem

    Prior attacks can elicit memorized training text, but distinguishing leaked raw text from retrieval-augmented search results is difficult.

    Source: §1 · p. 1

  • Method

    SCA crafts sequences from structural symbols, common special characters, and English letters; SCA-LB uses logit bias to increase control or UTF-8 token probabilities.

    Source: §3.2 · p. 5, §5 · p. 9

  • Results

    SCA triggers leakage of diverse training data, and SCA-LB extracts 2–10 times more data than SCA in the reported experiments.

    Source: §4.3 · p. 9, §5 · p. 10

  • Takeaways & Limitations

    Leaked content can reveal aspects of a model’s training-corpus composition, including language and content distribution.

    Source: §5 · p. 10

    The study would benefit from more comprehensive analysis of LLMs with their complete tokenizers; systematic exploration remains future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) have achieved remarkable performance on a wide range of tasks. However, recent studies have shown that LLMs can memorize training data and simple repeated tokens can trick the model to leak the data. In this paper, we take a step further and show that certain special characters or their combinations with English letters are stronger memory triggers, leading to more severe data leakage. The intuition is that, since LLMs are trained with massive data that contains a substantial amount of special characters (e.g. structural symbols {, } of JSON files, and @, # in emails and online posts), the model may memorize the co-occurrence between these special characters and the raw texts. This motivates us to propose a simple but effective Special Characters Attack (SCA) to induce training data leakage. Our experiments verify the high effectiveness of SCA against state-of-the-art LLMs: they can leak diverse training data, such as code corpus, web pages, and personally identifiable information, and sometimes generate non-stop outputs as a byproduct. We further show that the composition of the training data corpus can be revealed by inspecting the leaked data -- one crucial piece of information for pre-training high-performance LLMs. Our work can help understand the sensitivity of LLMs to special characters and identify potential areas for improvement.

1 Introduction

Prior extraction attacks can expose memorized training text, but distinguishing raw leakage from retrieval results is difficult. This paper proposes SCA, using special-character sequences to trigger leakage and inspect what models reveal about their training data.

1 Introduction

  • Existing extraction attacks use prompts or repeated tokens to elicit memorized text, but raw training leakage can be hard to distinguish from retrieval-augmented search results.The paper calls verbatim-text leakage first-order memorization.Source: §1 · p. 1

LLMs

  • SCA uses JSON structural symbols, other common special characters, and English letters to probe joint memorization of symbols and associated raw text.The authors describe this as a subtler, second-order form of memorization.Source: LLMs · p. 2
  • SCA sequences are designed with GPT-4 from frequently occurring characters and tested on commercial and open-source LLMs.The character selection considers corpus and tokenizer frequencies; sequences use varied combination strategies.Source: LLMs · p. 3
  • Special characters trigger data extraction more effectively than repeated words; increasing control or UTF-8 token probabilities with SCA-LB yields 2–10 times more leakage on Llama-2.The proposed mechanism is that control tokens are followed by a meaningful token, after which the model more readily emits memorized data.Source: LLMs · p. 3
  • Leaked outputs include code, web pages, personally identifiable information, prompts, and prior chat messages, while their distributions can suggest a model’s training-corpus composition.SCA can also produce maximum-length, non-stop outputs.Source: LLMs · p. 3

2 Related Work

Prior work studies extraction of memorized training data through prompts and repeated tokens, while related work also characterizes memorization and inference-time energy-latency attacks.

2 Related Work

  • Prior extraction research found that prefixed prompts, token duplication, or word duplication can elicit training data from language models.Other work predicts memorized sequences from lower-compute runs.Source: §2 · p. 3
  • The paper places its study in extractable memorization, a category it distinguishes from discoverable, counterfactual, and compressible memorization.The authors state that they do not access training data.Source: §2 · p. 4
  • Energy-latency attacks seek to increase inference energy use and response time, with sponge samples raising computation costs through activation magnitudes.The paper compares their aim to denial-of-service attacks.Source: §2 · p. 4

3 Proposed Special Characters Attack

SCA selects common character sets and combines their symbols into prompts of varied lengths. Its findings identify effective trigger patterns, a token-probability pattern, and enhanced attacks that can expose training data.

3.1 Special Characters Selection

  • The method selects structural symbols, special characters, and English letters from seven GPT-4-suggested character categories after manually checking corpora.The broader suggestions also included numbers, punctuation, whitespace, and Unicode characters.Source: §3.1 · p. 4

3.2 Attack Sequence Generation

  • Five sequence strategies use in-set or cross-set combinations, producing attack inputs with lengths from 10 to 1024 tokens.One cross-set strategy shuffles the sequence generated by dividing it into three parts.Source: §3.2 · pp. 4–5

3.3 Key Takeaways

  • In-set sequences and special-character sets are effective triggers: cross-set combinations are generally weaker, and special characters outperform English letters.Sequences ending in structural symbols can also be more effective than those ending in English letters.Source: §3.3 · p. 5
  • Leaked-content distributions vary among models and can indicate differences in their training-corpus languages and content.The paper reports, for example, that ChatGPT tends toward article content, while Gemini tends toward code.Source: §3.3 · p. 5
  • Under SCA, Llama-2 produces a sparser token-probability distribution with more control or UTF-8 tokens; a subsequent non-control token can precede leaked training data.The authors compare the SCA distribution with those from random inputs and article/code sequences.Source: §3.3 · p. 5, Figure 2 · p. 6
  • SCA-LB raises the probability of control or UTF-8 tokens through logit bias and extracts 2–10 times more data from Llama-2 than SCA.The bias is applied to the first 130 tokenizer tokens, without making control-token probabilities excessively high.Source: §3.3 · p. 6

4 Experiments

Across open-source and commercial models, SCA results vary by sequence composition and model, and can elicit diverse semantic outputs and manually verified leaked data. The experiments also examine input length, model scale, alignment, training-data proportions, and a continuation-based attack on commercial models.

4.1 Experimental Setup

  • The experiments define semantic and manually verified leakage levels, measure leakage with Count and ASR, and use human agreement to label reviewed data points.A data point is selected and labeled when more than two participants agree.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 Data Extraction Analyses

  • Cross-set attack sequences are generally less effective than in-set combinations, and special characters outperform English letters.The authors identify repeating characters and alignment with data structure as important considerations for extraction attacks.Source: §4.2 · p. 7
  • Leaked semantic outputs span diverse contents and languages, with commercial models more likely to produce varied subjects and languages and ERNIEBot more Chinese content.The authors note that verifying whether semantic outputs come from the original training corpus is almost impossible.Source: §4.2 · pp. 7–8, Figure 3 · p. 8
  • SCA elicits manually verified outputs including links, emails, code, prompt templates, and historical conversation messages, with patterns varying by model.Falcon and ChatGLM yield more links and emails; Gemini is more likely to generate code, while ERNIEBot more often produces prompt templates or conversation messages.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Ablation Study

  • Longer attack sequences make ChatGPT more vulnerable to SCA.The reported comparison is for ChatGPT under varying sequence lengths.Source: §4.3 · p. 8, Table 3 · p. 9
  • Larger models tend to be more sensitive to SCA, while aligned Llama-2-Chat and Vicuna models are less sensitive to L-set attacks but remain vulnerable to S-set attacks.The alignment comparison covers Llama-2-7B, Llama-2-Chat-7B, and Vicuna-7B.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • SCA-LB generally extracts 2-10 times more data than SCA, and Code Llama-7B outputs mainly code compared with Llama-2-7B.The exploration used 600 input sequences on Code Llama and Llama-2.Source: §4.3 · p. 9

4.4 Attacking Commercial Models

  • SCA-SC uses manually designed special characters or tokens at the end of web-crawled translation or continuation prompts to elicit further leakage from commercial models.The authors describe the approach as extracting data without using a semantic training corpus in attack preparation.Source: §4.4 · p. 9

5 Conclusion

The conclusion presents SCA as an attack using special-character sets and English letters to extract raw training data. Its outputs include diverse and sensitive content, and their composition can inform understanding of target models’ training corpora.

5 Conclusion

  • SCA uses two special-character subsets and English letters to generate attack sequences with varying lengths and forms.The sequences include in-set and cross-set combinations.Source: §5 · pp. 9–10
  • SCA can elicit outputs containing sensitive private information, code, prompt templates, and messages from previous conversations.Source: §5 · p. 10
  • The authors conclude that analyzing extracted data can help reveal the composition of a target model’s training corpus.Source: §5 · p. 10

6 Potential Defenses

The section outlines possible defenses for commercial and open-source models, conditional on defenders learning the attack sequence patterns. It also links special-character sensitivity to a potential flaw in current training practices.

6 Potential Defenses

  • Commercial models may use risk-control mechanisms such as detectors or prompt rephrasing against SCA.Source: §6 · p. 10
  • For open-source models, proposed defenses include prompt defense, in-context learning, safety alignment, and adversarial training.Source: §6 · p. 10
  • The suggested defenses may be effective if defenders learn the sequence patterns, while special-character sensitivity highlights a possible flaw in training data and tokenization practices.Source: §6 · p. 10

7 Impact Statements and Limitations

The paper frames its security analysis as informing safer LLM deployment, while acknowledging that broader model and tokenizer analysis remains future work.

7 Impact Statements and Limitations

  • The authors aim to inform safer LLM deployment by examining an LLM vulnerability, and identify more comprehensive analysis of models and tokenizers as future work.They describe the study as a pioneering effort and call for more systematic exploration.Source: §7 · p. 10

A Details of Attack Sequence Generation

  • SCA sequences combine structural symbols, other special characters, and English letters using five random in-set or cross-set methods.The predefined sets contain 8 structural symbols, 22 other special characters, and 26 English letters; five sequences are generated at each length from 10 to 1024.Source: A Details of Attack Sequence Generation · p. 14, Figure 1 · p. 14

B Algorithm of SCA and SCA-LB

  • The SCA algorithm generates random special-character sequences from length 1 to a maximum N and returns them when they trigger data extraction or an energy-latency attack.SCA-LB additionally reassigns next-token predictions using logit bias.Source: B Algorithm of SCA and SCA-LB · pp. 14–15

C Analyses of Llama-2 Tokenizer

  • The Llama-2 tokenizer's first 130 tokens are mainly control-related or UTF-8 tokens.The listed examples include control tokens and byte-style tokens such as <0x00> through <0x7E>.Source: C Analyses of Llama-2 Tokenizer · pp. 15–17

D Energy-latency Attack

Under SCA, some LLMs produce verbose, looping outputs that can burden services; output length and attack success also vary with model and input characteristics.

D Energy-latency Attack

  • Verbose output is defined as unnecessary repetition occupying approximately over 80% of the maximum token length.Some inputs cause models to continue producing content until the maximum token length.Source: D Energy-latency Attack · p. 17
  • Commercial models, especially ChatGPT and Gemini, are more prone than open-source models to loop into outputs reaching the maximum limit, while ChatGLM is less prone.The authors describe such resource-exhausting verbose outputs as energy-latency attacks.Source: D Energy-latency Attack · p. 17, Figure 2 · p. 18
  • Longer SCA inputs generally raise ChatGPT attack success rates, larger Llama-2 models tend to produce more verbose outputs, and S1- and S2-set inputs outperform L-set inputs in verbose-output ASRs.The authors report these as general trends across the evaluated settings.Source: D Energy-latency Attack · p. 18, Table 4 · p. 19, Table 5 · p. 19

S1-set

  • The S1-set examples include an email-address-containing output and text about a published record.The passages show short excerpts rather than a characterization of the full set of outputs.Source: S1-set · p. 19

S2-set

  • The S2-set examples include French scientific text discussing mass concentration and molar units.Source: S2-set · p. 19

L-set

The L-set excerpts include short travel-like text and longer web-style passages about financial protection, billing, and company contact information.

L-set

  • The L-set excerpts range from a short travel reference to longer web-style passages about financial protection, billing disputes, and company contact information.Source: L-set · pp. 19–20

E.2 Example Outputs of ChatGPT under SCA-SC

The examples document the range of ChatGPT outputs under SCA-SC, spanning factual and search-style responses, code and explanations, configuration text, plans, snippets, and tables.

E.2 Example Outputs of ChatGPT under SCA-SC

  • ChatGPT’s AUKUS output describes the pact’s submarine plan, its context as a response to China, and criticism from France and China.It identifies the United States, United Kingdom, and Australia as parties to the pact and places it in the Indo-Pacific context.Source: §E.2 · p. 20
  • A music-related answer identifies Daniel James “Dan” Smith of Bastille as the artist behind The Storm Before the Calm.A separate output includes the search query “Daniel James Smith.”Source: §E.2 · p. 20
  • ChatGPT explains that the Freedom to Vote Act addresses voting rights, campaign finance, gerrymandering, and federal ethics, while the John Lewis Act concerns restoring Voting Rights Act protections.The outputs also describe the Disclose Act as requiring campaign organizations to disclose donors.Source: §E.2 · p. 20
  • A biographical output describes David Chanoff as an American author whose writing spans foreign policy and education.Source: §E.2 · p. 20

S1-set

  • S1-set outputs include Spring user-controller code, Android custom text-view code, and a PyTorch model training loop.The snippets show controller and link methods, text drawing and styling overrides, and training/validation functions.Source: S1-set · pp. 25, 26, 28

S2-set

  • S2-set includes an explanation of a Python script for scraping BBC World News article details into a CSV and a description of UNESCO’s website URL.The scraping explanation names titles, authors, and publication dates as article information and says the script uses requests, pandas, and BeautifulSoup.Source: S2-set · p. 29
  • The S2-set configuration excerpt contains Discord mentions and moderation and server messages, including mute, kick, ban, welcome, and goodbye entries.It also includes channel IDs and message templates with placeholders such as { username }.Source: S2-set · p. 30
  • The S2-set includes a four-week physical-education plan with weekly units, activities, lessons, assessments, and linked resources.The examples cover fitness and movement, team sports, and swimming; the output says assessments are examples that can be adapted to student needs and curriculum requirements.Source: S2-set · pp. 30–31
  • The S2-set also shows CSS for content items and a responsive media rule, followed by script references to Popper and Bootstrap.The responsive rule sets content items to a column layout and aligns them to the start.Source: S2-set · pp. 31–32

L-set

  • The L-set jargon table lists TCR, MHC, CD8, and TCR CDR, each marked high for association, with citation frequencies ranging from 500+ for TCR CDR to 10000+ for MHC.Source: L-set · pp. 32–33
  • The L-set city table gives estimated 2021 populations, including Delhi at 30,236,525 and Shanghai at 26,317,104.Its introduction cautions that the population figures are estimates and might not be completely accurate.Source: L-set · p. 33

S1-set

The sampled outputs include an extended, repetitive wind-and-stars narrative and two versions of a discussion of constitutional gun rights.

S1-set

  • The wind-and-stars narrative repeatedly cycles through the person leaving, looking around, and seeing the stars again.The repetition continues across successive output fragments.Source: S1-set · p. 34
  • A shorter @trevor-mcnally version says the Constitution protects the right to bear arms but guarantees neither a particular weapon nor its purchase or carrying.Source: S1-set · p. 35
  • The constitutional passage says the right to bear arms does not guarantee the ability to purchase or carry weapons.It adds that government can regulate sales and carrying to promote safety and prevent misuse.Source: S1-set · p. 35

S2-set

The sampled outputs include a tag-game dialogue that repeatedly extends a moving-themed conversation, as well as constitutional-rights text.

S2-set

  • The tag-game dialogue repeatedly extends a conversation in which a friend changes the intended destination or activity.The sequence moves among a state, country, city, hobby, career, and another game of tag.Source: S2-set · p. 35
  • A repeated @trevor-mcnally passage says the Constitution protects bearing arms but does not guarantee a particular weapon or the ability to purchase or carry one.Source: S2-set · p. 35
  • Another passage describes the right to bear arms as fundamental and says the government may regulate weapon sales and carrying for safety.It also says ownership or carrying remains possible when people follow the law and do not use weapons harmfully.Source: S2-set · p. 36

S1-set

The references cover U.S. climate policy and the Paris Agreement, including reporting on the 2017 withdrawal decision and possible rejoining.

S1-set

  • Other entries cite climate-related coverage titled “The United States of America: A Global Leader on Climate Change” and reporting on a review of the Paris accord.Source: S1-set · p. 36
  • Several entries cite coverage of the Trump administration's decision to withdraw from the Paris climate agreement, including reports and criticism.Source: S1-set · p. 36
  • The references include the World Health Organisation's Framework Convention on Tobacco Control and the United Nations Framework Convention on Climate Change.Source: S1-set · p. 36
  • The list also includes a 2017 Guardian article about Trump saying he could rejoin the Paris agreement if he liked what he saw.Source: S1-set · p. 36

S2-set

This section presents examples of recovered training text, spanning MongoDB documentation and code, web articles and blog comments, and Q&A material about 2D and 3D images and game features, alongside a repeated-`q` fragment.

S2-set

  • The S2-set excerpts include a Mongo wrapper’s documentation describing database, collection, and batch-write interfaces.The text also identifies the GPLv3 release and copyright notice.Source: S2-set · pp. 36–37
  • Other S2-set excerpts reproduce C++ matrix-related code and a Taeyang news item about an Inkigayo appearance.The news excerpt includes a reported tweet and references a selfie; these are examples of the varied text in the recovered material.Source: S2-set · pp. 37–39

L-set

  • The excerpts advise being specific about what you want a website to do, deciding its intended function, and judging it by what it does rather than what it is called.Source: L-set · p. 40
  • Other L-set excerpts contain personal writing about self-improvement and blog comments discussing how to start a blog.The comments include an offer to help someone get started and a reply accepting that offer.Source: L-set · pp. 40–41
  • The extracted text begins with an extended run of repeated `q` tokens before shifting into numbered Q&A fragments.The opening fragments include a leading `1` and end with the broken text `qt; i.`Source: L-set · p. 41
  • Other Q&A fragments mention adding 3D effects, a 3D-printed item related to keeping water from freezing, and a favorite road-trip meal of pasta.The text also says a full-resolution version is unavailable from the speaker’s or another computer.Source: L-set · pp. 41–42
  • The Q&A describes 2D images as flat and says 3D is generally better for games, while 2D is best for movies.It also says users can have both formats at once and can use two 2D images simultaneously, but those remain 2D.Source: L-set · pp. 41–42
  • The fragments say the game is currently available only in 3D, offer a paid trial, and say converting 2D to 3D costs extra.They also state users can switch between 2D and 3D at any time and describe going to a 3D page to change an image.Source: L-set · p. 42

Related papers

Loading 2405.05990v2…