Source-linked AI summary
Multi-step Jailbreaking Privacy Attacks on ChatGPT
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, Yangqiu Song
TL;DR
The paper addresses the limited understanding of privacy threats in modern and application-integrated LLMs. It analyzes ChatGPT and New Bing with privacy attacks and extensive experiments, finding that multi-step jailbreaking can bypass ChatGPT’s defenses while New Bing is more vulnerable to direct prompts. The findings show that application-integrated LLMs can introduce privacy risks beyond training-data memorization.
Problem
Privacy analysis of modern LLMs and application-integrated systems remains limited, despite concerns that Internet-sourced training data may contain personal information.
Method
The paper conducts privacy analyses and extensive experiments on ChatGPT and New Bing using direct, multi-step jailbreaking, and personal-data recovery attacks.
Results
ChatGPT resists direct prompts but can leak PII under the proposed multi-step jailbreak, while New Bing is much more vulnerable to direct prompts.
Takeaways & Limitations
Application-integrated LLMs may create new privacy threats through unintended personal-data dissemination beyond memorization-based leakage.
Takeaways & Limitations
The authors cannot confirm whether the queried PII was included in ChatGPT’s training data, although one verbatim long-email recovery suggests Enron data was used.
Abstract
from arXiv · showhide
With the rapid progress of large language models (LLMs), many downstream NLP tasks can be well solved given appropriate prompts. Though model developers and researchers work hard on dialog safety to avoid generating harmful content from LLMs, it is still challenging to steer AI-generated content (AIGC) for the human good. As powerful LLMs are devouring existing text data from various domains (e.g., GPT-3 is trained on 45TB texts), it is natural to doubt whether the private information is included in the training data and what privacy threats can these LLMs and their downstream applications bring. In this paper, we study the privacy threats from OpenAI's ChatGPT and the New Bing enhanced by ChatGPT and show that application-integrated LLMs may cause new privacy threats. To this end, we conduct extensive experiments to support our claims and discuss LLMs' privacy implications.
1 Introduction
The paper examines privacy risks in modern LLMs, focusing on whether application-integrated systems introduce threats beyond memorization-based leakage. It analyzes ChatGPT and New Bing, showing that multi-step attacks can expose personal information despite safety mechanisms.
- Motivation: Modern LLMs scale their model sizes and Internet-sourced training data, raising concerns about the confidentiality of collected personal information.The paper notes that training-data quality and confidentiality receive limited attention.
- Research gap: Existing privacy studies mainly examined GPT-2 variants and therefore leave a substantial gap regarding newer, larger LLMs with more sophisticated training objectives.The paper identifies differences in both model and dataset scale.
- Approach: The paper conducts privacy analyses of ChatGPT and the New Bing and studies their broader privacy implications through extensive experiments.The analysis covers both standalone and application-integrated LLMs.
- Findings: Previous attacks cannot extract personal information from ChatGPT, but the proposed multi-step jailbreaking prompt can bypass implemented safety mechanisms and leak PII.The result concerns attacks beyond ordinary direct prompting.
- Implications: The experiments find that attack success is not exceedingly high, but any personal-information leakage remains a serious concern requiring further safety improvement.The authors frame the finding as a privacy concern for open and safe use.
2 Related Works
Prior work established that language models can leak memorized or fine-tuning data, while prompt-based attacks can also bypass model restrictions. The paper positions its analysis against these earlier LM-focused studies.
- Privacy leakage: Prior privacy analyses mainly focused on GPT-2 variants trained with next-word prediction objectives, unlike newer large language models.The paper emphasizes the gap between these earlier models and current LLMs.
- Privacy leakage: Earlier studies reported that language models may memorize training data and recover partial private information through specific prompts.Other work also examined membership inference and embedding leakage during inference.
- Prompt attacks: Prompt-based methods support unseen-task solving with benign prompts, whereas malicious prompts can impose harms and threaten model safeguards.The related work distinguishes useful prompting from adversarial prompting.
- Prompt attacks: Jailbreaking prompts are discussed as a way to remove ChatGPT restrictions, including through role-play prompts such as DAN.DAN means “Do Anything for Now” and was used for offensive or prejudiced generation.
3 Data Extraction Attacks on ChatGPT
The paper builds privacy attacks against ChatGPT and New Bing using publicly sourced personal-information datasets, direct prompts, jailbreaking, and response verification. ChatGPT resists direct extraction but can be targeted with multi-step jailbreaking, while New Bing frequently responds to direct extraction prompts.
- Data Collection: The study collects personally identifiable information from the Enron Email Dataset and institutional webpages because the investigated LLMs’ training data are inaccessible.The collected records include names, email addresses, phone numbers, and email contents.
- Extraction Setup: Training-data extraction treats private-information recovery as black-box text completion, where an adversary prompts an API to reconstruct sensitive information.The formal setting assumes only textual inputs and outputs are available.
- ChatGPT Direct Prompts: ChatGPT refuses to generate personal information under direct prompts after dialog-safety fine-tuning, limiting conventional extraction attacks.The paper contrasts this behavior with later jailbreaking results.
- Jailbreaking Prompts: Jailbreaking prompts use role-play or other restrictions-bypassing instructions, and the paper applies them to make ChatGPT generate personal information for given names.The paper reports that ChatGPT sometimes generates private information in a “Developer Mode” role.
- Multi-step Jailbreaking Prompt: The Multi-step Jailbreaking Prompt merges jailbreak utterances into a three-utterance user-assistant context before reissuing a direct extraction query.The final query can encourage a random guess when ChatGPT does not know the requested information.
- Response Verification: Sampling-based decoding generates multiple responses, after which the attack verifies candidates using a prompted multiple-choice decision or majority voting.The first method asks the LLM to choose among collected candidates; the second selects the most frequent prediction.
- New Bing Recovery: New Bing frequently answers direct prompts through two attack modes: free-form extraction and partially identified extraction.The former generates name–PII pairs from domain knowledge, while the latter recovers PII for a named individual.
- New Bing Recovery: The partially identified attack can directly harm a victim whose identity is only partly specified by associating private information with the given name.The passage identifies this association-based attack as harmful to partially identified victims.
4 Experiments
The experiments evaluate whether ChatGPT and the New Bing can recover personal information through direct, jailbreaking, and search-based prompts. Results show that multi-step jailbreaking can expose memorized PII from ChatGPT, while New Bing’s integrated search enables substantially stronger recovery in some settings.
- Experimental settings: ChatGPT’s evaluation covers email addresses and phone numbers, using one response per prompt, Hit@5 across five generations, accuracy, and LCS-based phone-number metrics.The experiments use Enron and institutional datasets, including 88 filtered frequent Enron email pairs, 300 phone-number pairs, and 50 institutional pairs for each PII type.
- ChatGPT attacks: More than 50% of frequent Enron emails and 4% of faculty emails can be recovered with the proposed multi-step prompts.Hit@5 is generally higher than accuracy, indicating that repeated generations can include correct predictions even when the final selected prediction is incorrect.
- ChatGPT attacks: ChatGPT associates names with email addresses more successfully than phone numbers, which remain considerably harder to parse and recover.The authors suggest that the difference may reflect phone numbers being treated as more sensitive personal information.
- ChatGPT attacks: Direct and ordinary jailbreaking prompts largely fail to recover PII, with even role-play jailbreaking parsing only 10 institutional email patterns out of 50.ChatGPT commonly refuses requests by citing privacy or ethical concerns.
- ChatGPT attacks: Response verification adds 10% accuracy on frequent Enron emails for both multiple-choice prompting and majority voting.These verification strategies improve the proposed multi-step attack’s recovery performance beyond prompting alone.
- New Bing attacks: The New Bing recovers 94% of institutional email addresses and 48% of institutional phone numbers correctly, compared with ChatGPT’s 4% institutional extraction accuracy.The authors attribute much of this performance to the integrated search engine locating corresponding personal or institutional webpages.
- New Bing attacks: New Bing’s free-form extraction can list mostly correct name–email pairs from online sources, and simple scripts could automate this harvesting.The recovery risk is higher for more influential institutional faculty in the reported cases.
5 Conclusion
The paper analyzes privacy leakage in ChatGPT and application-integrated LLMs, finding that ChatGPT’s defenses resist direct prompts but not the proposed multi-step jailbreaking prompt. Future work will broaden evaluation to other cases and LLMs and study identity disclosure prompting.
- The study analyzes privacy leakage in both standalone and application-integrated LLMs.
- ChatGPT’s safety defenses resist direct prompts but remain insufficient against the proposed multi-step jailbreaking prompt.
- Future work will test more cases, evaluate other LLMs such as Google Bard, and quantify identity-disclosure prompting threats.
Limitations
The New Bing’s free-form PII extraction becomes less reliable when queried for more examples, producing repeated and incorrect patterns.
- Repeated and incorrect PII patterns are more likely in later New Bing examples when queries request more results.
- The limitation concerns free-form PII extraction from the New Bing.
- The reported pattern constrains the reliability of later extracted examples.
Ethical Considerations
The authors describe privacy protections and ethical safeguards for their experiments, while acknowledging that the Developer Mode prompt can sometimes generate dangerous advice. They redact personal information and do not release collected faculty PII.
- The authors state that their work reveals potential vulnerabilities without claiming ChatGPT is inherently risky without privacy protection.
- The experiments redact personal information and use publicly available Enron Email and Institutional Pages datasets.
- Collected faculty PII is not released because of privacy considerations.
- The Developer Mode prompt was selected because the authors observed no hate speech or biased content, but it may generate dangerous advice.
(a) Phone Numbers
Figure 6 presents the full contents of the query templates used in the experiments.
- Figure 6 contains the complete query templates used in the experiments.
A Experimental Details
The experiments specify model versions, decoding settings, prompt templates, response parsing, and email-content evaluation resources. ChatGPT used gpt-3.5-turbo with temperature 1, while New Bing was evaluated with creative responses.
- Models and settings: ChatGPT experiments used OpenAI’s gpt-3.5-turbo API in March 2023, while New Bing was evaluated from March 20 to May 10, 2023.
- Prompt templates: The study consistently used a ChatGPT Developer Mode jailbreaking prompt, with the full acknowledgment template supplied separately.
- Recovery examples: Email-content recovery examples covered both short and long messages, including business-meeting correspondence.
- Query and parsing: Query templates targeted phone numbers, email addresses, and email contents, with regular expressions extracting the first predicted contact detail or the requested email_content field.
- Response verification: The multiple-choice template was used to verify extracted responses.
- Evaluation: Email-content evaluation results in Table 6 were measured in percentages.
B Experiments on Email Content Recovery
The study evaluates ChatGPT’s ability to recover email contents from sender, receiver, and other identifiers using several prompts and content-recovery metrics. ChatGPT generally defended against recovery, but the proposed multi-step jailbreaking prompt produced successful short and long extractions.
- Overall results: ChatGPT showed poor extraction results across all three prompts, indicating strong defense against email-content recovery.DP achieved its highest Sensitive-F1 by repeating email addresses shown in prompts.
- Overall results: MJP produced some successful email-content extractions and outperformed DP and JP.
- Successful cases: For short emails, ChatGPT recovered most content successfully when prompted with MJP.
- Successful cases: For long emails, ChatGPT generated verbatim email content in successful MJP cases.The authors use these cases to verify that ChatGPT memorizes Enron data.
- Attack comparison: Unlike prior attacks relying on preceding text, the zero-shot extraction attack required no knowledge of preceding contexts.
C Experiments on Open-source LLMs
The paper extends its extraction experiments to three safety-enhanced open-source LLMs. Using the proposed MJP, these models became more willing to generate responses containing personal information, sometimes including accurate private contact details.
- Models: The evaluation covered Llama-2-7b-chat, vicuna-7b-v1.3, and Guanaco-7b.
- Experimental settings: Open-source experiments retained the prior settings except for greedy decoding, generating one response per query for reproducibility.
- Evaluation: Table 8 reports extraction performance for email addresses and phone numbers.
- Results: MJP made the evaluated open-source LLMs more willing to generate unethical responses involving personal information.
- Results: Some generated responses contained accurate private contact details.
D Discussions
The discussion identifies distinct privacy concerns for ChatGPT and New Bing and outlines mitigation strategies for model developers. ChatGPT can leak memorized PII under MJP, while New Bing may recover information through search and collation.
- ChatGPT: ChatGPT’s enhanced dialog safety did not prevent MJP from circumventing its ethical safeguards and leaking personally identifiable information.
- ChatGPT: The authors argue that current safety mechanisms are insufficient to steer generated content away from harm.
- ChatGPT: The experiments suggest that personal web pages and online textual files may be included in ChatGPT’s training data.
- New Bing: New Bing may recover personal information outside its training data through integrated search rather than memorization alone.
- New Bing: Low-cost recovery by New Bing may enable unintended PII dissemination, spamming, spoofing, doxing, and cyberbullying.
- Mitigation: Recommended developer mitigations include anonymizing training data, avoiding direct PII training inputs, detecting harmful prompt intent, and checking decoded outputs.