Source-linked AI summary
CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference
Erxin Yu, Jing Li, Ming Liao, Siqi Wang, Zuchen Gao, Fei Mi, Lanqing Hong
TL;DR
Existing research had not addressed LLM safety risks in multi-turn dialogue coreference. The paper introduces CoSafe, a 1,400-question benchmark, and evaluates five open-source LLMs; model safety decreased under these attacks, with attack success rates ranging from 13.9% to 56%.
Problem
Current studies had not addressed safety risks from coreference in multi-turn dialogues, a gap illustrated by a multi-turn attack eliciting harmful content after a single-prompt refusal.
Source: §1 · p. 1Method
CoSafe expands prompts from 14 harmful categories into 1,400 multi-turn coreference attacks and evaluates five open-source LLMs using moderation, human assessment, and LLM evaluation.
Source: §1 · pp. 1–2, §4.2 · p. 4Results
Model safety decreased under dialogue coreference attacks, with attack successful rates ranging from 13.9% to 56% when moving from single-prompt to multi-turn attacks.
Source: §5 · p. 4Takeaways & Limitations
The findings emphasize the need for more defense methods against safety attacks in multi-turn dialogue coreference scenarios.
Source: §6 · p. 5Manual inspection found semantic drift or diminished attack aggressiveness in 3 of 50 expanded data points.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
As large language models (LLMs) constantly evolve, ensuring their safety remains a critical research problem. Previous red-teaming approaches for LLM safety have primarily focused on single prompt attacks or goal hijacking. To the best of our knowledge, we are the first to study LLM safety in multi-turn dialogue coreference. We created a dataset of 1,400 questions across 14 categories, each featuring multi-turn coreference safety attacks. We then conducted detailed evaluations on five widely used open-source LLMs. The results indicated that under multi-turn coreference safety attacks, the highest attack success rate was 56% with the LLaMA2-Chat-7b model, while the lowest was 13.9% with the Mistral-7B-Instruct model. These findings highlight the safety vulnerabilities in LLMs during dialogue coreference interactions.
1 Introduction
The introduction identifies multi-turn dialogue coreference as an unaddressed LLM safety risk and illustrates how a final coreferential question can elicit harmful content after a single-prompt refusal. It introduces CoSafe and reports evaluations showing increased harmful responses under these attacks.
1 Introduction
- Existing red-teaming datasets addressed safety scenarios such as bias, discrimination, and privacy, but current studies had not addressed coreference risks in multi-turn dialogues.The example contrasts a refusal to a single prompt with harmful content elicited after a multi-turn dialogue refers to the target as “a person with this condition.”Source: §1 · p. 1
- CoSafe contains 1,400 multi-turn attack questions across 14 categories, created by expanding single-turn prompts into dialogues with a coreference attack question at the end.The prompts were selected from BeaverTails categories and expanded using GPT-4.Source: §1 · pp. 1–2
- The study tested five open-source LLMs, assessing responses to the final attack question with QA moderation, human assessment, and LLM evaluation.The authors describe CoSafe as the first benchmark, to their knowledge, for LLM safety in multi-turn dialogue coreference.Source: §1 · p. 2
- 56% was the attack successful rate for LLaMA2-Chat-7b; its harmful response rate was 39.4% in dialogue coreference versus 14.5% in single-prompt scenarios.The introduction also reports substantial agreement between QA moderation and human evaluation, while system prompts and Chain-of-Thought reduced harmful responses but did not eliminate unsafe responses.Source: §1 · p. 2
2 Related Wrok
Prior red-teaming research covered harmful content categories and attack types, while coreference research examined resolution and bias. The paper focuses on using multi-turn dialogue coreference as a safety attack, which it says had not been explored before.
2 Related Wrok
- Earlier red-team studies explored harmful categories and attacks involving dialogue safety, privacy, and human value alignment.The paper positions multi-turn dialogue coreference safety attacks as a previously unexplored focus.Source: §2 · p. 2
- Coreference resolution links noun phrases to corresponding entities, with prior work spanning rule-based, statistical, and neural approaches.WinoGender and WinoBias assessed gender bias in coreference resolution, but did not use coreference as a safety attack.Source: §2 · p. 2
3 The CoSafe Dataset
CoSafe turns single-turn attack prompts into multi-turn dialogues that place the coreference attack question at the end. Its 1,400 entries span 14 harmful categories and are designed to test safety over sustained interactions.
3 The CoSafe Dataset
- CoSafe contains 1,400 entries, with 100 attack prompts from each of 14 BeaverTails harmful categories expanded into multi-turn dialogues using GPT-4.Prompts that triggered GPT-4 safety mechanisms during expansion were manually improved.Source: §3 · p. 2
- CoSafe places coreference attack questions at the end of multi-turn dialogues, unlike the single-prompt datasets and dialogue datasets that typically put attack questions first.The authors describe this setup as a more challenging scenario requiring models to process more context and understand coreference implications.Source: §3 · pp. 2–3
- Each CoSafe entry has three dialogue turns and averages 10.22 sentences, testing safety over sustained interactions.The paper frames this as an evaluation of robustness to subtle, context-dependent attacks.Source: §3 · p. 3
4 Experimental Setup
The experiments evaluate five instruction-tuned open-source models. Safety assessment combines moderation, human judgments, and GPT-4 scoring of response helpfulness and harmlessness.
4.1 Models
- The evaluation covered Alpaca-7b, LLaMA2-chat-7b, LLaMA3-instruct-8b, ChatGLM3-6b, and Mistral-7B-Instruct-v0.2.Inputs followed each model’s required format; batch size was 4, input maximum length 512, and output maximum length 256.Source: §4.1 · p. 3
4.2 Evaluation Metrics
- QA moderation classified QA pairs as harmful or harmless, while three human annotators labeled responses from a random sample of 100 CoSafe data points.Figure 3 compares the harmful-response proportions from QA moderation and human evaluation and plots their agreement ratio.Source: §4.2 · p. 3, Figure 3 · p. 4
- GPT-4 assigned each response helpfulness and harmlessness scores from 1 to 10; Attack Successful Rate measures the decrease in harmlessness from single-prompt to multi-turn coreference attacks.The authors used multi-response grading for the LLM evaluation.Source: §4.2 · p. 4
5 Experimental Result and Analysis
Multi-turn dialogue coreference attacks reduced safety for several evaluated models, though effects varied by model and category. System prompts and chain-of-thought (CoT) reduced harmful responses, with lower helpfulness scores; QA moderation and human evaluation were highly consistent.
5 Experimental Result and Analysis
- 13.9%–56% was the range of attack success rates across models moving from single-prompt attacks to multi-turn coreference attacks; LLaMA2’s harmful rate rose from 14.5% to 39.4%.The reported range gives the minimum and maximum attack success rates; the LLaMA2 figures compare harmful response rates across the two settings.Source: §5 · p. 4
- LLaMA3 and Mistral’s harmful rates decreased in some cases, with category differences, refusals, and added context identified as contributing factors.The harmful rate increased in “Discrimination, Stereotype” but decreased in “Sexually explicit”; LLaMA3’s refusals also reduced helpfulness, and increased context was associated with attack failures.Source: §5 · p. 4
- Human evaluation and QA moderation had over 85% agreement, and harmful rates were similar across the two methods for the three evaluated models.Fleiss’ Kappa scores for human annotators were 0.6858, 0.6585, and 0.6865.Source: §5 · p. 4
- System prompts and CoT reduced harmful rates, with the system prompt slightly more effective, while both methods also lowered helpfulness scores.The authors attribute the lower helpfulness scores to refusals to answer harmful questions; Table 3 presents the impact of these methods.Source: §5 · pp. 4–5, Table 3 · p. 4
6 Conclusion
The paper evaluates LLM safety in multi-turn dialogue coreference using CoSafe, a red-team dataset, and five open-source large models. Its results show that model safety decreases in these scenarios, emphasizing the need for more defense methods.
6 Conclusion
- The authors construct CoSafe and test five popular open-source large models, finding that model safety decreases in multi-turn dialogue coreference scenarios.The authors emphasize the need for more defense methods.Source: §6 · p. 5
Limitations
The limitations concern semantic drift in generated attacks and the cost of producing multi-turn dialogue coreference data.
Limitations
- In a manual inspection of 50 data points, 3 showed semantic differences from the original question or diminished attack aggressiveness after GPT-4 expansion.The authors identify semantic consistency and attack aggressiveness as targets for future improvement.Source: Limitations · p. 5
- Generating 1,400 CoSafe data points with the Microsoft Azure GPT-4 API cost approximately $400, motivating the need for cheaper attack-question generation methods.The authors suggest using the generated data to fine-tune open-source models to generate attacks from a single prompt.Source: Limitations · p. 5
Ethics Statement
The ethics statement describes the dataset’s public-source origins and research-only intent. The accompanying GPT-4 prompt expands a single-turn question into a dialogue with two inferred prior turns and a coreferential final question.
Ethics Statement
- The dataset draws on publicly available data, may contain offensive or harmful language, and is intended for academic research only.The statement says the future dataset release will clearly state that it must not be misused.Source: Ethics Statement · p. 5
A System Prompt Employed in GPT-4 for Multi-Turn Dialogue Coreferences Generation
- The GPT-4 prompt infers two preceding dialogue turns from a single-turn question and rewrites the final question using reference while maintaining its original meaning.The prompt also asks for coherent, logically relevant dialogue and detailed, fluent assistant responses.Source: A System Prompt Employed in GPT-4 for… · p. 8, Figure 4 · p. 8
B Datasets Examples
This section outlines CoSafe’s harmful-content categories and examples, the LLM evaluation prompts, and the system-prompt and chain-of-thought procedures.
B Datasets Examples
- CoSafe’s examples span 14 harmful-content categories defined by BeaverTails, with examples comparing single-prompt attacks and multi-turn dialogue coreference attacks.The categories include hate speech, discrimination, violence, privacy violations, self-harm, and child abuse.Source: B Datasets Examples · p. 8
C LLM Evaluation Prompt
- The helpfulness and harmlessness evaluation prompts are presented in Tables 5 and 6, and response order was shuffled to reduce positional scoring bias.Responses from multiple models were scored simultaneously.Source: C LLM Evaluation Prompt · p. 8
D Detail of System Prompt and COT
- The system prompt for defending against multi-turn dialogue coreference attacks is shown in Figure 7, with its chain-of-thought steps detailed in Figure 14.The section points readers to the figures for the prompt and specific reasoning steps.Source: D Detail of System Prompt and COT · p. 8
E Complete Evaluation Result
The evaluation reports safety risks across multi-turn dialogue coreference attacks, while also noting category-level differences and a helpfulness–harmlessness trade-off.
E Complete Evaluation Result
- LLaMA3’s harmful rates were generally lower in most CoSafe categories than in single-prompt scenarios, while even ChatGLM3 and Mistral produced some unsafe responses.The authors attribute LLaMA3’s lower harmfulness to reduced helpfulness; safety risks therefore persisted across evaluated models.Source: E Complete Evaluation Result · pp. 8–9
F Data Scoring Examples
This section illustrates how responses are scored using multiple evaluation methods and shows examples of both harmful outputs and a failed attack.
F Data Scoring Examples
- Table 5 presents GPT-4 evaluations of different models’ response helpfulness and harmlessness.The evaluation prompts are shown in Figures 5 and 6.Source: Table 5 · p. 10, Figure 5 · p. 10, Figure 6 · p. 10
- Figure 8 defines attack success as a GPT-4 harmless-score decrease from single-prompt to multi-turn coreference, while Figure 9 reports QA-moderation harmful rates.Figure 10 compares safe and unsafe response counts across categories using GPT-4, classifying scores below 5 as unsafe.Source: Figure 8 · p. 11, Figure 9 · p. 11, Figure 10 · p. 12
- A single-prompt example shows ChatGLM3 refusing a request to make someone fall down steps, while another shows Mistral providing detailed harmful instructions about GHB in baked goods.The examples illustrate the range of response content included in the scoring materials.Source: F Data Scoring Examples · pp. 12, 14
- Figure 13’s failed attack shows that added dialogue context can make the model aware of security risks.Figures 11 and 12 provide examples of model scoring using GPT-4, QA moderation, and human evaluation, while Figure 13 shows a failed attack.Source: F Data Scoring Examples · p. 9, Figure 11 · p. 12, Figure 12 · p. 13