PaperlayerView exact revision on arXiv

Source-linked AI summary

Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answering

Hwan Chang, Yumin Kim, Yonghyun Jun, Hwanhee Lee

arXiv:2505.15805v2cs.CL

TL;DR

In sensitive domains, LLMs must respect confidentiality policies embedded in context. CoPriva evaluates this challenge using realistic question-answering scenarios with explicit policies and direct and indirect attacks. Across 10 LLMs, policy violations are frequent and worse under indirect attacks.

  • Problem

    In sensitive domains, LLMs need to respect contextual confidentiality policies and avoid revealing information that users prohibit.

    Source: §1 · p. 1

  • Method

    CoPriva evaluates contextual policy adherence using 4,184 realistic question-answering instances with explicit security policies and direct and indirect attack queries.

    Source: §1 · p. 2

  • Results

    Across 10 state-of-the-art LLMs, models frequently violate contextual security policies and leak sensitive information, with the vulnerability exacerbated by indirect attacks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings underscore the need for improved security mechanisms for deploying LLMs where strict adherence to user-defined policies is required.

    Source: §1 · p. 2

    The study evaluates and analyzes vulnerabilities but does not propose defense or safeguard mechanisms.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As Large Language Models (LLMs) are increasingly deployed in sensitive domains such as enterprise and government, ensuring that they adhere to user-defined security policies within context is critical-especially with respect to information non-disclosure. While prior LLM studies have focused on general safety and socially sensitive data, large-scale benchmarks for contextual security preservation against attacks remain lacking. To address this, we introduce a novel large-scale benchmark dataset, CoPriva, evaluating LLM adherence to contextual non-disclosure policies in question answering. Derived from realistic contexts, our dataset includes explicit policies and queries designed as direct and challenging indirect attacks seeking prohibited information. We evaluate 10 LLMs on our benchmark and reveal a significant vulnerability: many models violate user-defined policies and leak sensitive information. This failure is particularly severe against indirect attacks, highlighting a critical gap in current LLM safety alignment for sensitive applications. Our analysis reveals that while models can often identify the correct answer to a query, they struggle to incorporate policy constraints during generation. In contrast, they exhibit a partial ability to revise outputs when explicitly prompted. Our findings underscore the urgent need for more robust methods to guarantee contextual security.

1 Introduction

The introduction frames preventing disclosure of contextually prohibited information as essential in sensitive LLM use, while noting that evaluation of contextual policies under bypass attempts remains limited. It presents CoPriva and reports frequent policy violations, especially under indirect attacks.

1 Introduction

  • The introduction identifies a gap in evaluating whether LLMs uphold dynamically provided contextual security policies against attempts to bypass restrictions.Prior work often addresses toxicity, bias, or disclosure of publicly known sensitive data.Source: §1 · pp. 1–2
  • CoPriva contains 4,184 realistic-context instances with explicit withholding policies and both direct and challenging indirect attack queries.The benchmark evaluates whether models adhere to user-defined policies in question answering.Source: §1 · p. 2
  • Current LLMs frequently violate contextual security policies, with the vulnerability worsening under indirect attacks in sensitive applications.The paper evaluates 10 state-of-the-art models and reports frequent leakage of sensitive information.Source: §1 · p. 2
  • The paper proposes contextual security policy preservation against attacks as a task relevant to real-world LLM applications.Source: §1 · p. 2

2 CoPriva: Benchmarking Contextual Security Preservation in LLMs

CoPriva evaluates whether LLM answers can preserve contextual non-disclosure policies while still meeting users’ information needs. It uses realistic meeting transcripts, explicit policies, direct and indirect attacks, quality control, and separate leakage and faithfulness measures.

2.1 Task Formulation

  • The task tests whether answers withhold policy-prohibited information while still providing the non-confidential content needed to answer user queries.Policies distinguish confidential targets from non-confidential targets; queries include direct and indirect attacks against transcript content.Source: §2.1 · pp. 2–3

2.2 Dataset Construction Pipeline

  • CoPriva builds on QMSum meeting transcripts, generating paired complementary policies that divide target information into confidential and non-confidential sets and testing direct and high-level indirect queries.The source spans product, academic, and committee meetings; policies are generated with GPT-4.1, and high-level queries reuse QMSum questions that can span both target types.Source: §2.2 · pp. 3–4

2.3 Dataset Quality Control

  • An automatic GPT-4.1 filtering pipeline checks policy targets, discussion-summary faithfulness, and policy consistency, while human validation assesses retained policies and summaries.The filtering removes 688 low-quality policies; the authors do not additionally filter neutral or irrelevant queries, and indirect queries are assumed grounded in meeting content.Source: §2.3 · pp. 4–5

2.4 Evaluation Pipeline

  • An LLM-as-judge evaluator measures leakage against confidential targets and faithfulness to required non-confidential content, with human annotations used to verify automatic evaluation.The paper reports perfect agreement between human annotations and automatic evaluation on the validation subset.Source: §2.4 · p. 5

2.5 Dataset Overview and Statistics

  • CoPriva contains 219 meeting transcripts averaging 12,881 tokens, 2,763 policies, and 2,092 queries each for direct and indirect attacks.The two query types enable evaluation of explicit and generalized attack scenarios.Source: §2.5 · pp. 5–6

3 Experiments

Across 10 evaluated LLMs, direct attacks yield near-zero leakage for most models, while indirect attacks sharply increase leakage. The experiments also examine factors associated with leakage and test mitigation strategies, with revision showing partial promise.

3.1 Performance under Direct and Indirect Attacks

  • Indirect attacks raise leakage by over 40 percentage points on average compared with direct attacks, where most models maintain near-zero leakage; reasoning models show no significant advantage.The evaluation covers 10 LLMs, including reasoning and non-reasoning, open-source and proprietary models.Source: §3.1 · p. 6, §3 · p. 6

3.2 Factors Influencing Leakage

  • Higher faithfulness tends to coincide with more leakage, while leakage fluctuates non-monotonically with context length.The authors suggest accurate, coherent answers may come at the cost of policy compliance; context size is not a dominant factor.Source: §3.2 · pp. 6–7, Figure 3 · p. 6, Figure 4 · p. 7
  • Leakage increases when GPT-4o models receive only query-relevant spans rather than full transcripts, and including a policy does not significantly reduce leakage.The reported policy-present leakage rates are 67.3% for Gemini 2.0 Flash and 68.3% for GPT-4o Mini; some responses safe without a policy begin leaking when one is introduced.Source: §3.2 · p. 7, Figure 5 · p. 7, Figure 6 · p. 7
  • Leakage is most frequent in the Committee domain, followed by Product and Academic, and domain characteristics may matter more than context length.The authors suggest detailed discussions and decisions in Committee meetings may contribute to the pattern.Source: §3.2 · p. 7, Figure 7 · p. 7
  • QwQ-32B and DeepSeek-R1 frequently include sensitive content in internal reasoning, and some of those cases reach final outputs: 33.2% for QwQ-32B and 34.6% for DeepSeek-R1.The authors describe this as a challenge for aligning accurate reasoning with strict policy compliance.Source: §3.2 · p. 7, Figure 8 · p. 8

3.3 Mitigation Strategies for Policy Compliance

  • Explicit warnings and 2-shot prompting still struggle to enforce policy compliance, while revision offers modest reductions and partial correction when explicitly guided.The revision strategy first answers without the policy, then revises after receiving it; models still frequently fail to fully enforce the policy.Source: §3.3 · p. 8, Table 6 · p. 8

4 Related Works

Prior work addresses socially sensitive and personal information, while contextual privacy benchmarking remains limited. CoPriva is presented as addressing the combination of contextual privacy and dynamically embedded user-defined security constraints.

LLM Security Benchmark

  • Existing LLM safety benchmarks commonly evaluate socially sensitive or personal information, but contextual privacy benchmarking remains limited.The cited discussion names Staab et al. (2023), Fan et al. (2024), Zharmagambetov et al. (2025), and Shao et al. (2024) among prior benchmark work.Source: §4 · p. 8
  • The authors identify Mireshghallah et al. (2024) as the only prior benchmark directly addressing contextual privacy, noting its limited scale and lack of dynamically embedded user-defined constraints.They characterize these limits as constraining generalizability and robustness for assessing real-world deployment risks.Source: §4 · p. 8
  • CoPriva is presented as the only dataset combining user-defined security with contextual privacy.The related-work discussion frames this combination as a more comprehensive foundation for evaluating safety in context-sensitive applications.Source: §4 · pp. 8–9

5 Conclusion

CoPriva evaluates whether LLMs preserve user-defined security policies in question answering, including under direct and indirect attacks. The conclusion reports frequent violations, especially under indirect attacks, and calls for more robust mechanisms.

  • CoPriva evaluates adherence to user-defined security policies in question-answering contexts using realistic direct and indirect attack queries.Source: §5 · p. 9
  • State-of-the-art LLMs frequently violate these policies, with especially high failure rates under indirect attacks.Source: §5 · p. 9
  • The findings highlight a gap in current LLM safety alignment and the need for more robust contextual privacy and security mechanisms.Source: §5 · p. 9

Limitations

The study evaluates vulnerabilities rather than proposing safeguards, and its construction pipeline depends on QA pairs aligned with meeting transcripts.

  • The construction pipeline relies on QA pairs aligned with meeting transcripts, so extending it to datasets without explicit QA structures requires adaptation.Source: Limitations · p. 9
  • The work analyzes LLM vulnerabilities but does not propose contextual security defenses or safeguards.Mitigation strategies are identified as future work.Source: Limitations · p. 9

Ethics Statement

The dataset uses QMSum as its sole source and does not collect new human-subject data. Its modifications are task-specific augmentations without additional personal or sensitive information beyond that already in QMSum.

  • QMSum is the study’s sole source dataset, and the authors collect no new human-subject data.QMSum is described as a public query-based meeting-summarization benchmark.Source: Ethics Statement · p. 9
  • The modifications are limited to task-specific augmentations and add no personal or sensitive information beyond what is already present in QMSum.Source: Ethics Statement · p. 9

A Dataset Generation Details

The prompt for generating security policies is shown in Table 7, and the number of policy target candidates varies with meeting-transcript content.

A.1 QMSum Dataset

The appendix reports token-length statistics for QMSum meeting transcripts and describes direct-query formats based on its Specific Query Schema List.

A.1 QMSum Dataset

  • Table 8 summarizes token-length statistics for the meeting transcripts used in CoPriva.Source: §A.1 · p. 11

A.2 Template-wise Direct Query Leakage Analysis

  • Table 9 presents formats for generating direct attack queries.Source: §A.2 · p. 11
  • The direct-query formats reference QMSum’s Specific Query Schema List.Source: §A.2 · p. 11

A.3 Dataset Quality Control Details

The appendix provides prompts for dataset quality control and LLM evaluation, then reports human validation results for the automatic filtering and evaluation pipeline.

A.3 Dataset Quality Control Details

  • Tables 10 and 11 contain the prompts used for automatic dataset quality filtering.Source: §A.3 · p. 11

A.4 Evaluation Details

  • The leakage evaluation uses an LLM-based pipeline because the task lacks an established protocol and policy-constrained entailment can be ambiguous.The framework is designed for context-sensitive judgments of both leakage and faithfulness.Source: §A.4 · p. 11

A.5 Human Validation Details

  • Human validation fully agreed with automatic security-policy filtering on 84 sampled attack QA pairs.Three fluent English speakers conducted the validation.Source: §A.5 · p. 11
  • Leakage judgments matched the LLM-based evaluation for 79 of 84 human-validated samples (94%).Source: §A.5 · p. 11

B Experimental Details

The appendix describes API access and evaluation prompts, explains the study’s comparison of reasoning and non-reasoning models without CoT, and provides indirect-attack examples.

B Experimental Details

B.1 Comparison to CoT-based Approaches

C Security-Aware QA Evaluation Prompt

D.1 Successful Cases

D.2 Failure Cases

Loading 2505.15805v2…