Contextual Integrity in LLMs via Reasoning and Reinforcement Learning

1 Introduction

• CI requires information disclosed to complete a task to be appropriate to its context, while broad information access can make strict separation infeasible in practice.For example, conventional retrieval may return relevant information without considering CI.Source: §1 · pp. 1–2

1.1 Our Contributions

• The paper proposes CI-focused RL post-training to improve LLM reasoning and reduce inappropriate disclosure through structured contextual assessment.The authors describe RL as a way to instill CI reasoning capabilities in LLMs.Source: §1 · p. 2, §1.1 · p. 3
• An automatically generated dataset of approximately 700 examples spans diverse scenarios and CI norms, supporting reduced inappropriate sharing while maintaining high task performance across model families.The reported performance includes results on the dataset and its disjoint test set.Source: §1.1 · p. 3
• On human-annotated PrivacyLens, the method reduces privacy leakage by up to 40%, demonstrating transfer from synthetic training scenarios.The authors describe this as transfer of CI reasoning capabilities to real-world contexts.Source: §1.1 · p. 3
• The authors identify their use of RL to instill CI reasoning as a first and argue that CI reasoning should be part of alignment for real-world LLM agents.They report successful transfer from synthetic scenarios to human-annotated CI benchmarks.Source: §1.1 · p. 3

2 Background

• Contextual integrity concerns proper information flow among context-specific actors and attributes under transmission principles such as purpose, terms, and social norms.The paper notes that CI norms can be variant, subjective, and evolving, which can make them difficult to encode completely.Source: §2 · p. 3
• GRPO reduces RL computational overhead by eliminating the need for a critic network while optimizing the LLM policy.Its objective includes a KL-divergence weight and a clipping parameter.Source: §2 · p. 3
• GRPO samples groups of complete answers per prompt and estimates each answer’s advantage from its reward relative to the group.The advantage uses the group reward mean and standard deviation.Source: §2 · p. 3

3.1 Chain of Thought Reasoning for Contextual Integrity

• CI-CoT prompts models to reason about task context and each personal attribute’s relevance before producing a separate final response.The prompt asks whether sharing an attribute is necessary, helpful, optional, or inappropriate.Source: §3.1 · p. 4

3.2 Synthetic dataset curation

• Synthetic dataset seeds vary task domains and contexts under confidentiality, proportionality, and consent transmission principles.The seed scenarios include domains such as healthcare, finance, education, and work.Source: §3.2 · p. 5
• GPT-4 expands seeds into vignettes with task and CI context, then creates natural-format examples with required and restricted information intermixed and annotated.The final examples use keywords from required and restricted values to support string-matching rewards.Source: §3.2 · p. 5

3.3 Contextual integrity optimized RL training

• CI-RL uses GRPO with a reward combining CI scoring and a formatting criterion requiring explicit reasoning and task completion tags.The CI score evaluates the task-completion text against required and restricted keywords.Source: §3.3 · pp. 5–6

4 Experiments

• The dataset is split into 590 training, 66 evaluation, and 73 test examples, and experiments vary model size and family.The test results use the best checkpoint selected by validation score; models include Qwen2.5 sizes from 1.5B to 14B, Llama-3.1-8B, and Mistral-7B.Source: §4 · p. 6
• Integrity measures exclusion of restricted information, Utility measures inclusion of required information, and Complete requires both.The metrics are averaged over test examples.Source: §4 · p. 6

4.1 Results

• CI-RL improves Integrity and Complete across all models while maintaining comparable or improved Utility.Qwen2.5-7B-Instruct after CI-RL reaches 75.0% Integrity and 48.4% Complete, surpassing baseline Qwen2.5-14B-Instruct at 51.6% Integrity and 37.5% Complete.Source: Table 1 · p. 7, §4.1 · p. 7

4.2 Ablation Studies

• After CI-RL training, instruction-tuned LLMs score substantially higher than LRMs on integrity, utility, and task completion.The authors hypothesize that distilled models’ scientific and code-domain optimization may limit broader real-world understanding; they do not consider the gap inherent to the LRM paradigm.Source: §4.2 · p. 7, Table 2 · p. 7
• Changing the reward function’s weights for required and restricted keywords can shift the model’s prioritization between integrity and utility.The reward-weight ablation results are presented in Appendix A.Source: §4.2 · p. 7

5 Evaluation - PrivacyLens

• PrivacyLens scores helpfulness from 0 to 3, counts responses disclosing sensitive attributes in leakage rate, and calculates adjusted leakage rate only for helpful responses scoring 2 or 3.Adjusted leakage rate measures leakage among useful responses, addressing the safety-helpfulness trade-off.Source: §5 · p. 8

5.1 Results

• GPT-4o (version 2024-11-20) at temperature 0 judged PrivacyLens results; OpenAI models were omitted to avoid judge bias toward its own generations.Quantitative results are summarized in Table 3.Source: §5.1 · p. 8
• CI-CoT reduces leakage rate and adjusted leakage rate, but its more conservative disclosure is accompanied by reduced helpfulness.The table compares models on PrivacyLens; both leakage measures are lower with the CI-CoT prompt.Source: §5.1 · p. 8, Table 3 · p. 9
• CI-RL achieves lower leakage rates and adjusted leakage rates while preserving or enhancing helpfulness; it improves over CI-CoT on PrivacyLens despite longer contexts and detailed tool use.The benchmark differs substantially from the synthetic dataset in context length and emphasis on tool use.Source: §5.1 · p. 8
• Across experiments, frontier models have lower leakage rates and higher helpfulness than their substantially smaller open-weight counterparts.Source: §5.1 · p. 9

• CI-Bench [5] and Confaide [25] find that LLM assistants struggle with nuanced contextual-integrity judgments and social reasoning.Source: §6 · p. 9
• PrivacyLens [30] evaluates privacy leakage in agent actions and reports a gap between responses to probing questions and behavior in real agentic scenarios.Source: §6 · p. 9
• Prior agent defenses use data minimization, firewalls, or information-flow cards [3], [1], [7], while this work trains LLMs directly, an orthogonal and potentially complementary approach.Source: §6 · p. 9

7.1 Limitations and Future Work

• The feasibility study uses synthetic CI data because high-quality human annotations are scarce and expensive to collect at scale.The authors call for human-annotated datasets to further validate and refine the findings.Source: §7.1 · pp. 9–10
• The keyword-based reward is auditable but coarse; richer reward models or rubric-guided judges could better capture context-sensitive norms, given abundant high-quality data.Source: §7.1 · p. 10
• The discussion frames CI-RL as potentially complementary to external privacy defenses and notes that changing norms and custom policies require further support.Proposed directions include policy injection, versioning, and rule refresh.Source: §7.1 · p. 10
• Comparisons between supervised fine-tuning and reinforcement-learning approaches on CI frameworks remain an open research direction.Source: §7.1 · p. 10
• Training uses relatively simple semi-structured inputs, although gains also appear on free-form PrivacyLens chats and a single-model ConfAIde replication.The authors say more complex settings are needed to further validate robustness.Source: §7.1 · p. 10

7.2 Conclusion

• The approach reduces inappropriate information leakage while preserving task utility, with improvements across model families and on PrivacyLens.The authors describe transfer from synthetic training scenarios to the human-annotated benchmark.Source: §7.2 · p. 10

• The authors report that the paper’s claims accurately describe its CI reasoning, RL framework, synthetic-data experiments, and transfer to PrivacyLens.Source: NeurIPS Paper Checklist · p. 14
• The paper reports a detailed limitations discussion in Section 7.Source: NeurIPS Paper Checklist · p. 14
• The authors report documenting the synthetic-data pipeline, prompts, experimental settings, and hyperparameters for reproducibility.The pipeline and prompts are described in Section 3.2 and Appendix B; experimental settings are in Section 4 and hyperparameters in Appendix F.Source: NeurIPS Paper Checklist · pp. 15–16
• The authors report providing code in the supplementary material and the dataset on HuggingFace.Source: NeurIPS Paper Checklist · p. 16
• The authors did not complete multiple experimental runs to report error bars or standard deviations by the submission deadline.Source: NeurIPS Paper Checklist · p. 16

Integrity-utility trade-off via reward function design

• Increasing α generally improves task performance and Helpfulness, while increasing γ improves integrity and reduces leakage across both evaluation settings.Source: A Supplementary Results · p. 21, Table 4 · p. 21

ConfAIde Benchmark

• CI-RL also yields meaningful improvements on the ConfAIde benchmark.Source: A Supplementary Results · p. 21
• CI-RL improves correlation between human and model judgments on ConfAIde Tier 1 and Tier 2, which assess sensitivity and expected information flow.Source: A Supplementary Results · p. 21, Table 5 · p. 21

Tier 3: Theory of mind as context

• CI-RL substantially reduces private-information leakage in Tier 3’s more complex, multi-party scenarios.Source: A Supplementary Results · p. 21, Table 6 · p. 21

Tier 4: Private and public information flow

• 0.85 to 0.45: CI-RL reduced meeting-summary completions that leaked a secret or omitted required information.Source: A Supplementary Results · p. 22, Table 7 · p. 22

B.1 Examples

• Example vignettes illustrate Confidentiality, Proportionality, and Consent rules by distinguishing information appropriate to share from restricted details.Source: §B.1 · pp. 22, 24, 26

B.2 Prompts for seed generation

• Seed-generation prompts define cases through scenario, domain, user intention, sender, recipient, data subject, allowed and disallowed information, and a transmission principle.The prompts define Confidentiality as sharing only with authorized parties, Consent as requiring explicit consent, and Proportionality as limiting sharing to what is necessary.Source: §B.2 · p. 28

B.3 Prompts for dataset generation

• Dataset-generation prompts expand structured seeds into concrete stories, prior messages, and sender requests, retaining the seed’s permitted and disallowed information flows.Cases span email and chat scenarios and use uniquely identifiable information strings to make leakage challenging to assess while keeping ground truth easy to extract.Source: §B.3 · p. 29
• The generation prompt pairs a first-person user task with concrete, neutrally keyed information and concise allowed/disallowed annotations that mirror its nested structure.When the seed calls for prior conversations, it requests natural conversations and short, unique identifiers that can be checked automatically.Source: §B.3 · p. 30

C Prompt Example

• The example applies this prompt to a hotel-room booking request, with user attributes including booking details, insurance and document identifiers, and medical information.The scenario identifies the user as sender and hotel staff as recipient, and the sample attributes include check-in and check-out dates, room preferences, prescription details, and medical notes.Source: C Prompt Example · p. 31
• The example prompt asks the model to reason about the task and each attribute’s appropriateness under contextual integrity, then disclose only attributes justified in that reasoning.The requested assessment considers whether information is necessary, helpful, optional, or inappropriate in light of the context’s purpose, roles, and transmission principles.Source: C Prompt Example · p. 31

D An example model generation evolution during training over a validation example

• The example instruction requires reasoning about task context and each attribute before generating a response that uses only justified sensitive information.It asks the model to assess whether each attribute is necessary, helpful, optional, or inappropriate and justify the decision using contextual integrity.Source: D An example model generation… · p. 32
• The earlier response says salary and the reason for the career change can be shared, though they are secondary to the main news.Its draft email includes the reason for the career change and the company’s focus.Source: D An example model generation… · p. 32
• The later response marks Charlie’s salary as unnecessary and potentially intrusive without consent, while treating the job details as necessary.It classifies the reason for the career change and the company overview as optional, then says it will focus on essential details.Source: D An example model generation… · p. 33
• The later draft shares Charlie’s new role and company focus without disclosing salary or the reason for the career change.It describes Charlie as a Senior Product Analyst at GreenTech Innovations, a renewable energy software company.Source: D An example model generation… · p. 33

E Outputs on PrivacyLens

• Figure 4 presents an example of Qwen2.5-7B-IT + CI-RL running on PrivacyLens.The caption identifies an example, not a benchmark comparison.Source: Figure 4 · p. 34
• Figure 5 presents another example from the PrivacyLens outputs.The passage does not specify the model or report an outcome for this example.Source: Figure 5 · p. 35

F Supplementary Details in Experiments

• Training samples 16 inference results for each query in batches of 32 and uses four mini-batch optimization steps per GRPO update.The reported settings include a learning rate of 1 × 10^-6, KL weight β = 0.001, and clip ratio ϵ = 0.2.Source: F Supplementary Details in Experiments · p. 36
• The synthetic dataset spans 9 applications and 3 privacy principles, averaging 2.73 allowed and 2.08 disallowed attributes per instruction across 526 and 322 unique attribute types, respectively.A manual audit of randomly sampled instances found they adhered to CI expectations and scenario consistency.Source: F Supplementary Details in Experiments · p. 36
• CI-RL often produces more focused completions by discouraging unnecessary or privacy-violating content, although its reward function does not explicitly encourage longer outputs.The authors characterize overall response length as moderate while noting that lengths vary with task complexity.Source: F Supplementary Details in Experiments · pp. 36–37
• Weak correlations link thinking-token counts with both helpfulness and leakage scores, suggesting lengthy reasoning traces may not be necessary and could offer cost advantages in agentic settings.The authors report that longer chains of thought are not strongly associated with improved task performance or increased risk.Source: F Supplementary Details in Experiments · p. 37
• Training and evaluation use 8 nodes with 8 NVIDIA A100 GPUs each, and each task takes 8–68 hours depending on model size.Each GPU has 80 GB of memory.Source: F Supplementary Details in Experiments · p. 37

G Supplementary Related Work

• Chain-of-thought research spans step-by-step prompting, decoding refinements, reasoning-path search, rationale distillation, and faithfulness and safety concerns [39] [6].The passage directs readers to a comprehensive survey for a field taxonomy [6].Source: G Supplementary Related Work · p. 37
• PPO [29] and GRPO [31] have supported high-performing reasoning models, while RL post-training has been applied to math [31], visual generation [48], and explainable agents.The passage points to surveys for broader coverage [43] [36] [21] [42].Source: G Supplementary Related Work · p. 37
• Related safety work examines harmful-content manipulation [22], moderation guidance [24], unsafe content in long reasoning traces [17], reasoning-based guardrails [40], prompt injection that induces excessive reasoning tokens in untrusted contexts [20], and reasoning-disrupting backdoors [49].Source: G Supplementary Related Work · p. 37
• Contextual-integrity work focuses on maintaining integrity within a given context, unlike cited safety work that primarily addresses responses to unsafe queries.The paper characterizes these directions as orthogonal.Source: G Supplementary Related Work · p. 37