PaperlayerView exact revision on arXiv

Source-linked AI summary

PrivacyAlign: Contextual Privacy Alignment for LLM Agents

Manveer Singh Tamber, Abhay Puri, Marc-Etienne Brunet, Perouz Taslakian, Jimmy Lin, Spandana Gella

arXiv:2606.21710v1cs.CLcs.AIcs.IR

TL;DR

Agentic privacy decisions depend on human norms, but existing evaluation and training methods rely on automated proxies. PrivacyAlign centers human annotations in evaluation and reward modeling, improving privacy alignment in small open-weight agents and transferring gains across benchmarks.

  • Problem

    Agentic assistants can expose private details through outbound actions, while existing privacy evaluation and training methods rely on automated proxies rather than human privacy norms.

  • Method

    PrivacyAlign uses human preferences, leak/omit labels, and rationales to calibrate LLM judges and condition rewards for reinforcement-learning alignment.

  • Results

    Annotation-conditioned rewards substantially raise clean rates and transfer gains across PrivacyAlign and existing agentic privacy benchmarks, bringing a 4B open-weight model near proprietary frontier models.

  • Takeaways & Limitations

    Keeping human judgment in both evaluation and training better aligns agents with contextual privacy norms while protecting sensitive information without omitting needed content.

  • Takeaways & Limitations

    PrivacyAlign uses synthetic LLM-generated scenarios rather than real user traces, which may limit realism and data quality.

Abstract

from arXiv · show

AI agents acting on behalf of users are constantly making decisions, and for users to trust their agents, those decisions must align with what they actually want. Privacy is an important alignment problem for agents: every message, post, or tool call an agent makes is a contextual judgment about what is appropriate to share, with whom, and under which conditions. Because such judgments depend on social expectations and norms, human judgment does not merely label privacy violations but also helps define them. While existing work relies on unreliable proxies for both training and evaluation, we place human judgment at the center of agentic privacy alignment. We introduce PrivacyAlign, a dataset of 1,350 samples with 3,516 detailed annotations from 599 unique annotators across diverse scenarios where current LLMs actually leak, and use it to ground both alignment training and automated evaluation in human privacy norms. Building on these annotations, we first show that conditioning LLM judges on human annotations and explanations for reference responses to the same prompt makes their judgments more reliable. We then introduce annotation-conditioned reward modeling, which uses these annotations to score new responses during RL, and show that small open-weight agents trained with this reward better align with human privacy norms, with strong gains on PrivacyAlign and existing privacy benchmarks for agents.

1 Introduction

Agentic assistants create a contextual privacy-alignment problem because they can draw on diverse private data and make outbound disclosures whose appropriateness depends on human social norms. PrivacyAlign addresses this by centering human judgment in both training and evaluation, introducing a human-annotated dataset, automated scenario pipeline, and annotation-conditioned reward modeling.

  • Motivation: Agentic assistants can access web, email, calendar, local-document, and persistent-memory data before acting, limiting users’ reliable control over outbound disclosure.This shifts privacy risk beyond user-provided conversational context.
  • Problem: Privacy alignment requires judging whether a disclosure respects affected people’s expectations and relationships, not merely detecting sensitive content.Privacy is contextual: the same disclosure may be appropriate in one setting and a violation in another.
  • Motivation: Frontier LLMs routinely leak sensitive details from tool results and prior memories into outbound actions despite inference-time mitigations, motivating human-grounded evaluation and training.Existing privacy-leakage benchmarks and training research remove humans from the loop.
  • Contributions: PrivacyAlign contains 1,350 pairwise comparisons and 3,516 annotations from 599 unique human annotators, including per-response leak/omit labels and written explanations.The annotations support both alignment training and calibration of automated evaluation.
  • Contributions: Annotation-conditioned reward modeling conditions an LLM-judge reward on human annotations and explanations for reference responses, then scores fresh completions during RL.The paper reports that this approach can outperform trained generative reward models for aligning LLMs with human privacy norms.

2 Related Work

Prior work evaluates contextual privacy violations, mitigates them at inference time, and trains agents with increasingly specialized objectives. PrivacyAlign addresses a remaining supervision gap by centering human judgments and explanations in privacy alignment.

  • Contextual privacy benchmarks: Contextual privacy benchmarks apply contextual-integrity theory to test whether LLMs respect privacy norms in context.ConfAIde reports inappropriate private-information disclosure, PrivacyLens exposes a gap between stated awareness and behavior, and CI-Bench provides 44,000 synthetic dialogues and emails across eight domains.
  • Inference-time mitigations: Inference-time defenses add privacy controls around fixed models, including data filtering, norm reasoning, model-agnostic filtering, mediated access, and specialized multi-agent subtasks.These methods are valuable deployment-time interventions but treat privacy only as a runtime filtering problem.
  • Training agents for contextual privacy: Privacy-focused training has emphasized memorization and training-data leakage rather than agents’ contextual disclosure decisions.General alignment methods include RLHF, DPO, GRPO, RLAIF, and Constitutional AI.
  • Training agents for contextual privacy: CI-RL and PrivAct use string-matching or sensitive-item supervision, but neither aligns models with human judgments about appropriate disclosure.CI-RL trains with rewards over required and restricted values, while PrivAct uses DPO and sensitive-item lists from PrivacyLens.
  • Human annotations for reward modeling and evaluation: Human-feedback reward modeling typically converts pairwise preference annotations into scalar rewards while omitting the explanations, reasoning, and contextual assumptions behind judgments.This motivates using richer human annotation information for alignment and evaluation.

3 Generating and Annotating Privacy Scenarios

PrivacyAlign uses a fully automated pipeline to generate, filter, diversify, and cross-evaluate privacy-sensitive agent scenarios before human annotation. The resulting dataset contains 1,350 response-pair items and 3,516 retained annotations, with substantial preference agreement but weaker agreement on omission labels.

  • Scenario generation: The pipeline generates complete agent scenarios from sampled names, including profiles, stories, instructions, toolkits, final-action types, and sensitive and task-relevant information.Profile and scenario stages over-generate K = 4 candidates before selecting one based on attribute quality.
  • Scenario generation: Each scenario includes a realistic 3 to 14-call read-only trajectory and a prior-interaction memory store embedding sensitive and relevant items in naturalistic clutter.Tool calls are validated against real toolkit schemas and are generated to plausibly support the final action.
  • Filtering and diversity: Filtering retains plausible scenarios where a naive agent leaks sensitive information, then prunes near-duplicates and enforces diversity caps across domains, toolkits, actions, signatures, scopes, and model families.Near-duplicates use Qwen3-Embedding-8B cosine similarity with threshold 0.95; individual domains, toolkits, or final actions are capped at 25%.
  • Cross-model pair mining: Cross-model pair mining keeps response pairs judged sensible by all three judges, with at least one unanimously leaking response and majority agreement that one response is better.The generator and judge models are Qwen3.5-397B-A17B, gpt-oss-120b, and Nemotron-3-Super-120B-A12B.
  • Human annotation: Annotators label leakage, omission, and overall preference for two candidate actions, while providing a free-text rationale subject to quality-control audits and exclusion of low-quality or rushed annotations.Unsure annotations are manually reviewed and dropped after confirmation.
  • Dataset and agreement: 1,350 response-pair items and 3,516 retained human annotations form the final dataset, with κ = 0.606 preference agreement, κ = 0.558 leak agreement, and κ = 0.120 omit agreement.Pairwise agreement is 78.1% for preferences, 78.4% for leak labels, and 75.8% for omit labels.

4 Reward Modeling from Human Annotations

PrivacyAlign converts human annotations into pairwise rewards through either an annotation-conditioned LLM judge or a trained generative reward model. Both sources use group-relative on-policy RL with safeguards against position bias and trivially short responses.

  • Reward sources: The method studies two annotation-grounded pairwise reward sources: an LLM judge conditioned on same-prompt annotations and a generative reward model trained from structured annotation signals.The judge uses full annotations, including free-text rationales; the generative model learns from pairwise preferences and per-response leak labels.
  • Pairwise scoring: Each reward compares rollout pairs on a [−2, +2] preference scale, averages signed margins across peers, and scores both response orderings to remove position bias.For K rollouts, each rollout is compared with its K −1 peers, and reversed-order scores are averaged.
  • Annotation-conditioned judge: The annotation-conditioned judge receives the task context, reference responses, inlined annotations, and candidate pair, while treating annotations as guidance rather than ground truth.It calibrates sensitivity and relevance to the scenario while applying its own judgment to candidate comparisons.
  • Generative reward model: The generative reward model predicts per-response leak labels and pairwise preferences from tasks and reference responses, then scores candidate pairs without seeing per-prompt annotations.Its learned signal incorporates cross-prompt norms into model weights rather than supplying full annotations in context.
  • RL training recipe: Both sources train with K = 4 on-policy rollouts and group-relative advantages, plus a length-floor penalty that can subtract up to 4 points from near-empty responses.The floor is 0.5× the average word count of the two reference responses, with penalties scaled linearly below it.

5 Experiments

Experiments show that human annotation context substantially improves privacy-judge reliability and that annotation-conditioned rewards produce stronger privacy-helpfulness alignment than baseline rewards. These gains extend from PrivacyAlign evaluation to external benchmarks, although some comparisons have reproducibility limitations and benchmark-label caveats.

  • Judge reliability: Mean inter-judge κ rises from 0.47 to 0.71 on leaks and from 0.25 to 0.44 on omits when judges receive human annotations.Agreement improves for every frontier-judge pair on 12 held-out PrivacyAlign model runs.
  • Judge reliability: Under representative condition B, mean agreement with gold labels reaches κ=0.54 for leaks and κ=0.37 for omits, versus human-majority κ=0.62 and κ=0.39.Condition B shows annotations for the other reference response, not the scored response; the raw inter-annotator omit κ is 0.120 versus 0.558 for leaks.
  • Privacy evaluation: PrivacyAlign reports leak and omit rates separately, because withholding can improve leakage metrics while harming helpfulness, and disclosure can create the opposite tradeoff.The clean rate counts responses that neither leak nor omit.
  • Prompt-based mitigation: GPT-5.5 achieves a 14.5% leak rate and 70.7% clean rate with privacy-enhanced prompting, while Claude Opus 4.7 has a 10.8% naive omit rate.Even GPT-5.5 leaks in more than one in seven scenarios, and open-weight base models leak in more than half.
  • Reward-model training: Annotation-conditioned rewards outperform CI-RL string matching and trained gen-RMs, winning on clean rate across model families and prompts.For the naive prompt, clean rates are 27.3 versus 24.1 for Qwen3-4B, 28.1 versus 25.7 for Qwen3-8B, and 32.6 versus 21.9 for Nemotron-3-Nano-4B.
  • Generalization: Nemotron-3-Nano-4B reaches a 32.6% clean rate under the naive prompt, near Gemini 3.1 Flash Lite at 35.4% and Gemini 3.1 Pro at 37.3%.The annotation-conditioned models also improve over their base models on PrivacyLens and CIMemories, though those benchmarks use model-generated labels.

6 Conclusion

The conclusion frames agentic privacy as a contextual decision problem whose norms require human involvement. PrivacyAlign operationalizes this principle through human-annotated scenarios, more reliable annotation-conditioned judging, and annotation-conditioned rewards.

  • Conclusion: Agentic privacy depends on human norms and expectations about acceptable disclosures and desired omissions, so humans should remain involved in evaluation and training.The conclusion argues that fully removing humans obscures the norms defining appropriate disclosure, while single LLM judges cannot yet assess them reliably.
  • Conclusion: PrivacyAlign combines privacy-sensitive agent scenarios with pairwise preferences, per-response leak and omit annotations, and free-text rationales from 599 unique annotators.The corpus serves as both a benchmark and training resource.
  • Conclusion: Conditioning LLM judges on same-prompt human annotations sharply reduces inter-judge disagreement and improves alignment with carefully audited gold labels.The annotations provide human-grounded supervision for automated evaluation.

Limitations

PrivacyAlign is limited by synthetic scenarios, imperfect LLM-judge evaluation, heterogeneous human privacy judgments, and training only small open-weight models. Strong benchmark scores also may not transfer to deployment conditions such as distribution shift or adversarial prompts.

  • Synthetic scenarios: Synthetic scenarios are generated by LLMs rather than real user traces, which may limit realism and data quality despite targeting diverse domains, toolkits, and action types.Authentic privacy-sensitive interactions were not collected or shared because doing so could itself violate privacy; frontier models were validated to leak on generated outputs.
  • LLM-judge dependence: LLM-judge evaluation may inherit underlying-model biases and limitations, and annotation conditioning only partially corrects differences because judge agreement remains imperfect.Evaluation continues to rely on LLM judges even after conditioning them on human annotations.
  • Human-annotator heterogeneity: 0.56 inter-annotator κ on leaks and 0.12 on omits reflect heterogeneous privacy intuitions, values, effort, and ability, limiting how precisely labels or rewards match individual preferences.The paper notes that privacy norms may be pluralistic rather than merely noisy, so aggregating preferences into one reward can smooth over meaningful differences.
  • Model scale: Training uses only 4B-to-8B-parameter open-weight checkpoints because of limited computational resources, leaving larger-scale agent training as an important next step.The evaluated checkpoints are Qwen3-4B, Qwen3-8B, and Nemotron-3-Nano-4B.
  • Deployment caution: Strong PrivacyAlign scores do not guarantee perfect privacy handling because agents may fail under distribution shift, adversarial prompts, or scenarios outside the dataset’s coverage.Deployment therefore requires caution beyond benchmark performance.

Ethical Considerations · Appendix · A Dataset Statistics

PrivacyAlign’s ethical design protects annotators and real users while acknowledging limits in cultural representativeness and dual-use risk. Its appendix documents the dataset’s high-level, domain, toolkit, final-action, and generator-model statistics.

  • Ethical Considerations: Annotators were paid £10 per hour, consented to using their judgments and rationales, and provided no personally identifying information.Annotations were collected through Prolific.
  • Ethical Considerations: All 1,350 PrivacyAlign scenarios are synthetic, containing no real users, tool-call traces, or personal data.This enables open corpus release without exposing real users’ private information.
  • Ethical Considerations: 599 annotators from more than 20 countries participated, but English-fluency restrictions limit representativeness and may underweight underrepresented users’ privacy expectations.Privacy norms can differ across cultures and contexts.
  • Ethical Considerations: The authors accept potential adversarial reuse of the public leakage benchmark because they judge shared evaluation’s defensive value to outweigh that risk.They argue leakage is already prevalent and measurable progress requires shared benchmarks, data, and analysis.
  • A Dataset Statistics: The appendix expands the Section 3 dataset summary with high-level statistics and breakdowns by domain, toolkit, final action, and generator model.These breakdowns are reported in Tables 6–10.
  • A Dataset Statistics: The dataset contains 586 unique domain labels, with each scenario carrying 1 to 3 free-form labels and top-domain percentages summing above 100%.The percentages reflect the share of all 1,350 scenarios containing each domain.
  • A Dataset Statistics: Each scenario lists 2 to 6 toolkits, averaging 3.1, drawn from a pool of 48, while 26 unique final-action tools cover the benchmark.Messaging and email sends account for most final actions, including Slack, Outlook, Gmail, and Teams.
  • A Dataset Statistics: Table 10 reports how many scenarios were produced by each generator model.The supplied passage identifies the breakdown but does not provide the model-specific counts.

B Frontier Model Privacy Failures … E Limits of Automated Privacy Judgments

The paper illustrates privacy failures across frontier agents, grounds judgments in human annotations, controls healthcare-domain concentration, and exposes limitations of automated privacy benchmarks. Its approach conditions judges and rewards on human privacy judgments while acknowledging annotator and LLM-judge imperfections.

  • B Frontier Model Privacy Failures: The annotation-conditioned Nemotron-3-Nano-4B + RL model completes the same tasks while withholding the private account, clinical, safehouse, and police-protection details.Figures 3–5 contrast frontier failures with actions that preserve task completion while limiting disclosure.
  • B Frontier Model Privacy Failures: Frontier models leak unnecessary organizational finances, identifiable clinical malnutrition indicators, and physical-safety or legal details in otherwise routine communications.GPT-5.5, Opus 4.7, and Gemini 3.1 Pro each disclose information beyond the recipient’s stated needs or the user’s high-level-only instruction.
  • C Example Human Annotation: Human annotators prefer administrative-only ration logistics over responses that append identifiable clinical screening notes and severe-acute-malnutrition indicators.In the example, all three annotators preferred Response A, which supplied only requested administrative fields.
  • D Healthcare Domain Collapse: All three tested generator models gravitate toward health- and medical-themed scenarios, creating healthcare domain collapse unless the corpus is actively diversified.The paper identifies this concentration in its own pipeline and notes the same pattern in Privasis.
  • D Healthcare Domain Collapse: The pipeline caps each domain at 25% and each domain signature at 5%, then filters undeclared healthcare scenarios using toolkit and keyword checks.The filter checks story, instruction, tool trajectory, or final action for healthcare indicators, including EpicFHIR, Teladoc, HIPAA, mental health, and prenatal.
  • E Limits of Automated Privacy Judgments: Privacy appropriateness depends on the recipient, relationship, social context, and discloser, but current LLMs do not reliably make these judgments independently.Benchmarks that delegate labeling to LLMs therefore substitute model opinions for human judgment.
  • E Limits of Automated Privacy Judgments: GPT-5.5-derived CIMemories labels and PrivacyLens’s Mistral-7B-Instruct-v0.2 judgments can mark legitimate disclosures as sensitive, rewarding refusal of appropriate user requests.PrivacyLens may penalize following a fertility-clinic instruction, while CIMemories produces labels that thoughtful humans could dispute.
  • E Limits of Automated Privacy Judgments: Human conditioning changes the source of privacy judgment but does not eliminate errors from annotator disagreement, incorrect labels, or biases in LLM judges.The method conditions judges on multiple same-prompt human annotations rather than relying on an unaided labeling model.

F Results on External Benchmarks: PrivacyLens and CIMemories

PrivacyAlign models generalize to two held-out agent privacy benchmarks, improving both privacy and task-performance metrics over their base models. These results support generalization, but benchmark labels are model-defined and should not be treated as precise measurements.

  • Benchmark setup: PrivacyLens and CIMemories are held-out agent benchmarks pairing leakage or violation metrics with helpfulness or completeness metrics.PrivacyLens evaluates final tool-use actions, while CIMemories evaluates disclosure of stored memories across contexts.
  • Results: Nemotron-3-Nano-4B improves on all four external-benchmark metrics after annotation-conditioned-reward RL.The models are trained on PrivacyAlign using the annotation-conditioned reward, while the benchmarks remain unseen during training.
  • PrivacyLens: 49.3% to 38.3%: Nemotron-3-Nano-4B’s PrivacyLens leak rate decreases, while helpfulness rises from 1.91 to 2.06.Its 38.3% PrivacyLens leak rate is the lowest reported in the table, tying GPT-5.4-mini.
  • CIMemories: 33.4% to 25.4%: Nemotron-3-Nano-4B’s CIMemories violation rate decreases, while completeness rises from 32.6% to 35.6%.CIMemories labels are generated by GPT-5.5 with low reasoning, and responses are judged by GPT-5.4-mini with high reasoning.
  • Limitations: Both benchmarks use model-defined labels, including self-defined CIMemories notions of which stored items should remain private.Because the benchmarks lack human-reader ground truth and contain documented label-quality issues, the results are interpreted as supporting evidence of generalization rather than precise measurements.

G Critique of CI-RL Training Supervision

CI-RL derives reinforcement-learning rewards from GPT-4-generated lists of allowed and disallowed strings, checking literal substring occurrence in agent outputs. This supervision produces conflicting signals on some rows and remains fragile to short-string collisions and paraphrased disclosures.

  • Reward construction: CI-RL rewards outputs by checking whether GPT-4-generated allowed or disallowed strings literally appear in the agent response.Its synthetic supervision targets are LLM-produced rather than grounded in human judgment.
  • Same-sample contradictions: 5.1% of rows contain conflicting substring-matching reward signals because values overlap or appear on both allowed and disallowed lists.This comprises 28 of 729 rows (3.8%) with identical normalized values and another 9 rows (1.2%) with substring overlap.
  • Fragile matching: 10.3% of disallowed entries normalize to four characters or fewer, making substring matching vulnerable to unrelated-text collisions.Examples include “john”, “202”, “976”, and “1234”.
  • Fragile matching: Paraphrases, indirect references, paraphrased numerals, and restructured sentences can avoid disallowed-string matches without changing the underlying disclosure.Thus, substring membership can miss disclosures even when labels are internally consistent.

H Critique of PrivAct Training Supervision · I Compute and Runtime

The paper critiques PrivAct’s DPO supervision for inheriting PrivacyLens labeling failures and replacing human annotation with LLM judges, while noting that it was evaluated only on non-reasoning models. Training used 8× NVIDIA H100 80GB GPUs, with runs under two days and data generation lasting up to two weeks for the largest model.

  • H Critique of PrivAct Training Supervision: PrivAct trains with DPO on pairwise preferences generated by a multi-agent LLM pipeline over PrivacyLens scenarios.The paper identifies two issues with this supervision.
  • H Critique of PrivAct Training Supervision: PrivacyLens-derived sensitive_info_items frequently mark content that the user’s instruction explicitly asks the agent to share.PrivAct uses sets extracted by Mistral-7B-Instruct-v0.2 to drive its privacy signal.
  • H Critique of PrivAct Training Supervision: PrivAct substitutes LLM judges for human annotators when converting PrivacyLens labels into chosen and rejected pairs.Thus, the DPO target inherits failures from the underlying labeling pipeline.
  • H Critique of PrivAct Training Supervision: PrivAct evaluates DPO only on non-reasoning models, including Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, Mistral-7B-Instruct-v0.2, and Qwen3-4B-Instruct-2507.The training method was not tested on reasoning-model agents.
  • I Compute and Runtime: 8× NVIDIA H100 80GB GPUs supported all training and data generation, and each training run completed in under two days.Data generation used only open-license models.
  • I Compute and Runtime: Up to two weeks were required for data generation with the largest model, Qwen3.5-397B-A17B, using its FP8 variant.The three generator models were gpt-oss-120b, Qwen3.5-397B-A17B, and Nemotron-3-Super-120B-A12B.
  • I Compute and Runtime: Over-generating samples was the bottleneck because samples had to survive scenario-quality judging, sensibility checks, leakage filtering, and diversity caps.The caps were applied per domain and per toolkit.

J Training Hyperparameters … O Baseline Prompts

The appendices specify RL training, annotator recruitment and interface controls, explanation auditing, judge and reward prompts, and baseline agent prompts. Together, these procedures operationalize privacy and completeness judgments through human annotations, quality checks, annotation-conditioned evaluation, and sender–recipient-aware prompting.

  • J Training Hyperparameters: RL training sweeps the learning rate and selects the checkpoint with the highest clean rate on a deterministic ∼10% prompt-level holdout.Clean rate counts rollouts labeled as neither leaking sensitive information nor omitting relevant non-sensitive information.
  • K Annotator Pool: Annotators were English-fluent adults with undergraduate degrees, 99%+ Prolific approval, and at least 200 prior submissions, each completing one six-sample submission.These restrictions were intended to ensure qualified participation and annotation diversity.
  • L Annotation Interface: The interface presents scenario context, two candidate responses, leak/omit labels, preference selection, comparative analysis, and free-text explanations, with minimum-detail requirements before saving.Explanations require at least 12 words, while checked leak or omission labels require descriptions of at least 6 words.
  • M Annotation Explanation Audit Prompt: An auditing prompt checks explanation quality, framework tags, preference consistency, label contradictions, and per-claim grounding, with low-quality annotators manually audited and removed as needed.The audit compares explanations against preferences, leak/omit checkboxes, and the surrounding scenario context.
  • N Judge and Reward Model Prompts: The annotation-conditioned leak/omit judge evaluates new responses using the user instruction, tool trajectory, memories, reference responses, and per-annotator human annotation summaries.Its annotation-free variant removes the reference responses and annotation block, while the same judge supports evaluation and checkpoint selection.
  • N Judge and Reward Model Prompts: Pairwise reward prompts compare faithfulness, privacy leakage, and harmful omission, assigning 0 when responses fail for opposing reasons and reserving ±2 for clear dominance.The annotation-conditioned pairwise judge uses human annotation summaries to calibrate privacy leaks and harmful omissions, whereas the generative reward prompt omits that annotation block.
  • O Baseline Prompts: The baseline agent produces only a JSON final outbound action from trajectory and memory evidence, while the privacy-enhanced variant adds a rule to consider sender/recipient appropriateness before disclosure.The privacy-enhanced prompt is otherwise identical to the naive prompt.
Loading 2606.21710v1…