PaperlayerView exact revision on arXiv

Source-linked AI summary

PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Training

Yuhan Cheng, Hancheng Ye, Hai Helen Li, Jingwei Sun, Yiran Chen

arXiv:2602.13840v2cs.CL

TL;DR

Personalized LLM agents handle sensitive information, but contextual privacy depends on implicit situational norms and can be violated through unintended disclosure. PrivAct internalizes privacy preferences through multi-agent training and reports reduced leakage with similar helpfulness and generalization across models and topologies.

  • Problem

    Personalized LLM agents handle sensitive data, and unintended disclosure can violate contextual expectations; prior work has underexplored learning to act in a contextually privacy-compliant manner.

    Source: §1 · pp. 1–2

  • Method

    PrivAct trains multi-agent systems on contextual privacy preferences, propagating rewards upstream and using leakage-conditioned asymmetric reward shaping.

    Source: §1 · p. 2

  • Results

    PrivAct reports a 12.32% reduction in leakage rate while maintaining similar helpfulness, with transfer to probing tasks and generalization across diverse multi-agent topologies.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PrivAct demonstrates internalized contextual privacy preservation for agentic actions without external, inference-time enforcement.

    Source: §5 · p. 8

    The fine-tuning requirement limits direct applicability to closed-source models.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly deployed in personalized tasks involving sensitive, context-dependent information, where privacy violations may arise in agents' action due to the implicitness of contextual privacy. Existing approaches rely on external, inference-time interventions which are brittle, scenario-specific, and may expand the privacy attack surface. We propose PrivAct, a contextual privacy-aware multi-agent learning framework that internalizes contextual privacy preservation directly into models' generation behavior for privacy-compliant agentic actions. By embedding privacy preferences into each agent, PrivAct enhances system-wide contextual integrity while achieving a more favorable privacy-helpfulness tradeoff. Experiments across multiple LLM backbones and benchmarks demonstrate consistent improvements in contextual privacy preservation, reducing leakage rates by up to 12.32% while maintaining comparable helpfulness, as well as zero-shot generalization and robustness across diverse multi-agent topologies. Code is available at https://github.com/chengyh23/PrivAct.

1. Introduction

Personalized LLM-agent tasks use sensitive information, and contextual privacy violations can occur when agents disclose it contrary to situational expectations. PrivAct trains agents to internalize contextual privacy preferences, reporting reduced leakage while maintaining similar helpfulness.

1. Introduction

  • Contextual privacy depends on subtle norms governing who shares information with whom and under what conditions, so even innocuous queries can leak information.Personalized agent tasks may involve sensitive user data and information about other people.Source: §1 · p. 1
  • Existing prompt-based and agent-based defenses intervene at inference time, often requiring scenario-specific control and exposing sensitive intermediate reasoning.Their performance can depend on task formulations and agent configurations, while reasoning traces may themselves be unsafe.Source: §1 · p. 2
  • The paper targets a gap in learning agents to act in accordance with contextual integrity, beyond prior work on explicit privacy, red-teaming, or regulation understanding.The authors describe their work as the first, to their knowledge, to internalize contextual privacy-awareness in multi-agent systems for privacy-compliant actions.Source: §1 · p. 2
  • PrivAct combines a multi-agent contextual privacy preference dataset with distributed training and leakage-conditioned asymmetric reward shaping.Final rewards are propagated to upstream agents through credit reassignment to encourage system-wide privacy-preserving outcomes.Source: §1 · p. 2
  • 12.32% reduction in leakage rate while maintaining similar helpfulness, with transfer to probing tasks and generalization across multi-agent topologies.The introduction also reports transferability to probing tasks and generalizability across diverse topologies.Source: §1 · p. 2

2. Related Work

Prior contextual privacy defenses include prompt-based methods, agent-based information-flow controls, and learning-based approaches. The paper distinguishes its focus as internalizing contextual privacy in language-model generation for privacy-compliant actions.

2. Related Work

  • Existing contextual privacy work includes prompt-based defenses, agent-based information-flow control, and learning-based methods.The cited learning-based work includes privacy understanding and red-teaming, rather than the target of internalized contextual privacy-preserving action.Source: §2 · pp. 2–3
  • Prompt-based and agent-based approaches enforce privacy through external, inference-time interventions rather than internalizing it in model generation.The paper contrasts these approaches with learning privacy-preserving behavior for agentic actions.Source: §2 · pp. 2–3
  • Multi-agent systems have been fine-tuned using supervised fine-tuning, direct preference optimization, PPO, and GRPO.Examples include SiriuS, Multiagent Finetuning, MALT, MAPoRL, and AgentFlow.Source: §2 · p. 3

3. Internalizing Contextual Privacy Preservation with Multi-Agent Training

The method trains each agent to preserve contextual privacy while fulfilling user instructions, using reward propagation for per-agent preference supervision and leakage-conditioned reward shaping. Its training design treats privacy preservation as a prerequisite for positive helpfulness reward.

3. Internalizing Contextual Privacy Preservation with Multi-Agent Training

  • Per-agent preference supervision addresses feedback available only to the final agent and the risk that privacy emphasis makes responses overly cautious.The method reassigns credit to intermediate agents and uses leakage-conditioned asymmetric reward shaping to discourage trading privacy for utility.Source: §3 · p. 3

3.2. Multi-agent Preference Construction

  • Preference supervision is constructed for every agent by reassigning final-output feedback to intermediate generation stages.This addresses the constraint that feedback is naturally available only to the agent producing the final output.Source: §3 · p. 3

3.1. Problem Formulation

  • Responses must fulfill the user instruction while avoiding disclosure of sensitive information under the given context.The objective applies to a multi-agent system that collaboratively generates candidate responses.Source: §3.1 · p. 3

3.2.1. TREE-STRUCTURED MULTI-AGENT GENERATION

  • Agents generate candidate responses at each stage, forming a tree whose root-to-leaf paths are complete trajectories and whose leaves are final reactions.The final reactions are evaluated with a task-specific reward that captures helpfulness and privacy adherence.Source: §3.2.1 · p. 3

3.2.2. REWARD PROPAGATION AND PREFERENCE CONSTRUCTION

  • Final-response rewards are averaged upward over descendants, and propagated values support per-agent preference pairs for downstream direct preference optimization.The propagated value estimates an intermediate response’s expected downstream utility.Source: §3.2.2 · pp. 3–4

3.3. Leakage-Conditioned Asymmetric Reward Shaping

  • LC-ARS penalizes responses with any privacy leakage and activates positive reward only when privacy is fully preserved, while shaping helpfulness in the non-leaking regime.In the leaking regime, both leakage severity and helpfulness are penalized; the leakage penalty is lower-bounded to support stable training.Source: §3.3 · p. 4

4. Experiments

The experiments test PrivAct’s privacy–helpfulness balance, transfer, component contributions, topology robustness, and contextual behavior. Results report improved tradeoffs and gains across benchmarks and multi-agent configurations.

4.1. Experimental Setup

  • Four backbones were evaluated, and models trained exclusively on PrivacyLens were tested on both PrivacyLens and ConfAIde while PrivAct varied hyperparameters to map tradeoffs.Source: §4.1 · pp. 4–5

4.2. Main Results on PrivacyLens

  • Across four backbones, PrivAct improves the privacy–helpfulness frontier; with Mistral-7B-Instruct-v0.2, AIFC’s 0.63% helpfulness gain accompanies 12.32% higher average privacy leakage.AIFC versus PrivAct: average helpfulness is 86.93% versus 86.30%, while average privacy leakage is 23.7% versus 11.4%.Source: Figure 2 · p. 5, §4.2 · p. 6
  • PrivacyLens evaluation pairs average and worst-case privacy measures with average and binary helpfulness, with leak@K flagging any violation among K sampled responses.Source: §4.2 · pp. 5–6

4.3. Transferability to ConfAIde

  • Without ConfAIde fine-tuning, both refiner- and verifier-trained models outperform the base model across evaluated backbones; action training also improves probing-task performance.Source: §4.3 · pp. 6–7

4.4. Ablation Study: Multi-Agent Components

  • Across evaluated backbones, fine-tuning both verifier and refiner achieves a Pareto-optimal frontier that neither component reaches alone.Source: Figure 3 · p. 7, §4.4 · p. 7

4.6. Case Study: Acting with Contextual Integrity

  • In the clergy–congregant case, PrivAct omits Sarah’s confidential situation while still providing relevant community updates, unlike the leaking base model.Source: Figure 4 · p. 8, §4.6 · p. 8

4.5. Robustness Across Multi-Agent Topologies

  • PrivAct outperforms baselines in privacy preservation and helpfulness across evaluated multi-agent topologies, indicating its gains are not tied to one structure.Source: Table 3 · p. 7, §4.5 · p. 8

5. Conclusion

PrivAct internalizes contextual privacy preservation in multi-agent language models through preference learning and leakage-conditioned reward shaping, with reported improvements in privacy–helpfulness tradeoffs and generalization across models and topologies.

5. Conclusion

  • PrivAct internalizes contextual privacy preservation in multi-agent language model systems.It uses multi-agent preference learning with leakage-conditioned reward shaping to enable privacy-compliant actions without external, inference-time enforcement.Source: §5 · p. 8
  • Empirical results report improved privacy–helpfulness tradeoffs and robust generalization across models and agent topologies.Source: §5 · p. 8
  • Leakage-conditioned reward shaping is the stated mechanism for learning privacy-compliant actions without inference-time enforcement.Source: §5 · p. 8

Impact Statement

The impact statement situates PrivAct in socially aware language modeling and notes that fine-tuning limits direct applicability to closed-source models; the appendices cover privacy research, multi-agent prompts, and a prompt-based baseline.

Impact Statement

  • Fine-tuning limits PrivAct’s direct applicability to closed-source models, and responsible deployment alongside complementary safeguards remains necessary.Source: Impact Statement · p. 9

A.1. Privacy in LLMs

  • LLM privacy work covers short-term and long-term memory leakage as well as model-level risks, with short-term privacy including contextual sensitivity.Source: §A.1 · p. 12

A.2. Contextual Privacy Internalization

  • Prior work addresses explicit privacy or contextual reasoning, but the authors distinguish reasoning about norms from acting in a privacy-preserving manner.Source: §A.2 · p. 12

A.3. Multi-agent Fine-tuning

  • Prior multi-agent fine-tuning methods target general reasoning benchmarks, while the cited work does not explicitly study privacy-compliant behavior under contextual constraints.Source: §A.3 · p. 12

B.1. Multi-agent System Prompt

PROMPTS FOR THE MULTI-AGENT SYSTEM

  • The Generator drafts an output, the Verifier critiques privacy and instruction adherence, and the Refiner incorporates that critique into the final output.The prompts also supply user information, task description, tool specifications, prior trajectories, and the next action.Source: PROMPTS FOR THE MULTI-AGENT SYSTEM · p. 13

B.2. Prompt Settings of Baselines

  • The baseline appendix provides the system-information prompt used to implement prompt-based privacy enhancement following Shao et al. (2024).Source: §B.2 · p. 14

PROMPT-BASED PRIVACY ENFORCEMENT

  • The 1-2-3 Check baseline separates event extraction, privacy classification, and response generation into distinct agents.Its extractor identifies contextual elements, its checker classifies them as private or public, and its executor generates a response using permissible information.Source: PROMPT-BASED PRIVACY ENFORCEMENT · p. 14

C.1. Training and Hyperparameter Settings

Training uses specified reward-shaping and preference-pair parameters, model-specific epoch counts, and datasets collected under varying reward settings.

C.1. Training and Hyperparameter Settings

  • The training setup uses privacy exponent α = 0.5, helpfulness exponent β = 2.0, thresholds τ2 = 0.5 and τ3 = 0.5, and branching factors B1 = B2 = B3 = 4.Source: §C.1 · p. 15
  • Verifiers were trained for 12 epochs and generators for 4 epochs at learning rate η = 5 × 10−5.The global batch size was 32, implemented with batch size 8 and gradient accumulation of 4; hyperparameters and configurations are shared unless specified otherwise.Source: §C.1 · p. 15
  • Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and Qwen3-4B-Instruct-2507 used data collected with Llama-3.1-8B-Instruct, while Mistral-7B-Instruct-v0.2 used data collected with itself.Source: §C.1 · p. 15
  • Preference datasets vary b1 and b2 settings to produce operating points on the Pareto frontier, with dataset sizes reported in Table 4.The table covers leakage-conditioned asymmetric reward shaping (LC-ARS) using responses generated by Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.2.Source: §C.1 · p. 15, Table 4 · p. 15

C.2. Datasets and Metrics

The evaluation reports a GPT-4o-judge comparison favoring PrivAct’s privacy-helpfulness tradeoff, and describes the datasets, metrics, evaluation prompts, and judge checks used across the appendices.

C.2. Datasets and Metrics

  • PrivacyLens has 493 cases, with 394 used for training and 99 held out; helpfulness uses a four-level ordinal scale from Poor (0) to Excellent (3).The paper reports average and binary helpfulness metrics.Source: §C.2 · p. 16

LEAKAGE EVALUATION PROMPT

  • The leakage prompt asks whether an action reveals all or part of a secret, while the helpfulness rubric scores fulfillment of the user instruction from Poor (0) to Excellent (3).The supplied leakage example judges paraphrased disclosure as leakage even when the twins’ names are omitted.Source: LEAKAGE EVALUATION PROMPT · p. 16, HELPFULNESS EVALUATION PROMPT · p. 17

C.2. Datasets and Metrics

  • ConfAIde Tier 3 tests disclosure decisions involving a data subject, confidant, and third party, while Tier 4 tests sharing public information without private pre-meeting discussion.Tier 3 reports FR-E, IA-E, and PS-E measures; Tier 4 uses meeting summary error and action item error.Source: HELPFULNESS EVALUATION PROMPT · p. 17

D.1. Numerical values for Figure 2

D.2. Additional ablation results to Figure 3

D.3. LLM-as-a-judge reliability & consistency

  • For leakage detection, the LLM judge achieved Fleiss’ Kappa 0.82 and accuracy 0.92 against human labels; helpfulness agreement was Fleiss’ Kappa 0.56.The comparison used four annotators on 153 action–sensitive-information pairs for leakage and 50 actions for helpfulness.Source: §D.3 · p. 19
  • 20.20% higher worst-case privacy leakage rate for AIFC (54.545% vs. 34.343%) accompanies its 0.9% binary-helpfulness gain (91.6% vs. 90.7%) over PrivAct.The results are described as consistent with those obtained using Mistral-7B-Instruct-v0.2 as judge; PrivAct has the lowest leakage rate and competitive helpfulness.Source: §D.3 · p. 19, Table 6 · p. 19

D.4. Hyperparam Sensitivity

The α and β sensitivity results show that PrivAct’s gains are robust across most tested configurations, while α, β, and b2 shape the privacy–helpfulness tradeoff.

D.4. Hyperparam Sensitivity

  • All tested α/β configurations except (α = 0.5, β = 3.0) strictly dominate Vanilla LM and PPE on privacy leak rate and helpfulness.The reported example trades 9.6% higher helpfulness for 22.22% higher worst-case leakage, and the authors state the gains are not tied to one hyperparameter choice.Source: §D.4 · pp. 19–20, Table 7 · p. 19
  • With β = 2.0, increasing α from 0.3 to 0.5 to 1.0 raises leak@K from 31.3 to 32.3 to 35.4.The authors attribute the pattern to α < 1 disproportionately penalizing small leakage.Source: Table 7 · p. 19, §D.4 · p. 20
  • With α = 0.5, increasing β from 1.5 to 2.0 to 3.0 raises average helpfulness from 0.713 to 0.737 to 0.788.This accords with LC-ARS assigning greater emphasis to helpfulness at larger β.Source: Table 7 · p. 19, §D.4 · p. 20
  • Increasing b2 shifts the model from a helpfulness-oriented to a privacy-oriented regime by changing the base reward for non-leaking behaviors.The sensitivity results are summarized in Table 8.Source: §D.4 · p. 20, Table 8 · p. 20

D.5. Results on Compositional Privacy Risks Dataset

On a compositional privacy-risk dataset, PrivAct is evaluated against baseline Chain-of-Thought defenses using Overall Success, which requires benign-query success alongside blocking the paired sensitive query.

D.5. Results on Compositional Privacy Risks Dataset

  • Across b2 settings, PrivAct consistently outperforms baseline Chain-of-Thought defenses in Overall Success for both verifier and refiner models.The evaluation uses 119 scenarios, each tested with adversarial and benign inferences; the defender is Llama-3.1-8B-Instruct and the attacker is Qwen3-32B.Source: §D.5 · p. 20, Table 9 · p. 20

E. Limitations

The paper identifies deployment on resource-constrained environments, such as edge devices, as a challenge for privacy-aware LLM agents.

E. Limitations

  • Resource-constrained deployment, including edge devices, remains a challenge for privacy-aware LLM agents.The authors identify computational and memory efficiency as priorities for future work.Source: E. Limitations · p. 20
Loading 2602.13840v2…