PaperlayerView exact revision on arXiv

Source-linked AI summary

EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents

Fengzhou Sun, Yuan Zhang, Xintong Yu, Jinyao Yan

arXiv:2609.35233v1cs.AI

TL;DR

Long-term agents acting for users need to respect disclosure boundaries that depend on social relationships and context, but prior work largely leaves long-term relational disclosure unexplored. EP-Mem overlays user-defined policies on token-level memory and introduces EP-Bench for evaluation. On EP-Bench, it reports 94.0% event-level privacy classification accuracy and raises exact disclosure-permission accuracy from 22% to 68%.

  • Problem

    Prior agent-memory privacy work focuses on short-term interactions, isolated tasks, or limited relational settings, leaving long-term disclosure across evolving social relationships underexplored.

    Source: Introduction · p. 1

  • Method

    EP-Mem overlays user-defined relational disclosure policies on token-level memory and uses a pluggable privacy engine across the memory lifecycle.

    Source: Introduction · p. 2, EP-Mem System Design · p. 3

  • Results

    On EP-Bench, EP-Mem achieves 94.0% event-level privacy classification accuracy and raises exact disclosure-permission accuracy from 22% to 68%.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    EP-Mem provides policy-conditioned control of relational disclosure while preserving or improving native retrieval performance on EP-Bench and LoCoMo.

    Source: Introduction · p. 2

    Policies define relationships from the user’s perspective, which may differ from the other party’s; fine-grained rules target a small set of core close relationships.

    Source: Elastic Privacy Memory Unit · p. 2, EP-Mem Overview · p. 2

Abstract

from arXiv · show

Large language model (LLM) agents face critical privacy risks when acting as delegates in human-agent-human communication. To prevent such breaches, agents must understand users' social relationships and adhere to context-dependent social information disclosure boundaries. Current studies on agent memory privacy focus on instantaneous interactions, leaving the long-term relational disclosure problem unexplored. In this paper, we propose EP-Mem, an Elastic Privacy Memory architecture that reframes privacy as user-owned boundary control across social roles. EP-Mem introduces (1) token-level memory driven by user-configurable a privacy policy that stratifies persons and events, combining domain-level default circulation rules with fact-level whitelist/blacklist exceptions; and (2) a pluggable sidecar with a privacy engine that aligns disclosure controls with memory across summary, detail, and boundary granularities, enforced throughout generation, storage, and retrieval. We construct EP-Bench, to our knowledge the first long-term multi-party benchmark with cross-session correlated events for policy-conditioned relational disclosure. Experiments show that EP-Mem achieves 94.0% privacy classification accuracy, improves disclosure-permission judgment from 22% to 68%, and reduces privacy leakage by 75.6%, while maintaining retrieval performance and cross-benchmark generalization.

Introduction

LLM agents acting as delegates in human–agent–human communication risk using information improperly unless they understand social relationships and disclosure boundaries. EP-Mem and EP-Bench address the underexplored problem of long-term, relationship-aware disclosure control.

Introduction

  • Delegating communication to agents can expose users to inaccurate or improper information use, including disclosure of credentials and private chats to unauthorized parties.The paper identifies social-engineering manipulation as one route to such disclosure.Source: Introduction · p. 1
  • Prior privacy-aware memory studies focus mainly on short-term interactions, isolated tasks, or limited relational settings, leaving evolving long-term disclosure boundaries underexplored.Social information disclosure boundaries govern what can be shared, to what extent, and with whom, conditioned on social ties, context, and sensitivity.Source: Introduction · pp. 1–2
  • EP-Mem models privacy as user-owned, dynamic boundary management across social roles and enforces policies through token-level memory and a pluggable privacy engine.Controls span memory generation, storage, and retrieval.Source: Introduction · p. 2
  • 94.0% event-level privacy classification accuracy and exact disclosure-permission accuracy rising from 22% to 68% are reported for EP-Mem on EP-Bench.The paper also reports reduced privacy leakage while preserving response quality and native retrieval performance.Source: Introduction · p. 2
  • EP-Bench is introduced as the first benchmark for policy-conditioned relational disclosure in long-term, multi-party dialogues.It supports cross-session memory evaluation with realistic social interactions, staged tasks, retrieval diagnostics, and reasoning traces.Source: Introduction · p. 2

Related Work

Related work covers long-term memory systems and privacy-oriented memory benchmarks, but persistent personal-memory disclosure in long-term human–agent–human relationships remains insufficiently addressed.

Long-Term Memory Systems

  • Long-term memory research emphasizes cross-session construction, compression, and retrieval, while overlooking privacy disclosure during memory retrieval.Examples include agentic memory evolution, structured representations, and efficient compression.Source: Related Work · p. 2

Memory Benchmarks

  • CPM frames privacy as controllable disclosure, while CI emphasizes appropriate information flow within specific social contexts.These perspectives motivate context-sensitive disclosure rather than preventing information flow altogether.Source: Related Work · p. 2
  • Privacy-oriented benchmarks assess whether and to whom memories should be disclosed, but mainly target short-term tasks or human–agent interactions rather than persistent relational disclosure.The cited examples include CIMemories, StratMem-Bench, AgentSocialBench, and MAGPIE.Source: Related Work · p. 2

EP-Mem Overview

EP-Mem places user-defined relational disclosure policies over long-term memory use. Its memory unit filters facts for a recipient, while its policy structure combines interpersonal and informational rules.

EP-Mem Overview

  • Elastic privacy means the scope of disclosed information depends on social relationships and interaction contexts.EP-Mem therefore needs both a memory unit that retrieves and extracts information according to privacy boundaries and context, and configurable policy files.Source: EP-Mem Overview · p. 2
  • The relationship between sender and recipient is defined from the sender’s perspective, and may differ from the recipient’s view.The framework assumes users configure fine-grained rules for a small set of core close relationships, with settings evolving alongside the social network.Source: Elastic Privacy Memory Unit · p. 2, EP-Mem Overview · p. 2

User Pre-configured Privacy Policy

  • Policy P has interpersonal and informational sides, aligning audience groups with privacy categories across five levels and two domains.Higher interpersonal levels indicate lower trust and a smaller disclosable scope; domains can further partition boundaries within a level.Source: Figure 2 · p. 3, User Pre-configured Privacy Policy · p. 3

Elastic Privacy Memory Unit

  • For a query, the agent uses the sender-recipient relationship and policy to search memory and generate a response.The disclosure decision is jointly determined by the query and the preconfigured policy; the system retrieves relevant memories and selects the disclosable subset.Source: Elastic Privacy Memory Unit · p. 3

User Pre-configured Privacy Policy

  • Informational categories map to privacy levels, while user-configurable whitelist and blacklist rules grant or revoke access for specific memory units.Whitelist and blacklist exceptions are evaluated per memory unit during memory construction and retrieval.Source: User Pre-configured Privacy Policy · p. 3

EP-Mem System Design

EP-Mem incrementally overlays user-defined disclosure policies on a native memory system, controlling event-level facts while preserving native retrieval and compression. Its privacy engine governs memory processing and retrieval, then produces audience-safe context.

EP-Mem System Design

  • EP-Mem is an incremental plug-in framework that overlays user-defined relational disclosure policies on native memory while preserving core functions such as retrieval and compression.It adds targeted enhancements across five stages of the memory pipeline.Source: EP-Mem System Design · p. 3

Memory Material Preprocessing

  • Heterogeneous sources are standardized as sessions, with group conversations decomposed into two-party sub-sessions and labels added to metadata.Source: Memory Material Preprocessing · p. 3

Memory Unit Construction and Reprocessing

  • EP-Mem applies disclosure control to F2 event-level facts, while requiring native F3 summaries and F5 metadata and leaving other levels unchanged.A session may contain multiple events with different disclosure levels, and the privacy engine annotates facts with policy levels and whitelist/blacklist exceptions.Source: Memory Unit Construction and… · pp. 3–4

Memory Storage

  • The separate privacy layer retains the native memory-store format and links to the native system through a shared F3 index.When F0 atomic entries are available, EP-Mem additionally builds an F2-to-F0 index.Source: Memory Storage · p. 4

Memory Retrieval

  • Retrieval combines the native retrieval score and event-level proposition score through a weighted sum without modifying the original retriever.The described score is s = α snat + β sevt, with α + β = 1 and α, β ≥ 0.Source: Memory Retrieval · p. 4

Context Management via EP-Mem Privacy Engine

  • The privacy engine annotates facts, adjudicates disclosure permissions across facts and query units, and turns the results into audience-safe storylines and responses.For coreferent or fragmented mentions, whitelist grants are bound to query units without extending the blacklist; key-stage prompt updates append supplementary prompts without changing fixed judgment or desensitization rules.Source: Context Management via EP-Mem Privacy… · pp. 4–5

EP-Bench Dataset

EP-Bench evaluates policy-conditioned relational disclosure in long-term memory using cross-session social scenarios and a three-task difficulty ladder. Its tasks assess privacy classification, person-level disclosure judgments, and audience-differentiated answers.

Overview

  • EP-Bench is built for policy-conditioned relational disclosure in long-term memory and evaluates both privacy-aware disclosure and memory capabilities.Its construction pipeline covers policy design, storyline creation, QA authoring, evidence generation, and quality control.Source: Overview · p. 5
  • The benchmark contains 10 first-person scripts spanning about 7–18 months, with cross-session sensitive events, varying states and cover stories, and queries often requiring temporal, multi-hop, and disclosability reasoning.The scripts include social and independent character settings and diverse interaction types.Source: Overview · p. 5

Tasks and Metrics

  • EP-Bench evaluates event-level privacy classification, person-level disclosability judgment, and audience-differentiated answering with privacy, response-quality, differentiation, and retrieval metrics.The tasks increase in difficulty; retrieval uses Recall@10 over evidence sessions.Source: Tasks and Metrics · pp. 5–6

Experiments

Experiments evaluate disclosure control, memory retrieval, and transfer across EP-Bench, CIMemories, and LoCoMo. EP-Mem improves privacy judgments and leakage measures while largely preserving retrieval performance.

Experimental Setup

  • The evaluation uses EP-Bench, CIMemories, and LoCoMo, comparing prompting baselines, full systems, and component ablations; supplementary tests report similar trends across models.The main experiments use DeepSeek-V4-Flash; additional generators include DeepSeek-V4-Pro, DeepSeek-R1-32B, and Qwen3.6-27B-FP8.Source: Experimental Setup · p. 6, Results and Analysis · p. 7

Results and Analysis

  • 94.0% overall accuracy on EP-Bench event privacy-level judgment (94/100).Per-script accuracy ranged from 7/10 to 10/10.Source: Results and Analysis · p. 6
  • On Task 3, EP-Mem lowers PB from 0.738 with NoEngine_evid to 0.180; EP-Mem+Evo reaches 0.136.RoleHist reaches PB 0.477, while removing desensitization raises EP-Mem’s PB to 0.493.Source: Results and Analysis · p. 6, Table 1 · p. 7
  • On EP-Bench’s original set, EP-Mem raises Partial/Exact permission accuracy from 23%/22% to 70%/68%.EP-Mem+Evo reaches 83%/82% Partial/Exact; ExactT2E rises from 78% to 80%.Source: Results and Analysis · p. 6, Table 2 · p. 7
  • EP-Mem retrieval overlays match native retrieval with zero fusion weights and maintain or improve performance after tuning; under added noise, permission scores fall only mildly.With 2–3 unrelated same-script sessions, EP-Mem Partial/Exact scores move from 70%/68% to 66%/64.8%.Source: Results and Analysis · p. 7, Table 4 · p. 7
  • On CIMemories, EP-Mem achieves 46.5% Violation and 59.2% Completeness, the best completeness and most balanced privacy–utility trade-off reported.Filter-only has fewer violations but 35.7% Completeness.Source: Table 5 · p. 7, Results and Analysis · p. 7

Conclusion

The paper studies relationship-aware privacy disclosure for long-term LLM agents in human–agent–human interactions. It presents EP-Mem and EP-Bench, reporting effective, robust, and evolvable privacy control.

Conclusion

  • EP-Mem enforces user-defined disclosure policies through token-level memory and a pluggable privacy engine across the memory lifecycle.The architecture is grounded in CPM and CI.Source: Conclusion · p. 7
  • EP-Bench is introduced as the first benchmark for policy-conditioned relational disclosure in long-term multi-party interactions.Source: Conclusion · p. 7
  • Experiments demonstrate effective, robust, and evolvable privacy control with EP-Mem.Source: Conclusion · p. 7

Section A: Experimental Details

The appendix describes EP-Mem as a sidecar overlay on native memory systems, detailing attachment and storage choices alongside retrieval fusion and weight tuning.

Section A: Experimental Details

  • The appendix supplements retrieval, noise-robustness, and cross-model evaluations with additional experimental details.Per-script details for the Flash main experiments are also included.Source: Section A: Experimental Details · p. 9

A.1 EP-Mem Overlay on Native Memory Systems

  • EP-Mem attaches as a sidecar while leaving the host store and retriever unchanged, overlaying both MemGAS and SimpleMem.F2 attaches through the shared F3 index, with an F2-to-F0 index when F0 is available.Source: §A.1 · p. 9

Memory Storage.

  • For EP-Bench, SimpleMem uses one compression window per session, mapping each F0 entry to a single session; MemGAS already stores one record per session.SimpleMem places F2 and other sidecar layers under the shared F3 index; sidecar attachment may run asynchronously after ingest.Source: Memory Storage · p. 9

Memory Retrieval.

  • Retrieval combines native and event-fact scores as s(u) = α snat(u) + β sevt(u), with α + β = 1; F2 off recovers native ranking.MemGAS injects F2 as a fifth PPR channel; SimpleMem links each event fact to its top three same-session F0 entries.Source: Memory Retrieval · p. 9
  • On LoCoMo, no positive MemGAS fact weight improves all cutoffs; the selected setting slightly raises R@10 while permitting small drops elsewhere.For SimpleMem, small event-fact weights (≤0.15) stay at or above baseline at every cutoff, while larger weights (≥0.60) help mid/late ranks but hurt R@1.Source: Memory Retrieval · p. 9, Table S3 · p. 9, Table S1 · p. 9

A.2 Noise Robustness

Noise robustness is tested by adding unrelated same-script sessions to Task 2 items. Scores decline mildly, with most noise filtered during final disclosure adjudication.

A.2 Noise Robustness

  • Because native retrieval cannot yet stably recall EP-Bench evidence, the main disclosability results use gold evidence.Source: §A.2 · p. 10
  • The noise test adds 2–3 same-script sessions unrelated to the evidence to each Task 2 item; noisy scores average five runs at temperature 0.No-noise scores reuse Task 2 original-set Partial/Exact results.Source: §A.2 · p. 10
  • Unrelated sessions interfere only mildly, and most noise is filtered at final disclosure adjudication.The evolution path drops by a similar margin; remaining Partial-score drops are largely attributed to LLM jitter under reruns.Source: §A.2 · p. 10

A.3 Cross-Model Evaluation

Across generators, EP-Mem maintains strong person-level disclosability judgments, while EP-Mem+Evo performs best within every reported Task 2 block; Task 3 results favor EP-Mem+Evo on several metrics. The section also formalizes how policy levels, exceptions, multi-fact ceilings, and event coreference determine disclosure.

A.3 Cross-Model Evaluation

  • Across generators, EP-Mem keeps Task 2 Partial accuracy near 65%–70%, and EP-Mem+Evo is best within every block.Strong reasoners using full-text NoEngine can approach Flash EP-Mem results by inferring who already knows, but EP-Mem matches or exceeds full-text NoEngine with partial fact-level inputs and formal disclosability steps.Source: §A.3 · p. 10, Table S6 · p. 10
  • EP-Mem+Evo attains the best PB / PC / PD in the Flash and R1 blocks, and the best or second-best privacy in every Task 3 block.On Pro, RoleHist can look strong on PB / PD by restricting history to the target audience, but this does not replace disclosure-permission judgment and desensitizing projection.Source: §A.3 · p. 10, Table S7 · p. 11

A.4 Per-Script Main Results

  • The per-script tables report Flash results for Task 2 and Task 3, with evolution applied only to selected scripts and averages computed across all ten scripts.The averages are unweighted; scripts without evolution contribute EP-Mem scores to the EP-Mem+Evo average.Source: §A.4 · p. 10, Table S8 · p. 11, Table S9 · p. 11

B.1 Formulation of disclosure-permission judgment

  • The disclosure calculus allows a memory for receiver R only when its fact-level policy judgment permits disclosure, with blacklists overriding whitelists and default audiences.For composite fact sets, a privacy ceiling tightens the shared default gate and every fact must allow disclosure; event coreference can transfer only trusted real-truth whitelist grants to query units.Source: §B.1 · pp. 12–13
  • Privacy levels encode nested default audiences, while domain-marked levels in different domains have no audience inclusion and can yield the restrictive L0 ceiling when conjoined.Ordinary facts use their level's default audience; penetrating facts combine labels, and unauthorized categories remain at L0 until a matching category is defined.Source: §B.1 · p. 12

B.2 Metric definitions

This appendix section defines EP-Bench’s metrics and notation and describes prompts and templates for memory construction, privacy decisions, context-safe responses, and evaluation. It also covers deduplicated, audience-specific content extraction without privacy judgment and a case study evaluating policy-conditioned disclosure judgments, audience-differentiated answers, and permeability based on whether non-disclosable facts are written.

B.2 Metric definitions

  • Permeability metrics classify retrieved facts as violations by disclosure permission, while CDiff measures listener-verifiable units told to exactly one audience.CDiff merges semantic equivalents, averages nonzero scores across three extraction runs, and is zero when all runs score zero; audited scores evaluate agreement with the gold analysis’s implied allow set.Source: §B.2 · p. 13

B.3 Notation summary

  • Table S10 summarizes notation for effective unit whitelists and blacklists, query-unit disclosure verdicts, and fully-disclosable status.The table includes CoRef-adjusted effective whitelist and blacklist symbols and a Boolean for full disclosure across units.Source: §B.3 · p. 14, Table S10 · p. 14

Section C: Prompt templates

  • The prompt-template section covers memory preprocessing, privacy annotation and control, key-stage evolution, and EP-Bench evaluation, with English translations of core instructions and hard constraints.Experiments use Chinese-language templates; few-shots and lengthy output schemas are omitted.Source: Section C: Prompt templates · p. 15

C.1 Memory material preprocessing and unit construction

  • Memory construction normalizes chats into Sessions, segments group chats using reply links, and distills each Session into event-level facts before privacy fields are added.The extraction instructions require third-person event summaries based only on the Session and exclude assigning privacy categories, levels, or whitelist-blacklist fields at that stage.Source: §C.1 · p. 15

C.2 Privacy engine Privacy annotation.

  • Disclosure decisions use fact-specific deny rules, while context projection retains or rewrites permitted facts and omits or blurs non-disclosable content.The fixed order denies blacklisted facts, then applies whitelist requirements to L0 facts, cross-domain facts, and deeper same-domain facts; otherwise it allows disclosure. The response uses the disclosable storyline as framework and permitted facts for details, or politely deflects if none are allowed.Source: §C.2 · pp. 16–17

C.3 Evaluation templates

  • Evaluation templates define used facts through attributable, non-inferable details covering a fact’s core predicate and critical object, and score style, integration quality, privacy, and sufficiency.MIQ uses the worst integration failure; the overall LLM-judge score is the minimum of privacy and sufficiency.Source: §C.3 · p. 18
  • Evaluation templates extract concrete claims, attitudes, or details from replies to the same question core across receivers, excluding fillers and deduplicating paraphrases.They mark which role keys express each unit, do not invent absent content, and do not judge privacy or use fact identifiers.Source: §C.3 · p. 18

Section D: Case study

  • Case-study comparisons cover person-level disclosability judgment in Task 2 and audience-differentiated answering in Task 3, evaluated against the user’s preconfigured privacy policy.Task 2 compares NoEngine_full, NoEngine_evid, w/o Disc, and EP-Mem; Task 3 also includes RoleHist and w/o Desens., plus EP-Mem+Evo only in the coffee case. Unused non-disclosable facts count as no permeability, while writing one counts as permeability.Source: Section D: Case study · p. 19

D.1 Person-level disclosability judgment (Task 2)

The cases show that disclosure depends on policy permission rather than narrative proximity or who appears to know a fact. EP-Mem applies whitelist exceptions and same-domain defaults where the baselines either miss permitted recipients or select recipients without authorization.

Case 1: Same-level same-domain allowed; cross-domain blocked

  • EP-Mem discloses the student-wellbeing composite fact to colleagues Zhao Lei and Li Min, who are permitted by the work-domain default.The life-domain contacts Zheng Tao and Zhao Zhiwei require whitelist authorization, which the item does not grant.Source: §D.1 · p. 19
  • Baselines miss permitted recipients in the colleague case, showing that longer session text alone does not recover level–domain default circulation.Without disclosure-permission judgment, same-level same-domain circulation is not enforced.Source: §D.1 · p. 19
  • EP-Mem selects Xiaolin, the only candidate whitelisted on both dimensions of the composite income-and-cover-story fact.NoEngine_full selects Lao Zhang and Su’s father, while NoEngine_evid and w/o Disc select none; the case attributes the gain to explicit disclosure-permission judgment.Source: §D.1 · p. 19

D.2 Audience-differentiated answering (Task 3) Case 3: Year-end gift box (no evolution).

Case 3 tests whether answers about the year-end gift box vary appropriately by audience under the same privacy policy. EP-Mem avoids permeability for all three audiences, but its client-facing answer is less informative than the gold answer, and the “woman in red” cue exposes a perspective gap.

Case 3: Year-end gift box (no evolution).

  • For girlfriend Cheng Wantang, EP-Mem gives a general company-gift explanation, while w/o Desens. discloses the important-client context and the manager’s role.The “woman in red” reference also shifts across systems: it aligns to Tang Xiao under gold and EP-Mem, to the girlfriend under NoEngine_evid, and to a classmate under RoleHist.Source: §D.2 · pp. 19–20
  • For the client, the gold answer identifies a Longines watch prepared for him, while EP-Mem gives a more guarded, under-informative account.NoEngine_evid reveals additional details about the purchase and preparation; RoleHist instead describes a different gift.Source: §D.2 · p. 20
  • For admin Wang Sisi, the gold answer refers to the box she handled without naming the watch or client, while RoleHist describes an intern and thermos instead.EP-Mem keeps the account general, whereas w/o Desens. says Sisi helped make and wrap the box.Source: §D.2 · pp. 19–20
  • EP-Mem keeps no permeability for all three audiences; NoEngine_evid overshares, RoleHist invents, and the client-facing EP-Mem answer still needs sufficiency as well as secrecy.The “woman in red” cue maps differently across systems, while EP-Mem does not identify the Longines gift for the client.Source: §D.2 · p. 20

Related papers

Loading 2609.35233v1…