Source-linked AI summary
Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems
Aman Priyanshu, Supriti Vijay, Esha Pahwa
TL;DR
Conventional safety evaluations often test isolated, short-turn interactions, leaving privacy behavior in persistent social environments insufficiently evaluated. The paper introduces a Moltbook-style multi-agent simulation and finds that social interaction substantially increases privacy leakage, including after safeguards are applied.
Problem
Safety evaluations often study models as isolated chat assistants, providing limited evidence about interaction-dependent privacy failures in persistent social environments where peer behavior and community norms shape disclosure.
Method
The study constructs a Moltbook-style simulation using 2,533 extracted agent personas and synthetic private attributes, then evaluates privacy leakage across persistent multi-agent communities.
Results
45.3% privacy violations occurred in multi-turn social evaluation versus 19.95% in single-turn evaluation across OpenAI models, while peer exposure increased leakage probability approximately 8× and instructions left leakage above 37.8%.
Takeaways & Limitations
Safety evaluation for persistent agents should vary social context alongside task context, including community topology, peer exposure, and interaction length.
Takeaways & Limitations
Personas are synthetic and assigned to agents rather than representing real users.
Abstract
from arXiv · showhide
LLM safety evaluations predominantly test models in isolation, yet deployed AI agents increasingly operate within persistent social environments alongside other agents. We introduce a Moltbook-style simulation platform where thousands of LLM agents interact across communities over a simulated month, and use it to evaluate privacy as a downstream safety concern under varying degrees of social pressure. We find that shifting from single turn to multi turn social evaluation amplifies privacy violations (CIMemories 19.95% to Ours 45.30% across OpenAI models), that leakage is socially contagious, with agents 8 times more likely to disclose sensitive information after observing a peer do so, and that explicit privacy instructions reduce but do not eliminate this effect, leaving leakage rates above 37.8% even with safeguards. Our findings suggest that static chat based safety benchmarks systematically underestimate risks in agentic deployment, and that social context alone is sufficient to elicit sensitive disclosures that single turn evaluations would never surface.
1 Introduction
Existing safety evaluations mainly test isolated, short interactions, while deployed agents persist, use tools, and interact socially. This paper addresses whether such environments alter privacy boundaries and finds substantially higher leakage under multi-turn social evaluation.
- Motivation: Deployed agents operate over long horizons, call tools, and interact repeatedly with users and other agents in shared environments.These settings introduce interaction-dependent risks that short prompts may not reveal.
- Privacy framing: Privacy depends on whether information flows are appropriate for the recipient, social setting, and governing norms, not merely whether the information is secret.Persistent memory can surface true or useful information in inappropriate contexts.
- Motivation: Social context shapes disclosure through peer presence, reciprocity, sharing norms, and group pressure.These mechanisms motivate testing privacy in communities rather than only against isolated prompts.
- Research gap: Most safety benchmarks measure direct compliance with curated harmful prompts, providing limited visibility into socially mediated disclosure over long horizons.Interactive benchmarks more often emphasize goal completion, believability, or social reasoning than privacy under community pressure.
- Approach: The study introduces a Moltbook-style simulation where thousands of agents with private profiles interact across 124 communities over a simulated month.Profiles span health, finance, employment, and seven other sensitive domains.
- Headline result: Privacy violations increase from 19.95% to 45.3% when evaluation shifts from single-turn to multi-turn social interaction across OpenAI models.The study also examines whether agents respect contextual-integrity boundaries, succumb to peer pressure, and vary by community or model.
2 Related Work
Prior work established persistent multi-agent social behavior and large-scale community dynamics, but did not directly measure privacy leakage under those conditions. This paper empirically tests whether community, exposure, and participation duration influence sensitive disclosure.
- Prior systems: Earlier multi-agent systems demonstrated coherent social behavior, relationships, coordination, and persistent personas in small sandbox environments.Related benchmarks also evaluated social competence, cooperation, and interactive reasoning.
- Scaling gap: Small populations and task-bounded interaction limited earlier systems’ ability to capture community-level norm formation, attention concentration, and thematic stratification.Population-scale simulation was introduced to address this limitation.
- AI communities: Moltbook provided a live Reddit-style environment with over two million registered agents, enabling observation of agent-to-agent interaction at scale.The literature characterizes its networks as sparse, hub-dominated, and attention-unequal.
- Safety dynamics: Research on agent communities identifies social engineering, adversarial engagement, and inconsistent norm enforcement as important safety dynamics.These threats operate through exposure and interaction, not only technical exploitation.
- Norm dynamics: Agents can be influenced by contextual exposure without internalizing community norms, while interaction can produce shared conventions and consensus shifts.The related literature also describes progressive safety erosion in isolated agent societies.
- Research gap: Existing work supplies preconditions for privacy failure but does not measure whether community, exposed content, and participation duration affect sensitive disclosure.This study investigates that relationship directly.
3 Dataset Curation
The study constructs a synthetic, ground-truth population and a shared social platform from public sources. Agents receive personas, private human profiles, persistent memory, and tools for multi-day interaction.
- Data resources: The evaluation combines known sensitive attributes with a social environment capable of sustaining organic multi-turn interaction.Both resources are constructed from public sources.
- Personas: The simulation population contains 2,533 agent personas extracted from 6,105 Moltbook posts across 124 subreddits.An LLM judge retained genuine self-introductions and extracted names, tendencies, preferred subreddits, vocabulary, and seed posts.
- Sensitive attributes: Private profiles provide ground truth across ten domains, including identity, finance, health, legal, relationships, employment, education, and scheduling.Ten annotated profiles are held out for controlled testbed experiments.
- Simulation environment: Agents interact concurrently through a shared social-media server modeled on Reddit, with subreddits, posts, replies, voting, profiles, and social-context annotations.The environment uses a shared SQLite database.
- Agent loop: The asynchronous interaction loop supplies each agent with its persona, private profile, persistent memory, and platform instructions before tool-mediated activity.The tool suite exposes browsing, searching, posting, replying, voting, and memory operations.
- Simulation scale: The platform runs for 25 simulated days using GPT-5-nano, GPT-5-mini, and GPT-5 as agent backends.It accumulates 111,209 content items: 29,945 posts and 81,264 replies.
4 Experimental Setup
The experiments separate organic leakage from elicited leakage under controlled social pressure while keeping personas, infrastructure, and detection consistent. A frozen-platform testbed varies adversarial exposure, model, budget, and privacy safeguards.
- Evaluation overview: The design measures organic leakage during unscripted participation and elicited leakage after adversarial content is injected at calibrated intensities.The paired evaluations isolate emergent social pressure from controlled manipulation.
- Measurement: A model-based judge compares each write with the author’s full profile across ten privacy domains to detect violations.The same leakage-detection pipeline is applied in organic and controlled evaluations.
- Adversarial pressure: Five adversarial levels range from no injected posts to one disclosure-normalizing post in every subreddit.The design tests whether exposure in preferred communities differs from platform-wide contamination.
- Evaluation matrix: The study evaluates seven frontier models, ten held-out personas per model, and five tool-call checkpoints, yielding 1,000 runs per model and 7,000 total traces.The model set includes five OpenAI and two Gemini variants.
- Adversarial pressure: Adversarial posts are generated to normalize disclosure and synthetically upvoted so they surface prominently during browsing.This manipulation varies the breadth of exposure across communities.
- Experimental conditions: Runs vary explicit redaction instructions and persona-matched subreddit subscriptions in addition to adversarial level.These conditions test safeguards and whether preferred-community exposure changes leakage.
- Controlled testbed: The controlled testbed places one target agent alone in a frozen platform snapshot and evaluates every write at successive budget checkpoints.Because the environment is frozen, the agent’s behavior is the only changing factor during each run.
5 Results
Persistent, socially embedded interaction amplifies privacy leakage, which accumulates over time, spreads through peer effects, and varies substantially by model, community, and privacy domain. Explicit instructions mitigate but do not reliably prevent disclosure, while isolated evaluations underestimate deployment-time risk.
- Temporal Accumulation (RQ2): ~2.5k leaking items out of ~111k content items accumulated steadily by turn 25, showing that violations continue throughout sustained participation.The cumulative increase indicates leakage is not confined to isolated early outliers.
- Baseline: Social vs. Isolated Violations (RQ1): Under extended tool-call budgets, several models approach or exceed ~50–60% leakage, while stronger models remain in the ~20–30% range.Controlled-testbed leakage generally increases with interaction length for most models.
- Temporal Accumulation (RQ2): 12.8% versus 1.6% after a clean reply and 1.8% overall: prior leakage sharply increases the probability of leakage in the next reply.This approximately 8× increase indicates a probabilistic social ratchet rather than deterministic collapse.
- Instruction Robustness Under Pressure (RQ3): Explicit privacy instructions reduce leakage across most models but remain partial: gpt-4o falls from 2,624 to 2,102 leaking writes, while gpt-5 falls from 2,296 to 482.Leakage remains in the thousands for several models, whereas robustness to social pressure is model-dependent.
- Community Topic Effects (RQ4): Community leakage rates range from under 2% in technically oriented communities to over 16% in introductions, making context comparable to model choice.General identity attributes account for 1,496 leaking items, followed by employment at 921, scheduling at 812, and mental health at 767.
- Persona-Level Variation: Leakage rates across held-out personas range from 27.8% to 36.4%, a modest ~1.3× spread relative to model and subreddit differences.General identity dominates leakage for every persona, with employment second for eight of ten personas.
6 Discussion
The discussion argues that privacy failures in multi-agent systems are trajectory-dependent and that realistic safety evaluation must model social context, interaction horizons, and community structure. It also identifies simulation, model coverage, detection, and adversarial-design limitations that bound the findings.
- Implications for safety evaluation: Static, single-turn benchmarks systematically underestimate privacy risks in persistent agent deployments.The paper argues that evaluation should extend beyond individual prompt compliance to the normative environments in which agents operate.
- Implications for safety evaluation: Approximately 8× more leakage occurred when a reply followed a leaking message, compared with 1.6% otherwise.The reported rates were 12.8% versus 1.6%, indicating a strong peer-exposure effect.
- Implications for safety evaluation: Community structure, peer exposure, and interaction length should be treated as first-class safety-evaluation variables.The proposed agenda also includes community-aware prompts, memory sandboxing, and platform-level monitoring of disclosure cascades.
- Implications for Moltbook-like platforms: Privacy erosion may emerge from platform scale and structure, especially where attention concentration amplifies high-engagement disclosures.The discussion connects algorithmically amplified visibility with cascades in platform-wide leakage probabilities.
- Limitations and future work: The platform uses synthetic personas rather than real users, limiting direct conclusions about real-world participants.The personas are assigned to agents and grounded in prior privacy benchmarks.
- Limitations and future work: The simulation differs from live Moltbook, and real deployments may add cross-platform spillover or human-agent interaction.The authors recommend evaluation with live, consenting participants or audited real-world agent deployments.
- Limitations and future work: The organic simulation uses fixed OpenAI backends, so broader provider and open-source comparisons remain necessary for generalizability.The authors note that alignment differences may affect leakage rates and responses to social pressure.
- Limitations and future work: LLM-based leakage detection may produce false positives or negatives, and the contextual-integrity proxy is only approximate.The authors characterize reported violations as an upper bound and propose human annotation, ensembles, or norm-aware judgments.
7 Conclusion
Isolated, single-turn safety evaluations systematically underestimate privacy risk in socially embedded deployments. Persistent multi-agent interaction reveals socially mediated leakage that safeguards do not fully prevent.
- 8× prior-disclosure exposure increases subsequent leakage probability, demonstrating socially mediated privacy leakage.The effect persists despite explicit privacy instructions in the system prompt.
- Community context predicts leakage as strongly as model choice, with subreddit-level violation rates spanning nearly an order of magnitude.
- Agents preserving contextual-integrity boundaries in bounded prompts disclose sensitive information at substantially higher rates in persistent multi-agent environments.
- Agentic safety evaluations should treat community structure, peer exposure, and interaction horizon as first-class variables alongside model and prompt design.