PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networks

Prince Zizhuang Wang, Shuli Jiang

arXiv:2604.01487v2cs.AIcs.SI

TL;DR

Human-centered agentic social networks create an underexplored privacy problem as agents coordinate across domains, mediate between people, and interact across users. AgentSocialBench evaluates this setting with structured scenarios, profiles, social graphs, and privacy metrics, finding persistent leakage pressure and an abstraction paradox in prompt-based defenses. The results motivate approaches beyond prompt engineering for safer agent-mediated coordination.

  • Problem

    Privacy risks in human-centered agentic social networks remain underexplored because agents must coordinate across domains and users while protecting the personal information they hold.

  • Method

    AgentSocialBench evaluates more than 300 scenarios across seven dyadic and multi-party categories using sensitivity-labeled profiles, directed social graphs, privacy boundaries, leakage metrics, abstraction scores, and multiple instruction levels.

  • Results

    The benchmark finds that cross-domain coordination creates strong leakage pressure and that abstraction-oriented privacy instructions can increase partial disclosure despite reducing direct leakage.

  • Takeaways & Limitations

    Making agents safer in human-centered social coordination requires privacy-preservation approaches beyond prompt engineering.

Abstract

from arXiv · show

With the rise of personalized, persistent LLM agent frameworks such as OpenClaw, human-centered agentic social networks in which teams of collaborative AI agents serve individual users in a social network across multiple domains are becoming a reality. This setting creates novel privacy challenges: agents must coordinate across domain boundaries, mediate between humans, and interact with other users' agents, all while protecting sensitive personal information. While prior work has evaluated multi-agent coordination and privacy preservation, the dynamics and privacy risks of human-centered agentic social networks remain unexplored. To this end, we introduce AgentSocialBench, the first benchmark to systematically evaluate privacy risk in this setting, comprising scenarios across seven categories spanning dyadic and multi-party interactions, grounded in realistic user profiles with hierarchical sensitivity labels and directed social graphs. Our experiments reveal that privacy in agentic social networks is fundamentally harder than in single-agent settings: (1) cross-domain and cross-user coordination creates persistent leakage pressure even when agents are explicitly instructed to protect information, (2) privacy instructions that teach agents how to abstract sensitive information paradoxically cause them to discuss it more (we call it abstraction paradox). These findings underscore that current LLM agents lack robust mechanisms for privacy preservation in human-centered agentic social networks, and that new approaches beyond prompt engineering are needed to make agent-mediated social coordination safe for real-world deployment.

1 Introduction

Human-centered agentic social networks extend multi-agent coordination to teams serving individual humans, creating privacy risks that existing agent-only and coordination benchmarks do not capture. AgentSocialBench addresses this gap with a structured benchmark and reveals that abstraction-oriented privacy instructions can increase leakage.

  • Human-centered agentic social networks involve personalized agent teams serving individual humans, unlike agent-only interaction platforms.
  • Existing multi-agent benchmarks evaluate task completion and coordination but do not capture privacy preservation for agents serving individual users.
  • AgentSocialBench contains more than 300 scenarios across seven dyadic and multi-party categories, using synthetic multi-domain profiles, hierarchical sensitivity labels, directed social graphs, and explicit privacy boundaries.
  • The benchmark introduces category-specific leakage metrics, an information abstraction score, and a privacy instruction ladder for measuring prompt-based defenses.
  • Privacy instructions teaching agents to abstract sensitive information can paradoxically increase leakage when agents would otherwise remain silent.

2 Human-Centered Agentic Social Network

Human-centered agentic social networks connect users and domain-specialized agent teams through directed social structures, requiring information to cross domain, user, and trust boundaries. The benchmark frames privacy preservation as an empirical question across dyadic and multi-party coordination, including whether prompt defenses create new disclosure channels.

  • Each user is represented by a directed social-graph node served by domain-specialized agents holding slices of that user’s private information.
  • Agent teams coordinate internally across domains, mediate communication between humans, and interact with other users’ agent teams.
  • Dyadic interactions involve two parties, whereas multi-party interactions involve three or more participants connected through directed social graphs.
  • Privacy norms constrain information flows across domain, user, and trust boundaries, requiring sensitive information to be abstracted rather than disclosed verbatim.
  • Multi-party boundary interactions expand the privacy surface combinatorially because each private item must be checked against every recipient’s sharing rules.
  • The benchmark asks whether multi-party dynamics amplify privacy risks and whether lightweight prompt defenses reduce leakage or introduce unintended disclosure channels.

Benchmark

AgentSocialBench grounds evaluation in synthetic multi-domain user profiles, structured interaction categories, explicit privacy defenses, and judge-based leakage and utility metrics. Its design supports measuring both privacy-preserving abstraction and task performance across distinct coordination settings.

  • User Profiles: Each scenario uses a synthetic user profile spanning six domains, with attributes labeled from public sensitivity level 1 to highly sensitive level 5.
  • User Profiles: Multi-party scenarios add directed social graphs with asymmetric affinity tiers and human-expert-annotated criteria for both coordination success and privacy preservation.
  • Scenario Categories: The seven categories include dyadic cross-domain and mediated-communication scenarios in which agents must share necessary information without revealing underlying private facts.
  • Defense Mechanism and Privacy Instruction Levels: L0 provides no privacy guidance, L1 adds explicit rules and acceptable abstractions, and L2 adds domain thresholds, abstraction templates, and minimal-information reasoning.
  • Evaluation Metrics: Leakage Rate classifies private items as unmentioned, partially leaked, or fully leaked, while category-specific variants reflect different interaction structures.
  • Evaluation Metrics: Information Abstraction Score measures acceptable reformulation on a 0, 0.5, or 1.0 scale, alongside Task Completion Quality as a utility measure.

4 Experiments

Across eight LLM backbones, the benchmark finds that cross-domain coordination creates the strongest leakage pressure, while multi-party structures produce distinct, category-specific privacy dynamics. Privacy instructions improve abstraction and suppress explicit violations, but can increase partial or implicit leakage without measurable task-quality costs.

  • Overall Model Comparison (RQ1): Cross-domain coordination produces roughly double the leakage of mediated and cross-user interactions, while no model dominates every privacy dimension.More capable models achieve higher task quality but also leak more.
  • Multi-Party Social Dynamics (RQ2): Multi-party settings reshape privacy risks rather than uniformly amplifying them: group chat resembles dyadic mediation, while competition and affinity tiers show different dynamics.Competitive settings suppress self-disclosure, and affinity tiers are respected with near-perfect compliance.
  • Effect of Privacy Instructions (RQ3): Privacy instructions sharply improve information abstraction, but leakage effects diverge across categories and remain stable in task completion quality.Aggregate dyadic leakage decreases modestly; cross-domain leakage falls, whereas mediated, cross-user, and some multi-party leakage increase.
  • The Abstraction Paradox: The abstraction paradox arises when abstraction templates authorize partial references to topics that agents would otherwise leave unmentioned.The net effect depends on baseline leakage: defenses reduce more existing leakage in cross-domain scenarios but introduce more partial leakage in mediated and cross-user scenarios.
  • Behavioral Analysis: Defenses suppress explicit violations and improve minimal disclosure, yet implicit inference-based leakage remains high even under full defense.All models achieve near-perfect appropriate abstraction under L2, but abstraction language can reshape leakage from explicit to implicit forms.

5 Related Work

Prior multi-agent benchmarks and social platforms demonstrate coordination and emergent behavior, but do not systematically evaluate privacy in human-centered agentic social networks. Existing work therefore provides motivation and context without covering the benchmark’s target setting.

  • Multi-Agent Benchmarks and Privacy Evaluation: Existing benchmarks evaluate coordination, competition, social intelligence, secret-keeping, and emergent social behavior across multi-agent environments.Examples include MultiAgentBench, CuisineWorld, SOTOPIA, and large-scale social simulations.
  • Agentic Social Platforms: Agentic social platforms such as OpenClaw and Moltbook motivate the benchmark, but Moltbook is designed for agent-only interaction rather than agents serving individual humans.The related platforms provide context for the emerging setting without addressing its human-centered privacy dynamics.

6 Conclusion

AGENTSOCIALBENCH evaluates privacy preservation across realistic dyadic and multi-party agent interactions, showing that multi-party coordination and abstraction-based defenses create distinct leakage risks.

  • AGENTSOCIALBENCH evaluates more than 300 scenarios across seven categories and eight LLM backbones.The benchmark uses synthetic multi-domain user profiles with hierarchical sensitivity labels and directed social graphs.
  • Experiments identify cross-domain coordination as the strongest leakage pressure and multi-party settings as producing category-specific failure modes.The conclusion reports these patterns across the benchmark rather than for a single scenario.
  • Prompt-based defenses exhibit an abstraction paradox in which sanctioned privacy language increases the surface area of partial disclosure.This finding motivates approaches beyond prompt engineering for safer agent-mediated social coordination.
  • The scenarios cover cross-domain coordination, mediated communication, joint coordination, group coordination, hiring, competition, and affinity-aware sharing.Examples include communicating activity constraints without revealing diagnoses and aggregating salary preferences without exposing individual figures.
  • The benchmark represents privacy boundaries through hierarchical sensitivity labels, authorized abstractions, and restrictions on disclosing diagnoses, medications, finances, and other personal details.Scenario profiles distinguish prohibited disclosures from acceptable abstractions such as activity limitations, dietary preferences, or general experience.

B Privacy-Utility Tradeoff Analysis

Privacy instructions produce different privacy-utility tradeoffs across dyadic and multi-party categories. Dyadic settings show reduced leakage, whereas multi-party settings exhibit the abstraction paradox without clear Pareto dominance.

  • Dyadic categories: Dyadic leakage decreases from 0.36 (L0) to 0.32 (L2), a modest −11% reduction, while TCQ remains stable (0.79→0.77).Task completion rate is largely unchanged across the instruction levels.
  • Dyadic categories: IAS increases from 0.76 to 0.92 in dyadic categories, indicating improved abstraction without measurable utility cost.The passage identifies IAS as the primary beneficiary of substitutive defenses.
  • Multi-party categories: Multi-party leakage increases from 0.11 (L0) to 0.13 (L2), while IAS improves from 0.76 to 0.89.Task completion quality and completion rates remain stable or slightly improve.
  • Model comparison: No model achieves Pareto dominance across all categories: Claude Sonnet 4.6 has the best TCQ, while GPT-5 Mini has the lowest CD leakage but worst task quality.The result underscores the multi-objective nature of privacy-preserving coordination.

C Benchmark Details

The benchmark evaluates agent interactions with separate engines for dyadic and multi-party settings, using shared privacy controls and visibility-filtered conversation logs.

  • Evaluation engines: Dyadic categories use a turn-based engine with domain-specialized agents that maintain persistent and temporary memory.The dyadic categories are CD, MC, and CU.
  • Evaluation engines: Multi-party categories use a graph-driven engine with round-robin turns operating on directed social graphs.The multi-party categories are GC, HS, CM, and AM.
  • Shared infrastructure: Both engines share an LLM interface, privacy instruction injection, and conversation logging with per-agent visibility filtering.Agents observe only messages they sent, messages directed to them, or system broadcasts.
  • Metrics: Figure 5 compares privacy instruction levels L0, L1, and L2 across three metrics and three categories.Full results, including TCQ, appear in Figure 7 in the appendix.

D Defense-Measurement Entanglement

The study examines whether abstraction-based defenses and their measurement are entangled, concluding that increased partial leakage reflects a behavioral change rather than merely a judging artifact.

  • Measurement concern: The defense-measurement concern arises because L2 templates teach abstraction phrases that the judge classifies as partial leakage.The authors explicitly treat this possible entanglement as a methodological concern.
  • Measurement concern: The privacy judge is template-blind and receives only conversation history and privacy-boundary specifications.It does not receive the instruction level, system prompt, or acceptable-abstraction list.
  • Baseline design: L0 provides a disentangled baseline, allowing the L0-to-L2 difference to capture intended and unintended effects of the defense intervention.These effects include reduced full leakage, improved abstraction quality, and increased partial leakage through newly introduced abstraction language.
  • Design implication: Substitutive defenses increase partial leakage, whereas suppressive defenses avoid that side effect but may degrade task performance when information is needed for coordination.The authors frame this as a design tradeoff between privacy-preserving vocabulary and avoiding topics entirely.
  • Item-level validation: Among items with no leakage at L0, 10–20% transition to partial leakage at L2, while 77–93% of full-leakage items transition to no or partial leakage.The transition analysis supports both the unintended disclosure effect and the reduction of severe violations.

E Task Completion Quality Grading

The evaluation grades coordination outcomes against human-expert success criteria, while scenario-specific abstractions and system prompts define how agents communicate across dyadic and multi-party settings.

  • Task Completion Quality Grading: The evaluator assigns coordination outcomes one of five levels against human-expert-annotated success criteria.This grading procedure is described alongside representative examples in Table 8.
  • Task Completion Quality Grading: Scenario-specific abstraction mappings preserve task-relevant information while concealing underlying sensitive details.They also define the privacy boundary and supply vocabulary for the L2 abstraction defense.
  • Task Completion Quality Grading: Figure 6 organizes cross-domain leakage by source-to-target domain pair, with Lifestyle→finance and finance→schedule most vulnerable and social→schedule least vulnerable.Rows are sorted by mean leakage across models.
  • Task Completion Quality Grading: The benchmark evaluates dyadic and multi-party interactions using distinct prompts that encode participant roles, visibility, relationships, and privacy rules.Multi-party prompts distinguish group and private communication, while affinity-modulated agents apply the most restrictive rule among group recipients.
  • Task Completion Quality Grading: L1 adds explicit privacy rules and acceptable abstractions, while L2 composes domain-boundary prompting, abstraction templates, and minimal-information prompting.The defenses are additive: L2 = L1 + DBP + IAT + MIP.

I Statistical Methodology

The paper uses bootstrap resampling to quantify uncertainty and paired bootstrap tests to compare conditions within the same scenarios.

  • Statistical Methodology: Reported confidence intervals are 95% bootstrap intervals based on 1,000 resamples of per-scenario scores.The procedure estimates the sampling distribution of the mean rather than repeating evaluations.
  • Statistical Methodology: Significance claims use two-sided paired bootstrap tests with 10,000 iterations and scenario-level pairing.Resampling preserves within-scenario correlation and makes no distributional assumptions about metric values.

J Metric Definitions

The benchmark defines category-specific leakage metrics and additional measures for abstraction quality and privacy decision accuracy across interaction structures.

  • Metric Definitions: CDLR measures the fraction of private items leaked in cross-domain scenarios, averaged per scenario, model, and source-to-target domain pair.Partial and full leakage both count as leakage.
  • Metric Definitions: MLR analogously measures leakage in mediated communication from the non-served user’s perspective.Its aggregation follows the cross-domain leakage formulation.
  • Metric Definitions: CULR reports both the mean leakage across two users and the worst-case user’s leakage in cross-user scenarios.The privacy judge runs twice per scenario, once for each user’s private items.
  • Metric Definitions: MPLR measures leakage across all private-item and recipient pairs, capturing the combinatorial leakage surface in multi-party scenarios.The denominator contains every private item–recipient pair.
  • Metric Definitions: HALR measures cross-leakage across ordered participant pairs, while CSLR and CER separately measure self-leakage and competitor extraction.These metrics distinguish leakage direction and competitive extraction behavior.
  • Metric Definitions: ACS measures whether sharing decisions match tier-specific privacy rules, and IAS averages abstraction scores across cross-boundary communications.ACS is the fraction of correct decisions; IAS uses abstraction scores of 0, 0.5, or 1.0.

K Future Work

Future work targets architectural privacy mechanisms, more realistic deployment data, longitudinal evaluation, broader safety dimensions, and larger-scale human assessment.

  • Architectural Privacy Mechanisms: Architectural approaches such as framework-level information-flow control, learned privacy policies, and formal verification are proposed beyond prompt-based defenses.The abstraction paradox motivates these directions because prompt defenses have fundamental limitations.
  • Real-World Deployment Evaluation: Evaluating anonymized real deployment traces would provide ecological validity that synthetic benchmarks cannot.This is proposed as agentic social platforms mature.
  • Longitudinal and Cross-Session Privacy: Multi-session evaluation could capture aggregation risks missed when agents are evaluated in single-session settings.The proposed extension models agents accumulating knowledge across conversations.
  • Broader Social Dynamics: Integrating privacy with manipulation, misinformation, and social-influence evaluation would provide a more complete safety assessment.The current benchmark focuses on privacy preservation.
  • Human-in-the-Loop Evaluation: Large-scale human evaluation, including cross-cultural studies, could strengthen construct validity and reveal disagreements between LLM and human judges.This would test cases where automated and human assessments diverge.
Loading 2604.01487v2…