Source-linked AI summary
SocialVeil: Probing Social Intelligence of Language Agents under Communication Barriers
Keyang Xuan, Pengda Wang, Chongrui Ye, Haofei Yu, Tal August, Jiaxuan You
TL;DR
Existing social benchmarks often assume idealized communication, limiting evaluation of whether language agents can maintain and repair interactions under realistic barriers. SOCIALVEIL addresses this gap by simulating three cognitive communication barriers in controlled social episodes and evaluating both task and communication outcomes. Across its experiments, barriers consistently impair social intelligence, human evaluations support the framework’s fidelity, and adaptation methods produce only modest recovery.
Problem
Existing social benchmarks assume idealized communication, limiting diagnosis of agents’ ability to maintain and repair interactions under communication barriers.
Method
SOCIALVEIL uses a literature-grounded framework with three cognitive barriers, controlled interactive episodes, barrier-aware evaluation, and targeted adaptation interventions.
Results
Communication barriers consistently impair agents’ social intelligence, while human evaluation supports simulated-barrier fidelity and evaluation reliability.
Takeaways & Limitations
SOCIALVEIL brings social-interaction evaluation closer to realistic communication and shows that repair instructions and interactive learning do not restore barrier-free performance.
Takeaways & Limitations
The study primarily evaluates text-only, discrete, short-term interactions rather than multimodal and longer-term communication.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly evaluated in interactive environments to test their social intelligence. However, existing benchmarks often assume idealized communication between agents, limiting our ability to diagnose whether LLMs can maintain and repair interactions in more realistic, imperfect settings. To close this gap, we present \textsc{SocialVeil}, a social learning environment that can simulate social interaction under cognitive-difference-induced communication barriers. Grounded in a systematic literature review of communication challenges in human interaction, \textsc{SocialVeil} introduces three representative types of such disruption, \emph{semantic vagueness}, \emph{sociocultural mismatch}, and \emph{emotional interference}. We also introduce two barrier-aware evaluation metrics, \emph{unresolved confusion} and \emph{mutual understanding}, to evaluate interaction quality under impaired communication. Experiments across 720 scenarios and four frontier LLMs show that barriers consistently impair performance, with mutual understanding reduced by over 45\% on average, and confusion elevated by nearly 50\%. Human evaluations validate the fidelity of these simulated barriers (ICC$\approx$0.78, Pearson r$\approx$0.80). We further demonstrate that adaptation strategies (Repair Instruction and Interactive learning) only have a modest effect far from barrier-free performance. This work takes a step toward bringing social interaction environments closer to real-world communication, opening opportunities for exploring the social intelligence of LLM agents.
1 Introduction
SOCIALVEIL addresses the gap between idealized social benchmarks and real-world communication by simulating cognitive communication barriers and evaluating whether agents can maintain and repair interactions. Experiments show that these barriers impair social intelligence, while adaptation strategies provide limited recovery.
- Motivation: Communication barriers are systematic influences that hinder mutual understanding and expose failure modes missed by aggregate metrics.The paper argues that barrier-aware analysis provides diagnostic insight for socially aware AI systems.
- Motivation: Existing social benchmarks often assume shared linguistic, sociocultural, and emotional assumptions, overlooking disruptions common in real interaction.These disruptions include semantic vagueness, sociocultural mismatch, and emotional interference.
- Main Discoveries: 58%↓ mutual understanding occurs under Semantic Vagueness, while Emotional Interference produces a 49%↓ decline in relationship quality.These examples illustrate that different barriers disrupt distinct aspects of social interaction.
- Main Discoveries: avg ICC ≈0.78 and avg Pearson’s r ≈0.80 support the fidelity of simulated barriers and the reliability of the evaluation protocol.Human annotators also achieved avg Accuracy ≈68% in identifying barrier types.
- Contributions: SOCIALVEIL introduces a barrier-aware interactive environment, evaluation protocol, and interventions for studying LLM social intelligence under impaired communication.The framework complements goal-oriented measures by assessing interaction maintenance and misunderstanding repair.
- Adaptation: Repair instructions are largely ineffective, whereas interactive learning yields steady but limited improvements, leaving a gap to human-level resilience.The contribution claims that adaptive training helps only modestly under communication barriers.
2 SocialVeil: A Barrier-Aware Social Learning Environment
SOCIALVEIL is designed as a controlled, barrier-aware social environment that injects structured cognitive disruptions into interactive episodes and evaluates both task success and communication quality. Its setup uses asymmetric two-agent simulations in which only one agent receives the barrier condition.
- Design Goals: SOCIALVEIL requires barriers to preserve social goals, remain systematically structured, and support evaluation beyond goal-oriented outcomes.The environment therefore evaluates communication failures alongside conventional social-interaction dimensions.
- Barrier Taxonomy: The taxonomy identifies Semantic Vagueness, Sociocultural Mismatch, and Emotional Interference as cognitive communication barriers.These represent ambiguity, culturally misaligned interpretations, and affective interference with understanding.
- Barrier Design: Each barrier combines a high-level style prompt with reproducible parameters covering stance, interaction tactics, confusion mechanisms, and exemplar templates.The two-layer design makes disruptions controlled and repeatable while preserving a specified behavioral pattern.
- Simulation Setup: Each episode is a two-agent role-play with private goals and profiles, neutralized public scenarios, one barrier agent, and one unmodified partner agent.Scenario neutralization prevents public descriptions from leaking private goals, while asymmetry isolates the injected disruption.
- Simulation Setup: At each turn, agents generate utterances from dialogue history, goals, and profiles, with only the barrier agent receiving the barrier specification.Episodes last at most 20 turns or end earlier when an agent exits.
- Evaluation: Evaluation combines goal completion, relationship quality, and knowledge-oriented dimensions with barrier-aware measures of communicative disruption.The pipeline also includes automatic metrics and human evaluation of alignment and barrier navigation.
3 Research Questions and Experiment Setup
The study asks whether simulated communication barriers create structured disruptions, impair social interaction, and resist adaptation. It evaluates four partner agents and tests repair instruction alongside interactive learning with behavior cloning and self-reinforcement.
- The study evaluates barrier validity, barrier effects, and barrier adaptation as three guiding questions.
- Four partner agents are evaluated across proprietary and open-weight model families using a GPT-4o-backed evaluation protocol.
- Adaptation Strategies: Repair Instruction adds explicit guidance to ask clarifying questions and paraphrase for confirmation.
- Adaptation Strategies: Interactive Learning uses behavior cloning on filtered expert trajectories, then self-reinforcement with additional high-quality dialogues.
- Table 2 compares each model’s performance between baseline and barrier conditions across goal-oriented and barrier-aware metrics.
4 Experiment Results
Communication barriers produce structured, barrier-specific degradation in social interaction, while adaptation strategies yield only limited recovery. Social dimensions decline more than goal-oriented performance, and neither mitigation strategy restores baseline proficiency.
- Barrier Validity: Barriers form distinct, compact t-SNE clusters separated from baseline representations, with the three barrier types also separated from one another.This pattern indicates structured internal modes rather than random noise.
- Barrier Effects: Barriers consistently degrade performance across all evaluated models and both goal-oriented and barrier-aware dimensions.
- Barrier Effects: Semantic vagueness reduces mutual understanding by 58% on average, emotional interference reduces relationship quality by 49%, and sociocultural mismatch increases confusion by 49%.
- Barrier Effects: Goal completion and knowledge acquisition decline by 20−30%, whereas relationship quality drops 45% and mutual understanding drops 52% across barrier types.
- Adaptation: Repair Instruction produces trivial improvements because static, barrier-agnostic prompts often yield shallow repetitions or generic clarifications.
- Adaptation: Interactive Learning produces consistent 10−20% gains, but performance remains significantly below the barrier-free baseline.
- Adaptation: Neither adaptation strategy improves GOAL scores relative to baseline, suggesting a trade-off between barrier mitigation and goal-oriented behavior.
5 Discussion
The discussion tests whether simulated barriers produce recognizable behavioral and linguistic disruptions, and validates the evaluation protocol with human judgments. Results show barrier-specific effects, reliable annotation, and strong alignment between human and model ratings.
- Linguistic signatures: Barrier analysis correlates reference pronouns, hedging, sentiment polarity, and self-focus pronouns with evaluation outcomes.These features were extracted from conversations to test whether barriers trigger systematic linguistic shifts.
- Linguistic signatures: Reference pronouns and self-focus correlate with higher confusion and lower mutual understanding, while positive sentiment aligns with better relationships and goal attainment.Figure 4 reports these associations across four models.
- Barrier-specific effects: Semantic barriers most strongly impair Mutual Understanding, emotional barriers disproportionately erode Relationship Quality, and Cultural barriers uniquely elevate Unresolved Confusion.Each barrier’s effect is measured as its deviation from the mean of the other two barriers under the same model, metric, and scenario.
- Human evaluation: Unresolved Confusion achieved ICC = 0.77 and Mutual Understanding achieved ICC = 0.79, indicating good inter-rater consistency.Both results were statistically significant at p < .001.
- Human evaluation: Human annotators identified barriers with overall accuracy 0.68, while model and human scores showed convergent validity of 0.80 for confusion and 0.79 for mutual understanding.The accuracy estimate used cluster bootstrap resampling, and the convergent-validity results included 95% confidence intervals.
6 Related Works
Prior work evaluates social intelligence through static benchmarks, interactive environments, and agent simulations, but commonly assumes overly idealized communication. This leaves real-world conversational complexity and communication barriers insufficiently assessed.
- Static Benchmarks for Social Intelligence and Their Limitations: Static benchmarks draw on clinical, psychological, and social commonsense tasks but cannot capture the complexity and diversity of interactive settings.Consequently, evaluating intelligence in real-world social interactions remains a significant gap.
- Interactive Evaluation of Social Intelligence and Agents Simulation: Interactive research uses LLMs to simulate social interactions, credible human behavior, and collaborative software development.These studies span applications including social media design and agent-based social simulation.
7 Conclusion
SOCIALVEIL introduces a barrier-aware environment for evaluating LLM social intelligence under communication barriers. Its experiments show consistent impairment, human-validated simulation and evaluation, and only modest gains from adaptation strategies.
- Conclusion: SOCIALVEIL simulates and evaluates LLM agents’ social intelligence in the presence of communication barriers.The environment is designed to make social interaction evaluation more realistic.
- Conclusion: Communication barriers consistently impair agents’ social intelligence capabilities across the experiments.Human evaluations validate both the realism of the simulated barriers and the robustness of the evaluation protocol.
- Conclusion: Repair instructions are largely ineffective, while interaction-driven learning yields modest but consistent improvements.The conclusion frames these findings as motivation for more realistic social interaction environments.
Ethics Statement
The ethics statement describes safeguards for the barrier taxonomy, simulation protocols, and human evaluations. It also positions SOCIALVEIL as a diagnostic framework for identifying and repairing communication failures.
- Ethics Statement: The taxonomy and simulation protocols exclude hate speech, discriminatory stereotypes, and harmful biases.These safeguards are part of the framework’s ethical design.
- Ethics Statement: Human evaluations used informed consent, fair compensation, and anonymization to protect participant privacy.These procedures apply to the human-evaluation component of the study.
- Ethics Statement: SOCIALVEIL is intended to identify and repair communication failures as a diagnostic framework.The stated aim is to foster socially resilient AI development.
B The limitations of SOCIALVEIL and future directions
SOCIALVEIL is currently limited to text-only, short-term interactions and episode-level evaluation, motivating multimodal and longitudinal extensions. Its barrier taxonomy is grounded in a systematic literature review of cognitive-difference-induced communication barriers.
- Limitations and future directions: Text-only interactions leave non-verbal barriers such as prosody, facial expressions, and gestures outside the current framework.These modalities are identified as important for grounding, coherence, and mutual understanding.
- Limitations and future directions: Short-term, discrete scenarios do not model the cumulative and subtle effects of barriers across long-term interactions.Future work is proposed to represent continuous scenarios with evolving barriers.
- Limitations and future directions: The current evaluation diagnoses episode-level social intelligence but does not fully assess proactive barrier mitigation.The paper identifies proactive mitigation as a direction for future research.
- Barrier taxonomy development: A systematic literature review of cognitive-difference-induced interaction barriers identified Semantic Vagueness, Sociocultural Mismatch, and Emotional Interference.The study’s review draws on psychological and sociological theories related to interaction and communication.
E Details of Human Evaluation
The human evaluation uses six annotators to classify communication barriers and rate confusion and mutual understanding across sampled interaction scenarios. A web interface centralizes definitions, examples, transcripts, and annotation inputs.
- Human evaluation setup: Six annotators from two universities participated in the human evaluation.The annotator group was reported as 50% women and 50% men.
- Human evaluation setup: Annotators labeled 120 Qwen2.5-7B-Instruct scenarios, with 30 samples for each barrier type and the non-barrier baseline.Each scenario received ratings from at least three annotators.
- Annotation procedure: Annotators classified dialogues as semantic, cultural, emotional, or none and rated unresolved confusion and mutual understanding on 1–5 scales.Higher scores indicate no confusion and stronger alignment, respectively.
- Annotation procedure: The annotation interface provides barrier definitions, examples, full transcripts, and direct input fields in one place.The design is intended to reduce cognitive load and annotation errors.
F.1 Robustness Across Different Evaluator Backbone Models
Sensitivity analyses indicate that SOCIALVEIL’s observed degradation is robust to changes in evaluator and barrier-agent backbones. A transfer study additionally reports zero-shot gains on AgentSense after interactive learning.
- Evaluator backbone robustness: Social-intelligence declines persist across all barrier types when GPT-4o is replaced by Llama-3.1-8B-Instruct as evaluator.Absolute scoring differs, but relative degradation remains consistent across evaluator backbones.
- Barrier-agent backbone robustness: Social-intelligence declines persist across dimensions when Qwen2.5-7B-Instruct is used for both the barrier and partner agents.The results support consistency under a homogeneous model configuration.
- Cross-benchmark transfer: Qwen2.5-7B and Qwen3-4B show significant goal-completion improvements on AgentSense in a strict zero-shot transfer evaluation.The transfer study applies interactive learning within SOCIALVEIL before evaluation on AgentSense.
G Details For Evaluation Protocol
The evaluation protocol combines goal-oriented measures of social outcomes with barrier-aware measures of communication disruption. Prompts provide agents with scenario and profile context while keeping partner goals and private knowledge selectively hidden.
- Goal-oriented dimensions: Believability evaluates naturalness, identity consistency, repetition, politeness, and alignment with character traits.Higher scores indicate more believable behavior.
- Goal-oriented dimensions: Relationship evaluation considers the pre-interaction relationship and how it changes after the interaction.The rubric covers effects on personal relations, social status, and reputation.
- Barrier-aware dimensions: SOCIALVEIL adds Unresolved Confusion and Mutual Understanding to measure communication disruptions caused by barriers.Both barrier-aware metrics use 0–5 rubrics describing unresolved goals, ambiguity, viewpoint alignment, and shared understanding.
- Barrier-aware dimensions: The confusion rubric ranges from incoherent interaction with unanswered questions to fully resolved goals without ambiguity.Intermediate levels capture persistent misunderstandings, partial comprehension, and unclear secondary details.
- Barrier-aware dimensions: The mutual-understanding rubric ranges from talking past each other to full alignment on goals, intentions, and strategies.Intermediate levels distinguish generic agreement, partial recognition, and understanding that misses nuances.
- Agent prompt: Agent prompts specify scenario, agent and partner backgrounds, public information, goals, reasons, private knowledge, turn number, and available actions.Each agent sees its own goal while the partner’s goal and reason remain unknown; private knowledge should influence strategy without explicit disclosure unless useful.
- Agent prompt: The interaction pipeline permits agents to leave conversations and requires actions to be emitted as JSON containing an action type and argument.Prompts also require natural, realistic interaction and prohibit unnecessary repetition.