Source-linked AI summary
APRiL: An Open-ended Benchmark for Chinese Social Awareness in Large Language Models
Tian Lan, Han Zhang, Shanshan Wang, Yahan Yu, Jiang Li, Yugo Murawaki
TL;DR
Existing benchmarks often assess only particular social abilities, formats, or interaction settings, leaving limited evidence on social awareness across diverse contexts. APRiL is an open-ended Chinese benchmark that evaluates scenario analysis and socially aware responses. Across evaluated models, scenario understanding and response performance do not move in parallel, and higher-quality analyses are associated with better responses.
Problem
Existing evaluations rarely test joint interpretation of social cues, mental states, and conflicts followed by an appropriate response across varied settings.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
The findings support evaluating both social understanding and response performance because scenario analysis alone does not fully characterize response quality.
Source: §7 · p. 8APRiL is primarily designed for Chinese language and culture, and its conclusions may not transfer directly to other languages and cultures; evaluated models and scenarios also do not cover all existing models or real-world interactions.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large language models are increasingly used in social interactions, where appropriate responses require recognizing implicit cues, inferring mental states, identifying conflicts, and considering potential consequences. However, existing benchmarks often focus on specific social abilities, constrained task formats, or particular interaction settings, offering limited coverage of social awareness across diverse contexts. To address this gap, we introduce APRiL, an open-ended benchmark for Chinese social awareness with instance-specific rubrics. APRiL contains 2,196 Social Awareness Evaluation Instances across six categories and two complementary tasks: Social Scenario Analysis evaluates models' understanding of social cues, mental states, and conflict structures, while Socially Aware Response assesses whether they can translate this understanding into appropriate, natural responses. Experiments on 12 models from six families reveal a consistent understanding-to-response gap. Higher-quality scenario analyses improve responses, while weak analyses may reduce response quality. APRiL provides a systematic resource for advancing socially aware language models in Chinese contexts. Our data and code are available at https://anonymous.4open.science/r/APRiL-05DE/.
1 Introduction
Social interaction requires more than reading surface meaning: models must interpret implicit cues, relationships, mental states, conflicts, and consequences. APRiL addresses gaps in existing evaluations with an open-ended Chinese benchmark that separately examines scenario understanding and socially appropriate response generation.
1 Introduction
- Indirect expressions, silence, tone, and setting can carry social meaning, and missing that meaning may create interpersonal risk.For example, treating prolonged silence from a professor only as busyness may prompt follow-ups that pressure the professor and expose an applicant to risk.Source: §1 · pp. 1–2
- Existing benchmarks rarely assess joint interpretation of social cues, mental states, and conflicts followed by an appropriate response across diverse settings.Many use narrow scenario types or formats, limiting evidence across relationships and interaction settings.Source: §1 · p. 2
- APRiL evaluates Chinese social awareness through 2,196 instances in six categories and two open-ended tasks: scenario analysis and response generation.Social Scenario Analysis assesses cues, mental states, and conflict structures; Socially Aware Response assesses natural, appropriate responses in context.Source: §1 · pp. 2–3
- The paper evaluates 12 models from six families and examines the gap between scenario understanding and response generation, including the effects of analysis quality.The perspective is whether models integrate implicit social information and translate their understanding into appropriate responses.Source: §1 · pp. 2–3
2 Related Works
The related work connects social awareness to cue interpretation, mental-state inference, goals, and conflict. Existing LLM benchmarks cover distinct abilities and interaction formats, while APRiL combines open-ended scenario understanding with response generation across Chinese social contexts.
2.1 Social Awareness and Cognitive Foundations
- Social-information processing links social-cue interpretation with interaction goals, response construction, and consequence evaluation; Theory of Mind and conflict research motivate APRiL’s focus.The framework attends to observable cues, evidence-grounded mental-state inference, latent conflicts, and the response that follows.Source: §2.1 · p. 3
2.2 Social Understanding and Interaction in LLMs
- Prior benchmarks assess social norms, mental states, pragmatic meaning, emotional understanding, or interactive goals, often through constrained formats or specific settings.APRiL instead spans six Chinese social-context categories and separately evaluates fine-grained scenario understanding and open-ended response generation.Source: §2.2 · p. 3
3 The Dataset
APRiL’s dataset combines reviewed social scenarios with instance-specific rubrics for two evaluation tasks. Its 2,196 instances span six relatively balanced categories and include varied dialogue lengths and scoring criteria.
3 The Dataset
- For each instance, models see only the Visible Context and Dialogue; remaining annotations guide rubric construction and judging.APRiL comprises 2,196 Social Awareness Evaluation Instances.Source: §3 · p. 3
3.1 Dataset Construction
- With expert guidance, the authors organized scenarios into six categories, manually designed seeds, and retained 560 after review.GPT-5.5 expanded seeds into candidates; eight native Chinese-speaking graduate-level reviewers checked annotations and removed instances with obvious cues, unnatural language, or insufficient interaction constraints.Source: §3.1 · pp. 3–4
- Instance-specific rubrics derive from hidden annotations and ideal response strategies, using score points for expected content and deduction points for unsupported inferences or interactional risks.Sixteen annotators reviewed the rubrics, merging overlapping points and refining unclear criteria.Source: §3.1 · p. 4
3.2 Statistics
- APRiL contains 2,196 instances across six relatively balanced categories, with each category representing 14.48–20.36% of the dataset.Instances average 7.32 turns and 318.89 Chinese characters; rubrics average 12.58 SSA score points and 5.65 deduction points, and 4.91 SAR score points and 4.04 deduction points.Source: §3.2 · p. 4, Table 1 · p. 4
4 Evaluation Tasks
APRiL defines social awareness as interpreting social information and using it to respond appropriately to context and consequences. Its evaluation separates scenario analysis from response quality, using task-specific scoring criteria.
4 Evaluation Tasks
- Social awareness comprises identifying social cues, understanding relationships, mental states, and conflicts, then responding in a way that fits the relationship and setting while accounting for consequences.APRiL evaluates these abilities through Social Scenario Analysis and Socially Aware Response, with task-specific points reported on a 0–100 scale.Source: §4 · p. 4
4.1 Social Scenario Analysis
- Scenario Analysis evaluates social cues, evidence-based mental-state inference, and conflict structure, including competing interests, role responsibilities, and value trade-offs.Scoring rewards coverage and distinguishing fact from inference, while unsupported motive claims and generic analysis lower scores; matched score points and triggered deductions are pooled within dimensions, whose scores are averaged.Source: §4.1 · pp. 4–5
4.2 Socially Aware Response
- Socially Aware Response assesses whether a response fits the relationship, power structure, setting, and practical task while considering participants’ goals and concerns.Responses should avoid unsupported motive claims, address tensions through clarification, options, or boundary setting, and remain concise; excessive length triggers an additional deduction.Source: §4.2 · p. 5
5 Experimental Setup
The experiments compare 12 models from six families across scenario analysis and response generation. Runs use default settings, are repeated three times, and are scored by an LLM judge, with a human subset and alternative judges used for checks.
Models
- The evaluation covers 12 models from six families, including black-box systems and open-weight Ministral and Qwen3 models.DeepSeek and Kimi models are tested in thinking and non-thinking modes; Qwen3 is tested in non-thinking mode.Source: §5 · p. 5
Running
- SSA and SAR run independently; open-weight models use four 80GB NVIDIA A100 GPUs, and each model is run three times with default parameters.The reported scores are means and standard deviations across runs.Source: §5 · p. 5
LLM-as-a-Judge
- GPT-5.6-Luna scores rubric matches and deduction triggers at temperature 0, and three alternative judges are used to test dependence on the primary judge.Prompts, scoring examples, human inter-rater agreement, and human–judge correlations are reported in appendices.Source: §5 · p. 5
Human Performance
- Four Chinese adults with bachelor’s degrees complete both tasks on a randomly selected subset of 110 instances, with their answers scored by the same judge.The reported human result is the participant mean.Source: §5 · p. 5
6 Results and Analysis
Across seven analyses, the results show distinct strengths in scenario analysis and response generation, with performance affected by rubric deductions, analysis quality, model scale, reasoning mode, response length, and category. The recurring pattern is that stronger scenario analysis does not automatically translate into stronger responses.
6.1 Main Results
- Black-box models outperform open-weight models by 19.87 points on SSA and 21.88 points on SAR; DeepSeek-V4-Pro (T) leads overall SSA dimensions, while GPT-5.1 leads Social Cues and Gemini-3.8-Flash leads SAR.Mental States is the lowest SSA dimension across models and reasoning modes, averaging 57.27 versus 73.50 for Social Cues and 73.82 for Conflict Structure.Source: §6.1 · pp. 5–6, Table 2 · p. 6
6.2 Matched and Triggered Points
- DeepSeek-V4-Pro (NT) scores lower than GPT-5.1 on SSA (81.44 vs. 82.57) despite matching more score points (87.60% vs. 85.61%), as its deduction trigger rate is higher (13.77% vs. 6.65%).SAR has 0.82 deduction points per score point, nearly twice SSA’s density; Qwen3-14B’s low SSA trigger rate (5.29%) contrasts with its SAR trigger rate of 30.75%.Source: §6.2 · p. 6
6.3 Understanding-to-Response Gap
- SSA and SAR are positively associated (Spearman’s ρ = 0.511), but models score higher on SSA, and strong analysis does not guarantee an appropriate response.Ministral-8B combines strong SSA with much weaker SAR; the roommate example exposes deliberately withheld information and produces an overlong reply.Source: §6.3 · pp. 6–7
6.4 Does Scenario Analysis Improve Responses?
- Own SSA raises GPT-5.1’s SAR from 37.65 to 45.07 and Gemini-3.8-Flash’s from 42.88 to 47.33; gold SSA raises them further to 48.69 and 52.88.Gold SSA supplies annotated Social Cues, Mental States, and Conflict Structure while instructions and scoring remain unchanged.Source: §6.4 · p. 7
- Weak SSA raises GPT-5.1’s SAR to 40.83 but lowers Gemini’s to 41.95; Gemini’s deduction trigger rate rises from 19.18% to 21.94%, offsetting increased coverage.The example shows Gemini adopting weak SSA’s framing, while GPT-5.1 retains the scenario’s permission to remain seated.Source: §6.4 · p. 7
6.5 Effects of Model Scale and Thinking Mode
- Ministral improves from 3B to 8B on SSA (56.03 to 76.74) and SAR (4.07 to 18.54), while Qwen3 shows no monotonic SSA trend and declining SAR with scale.Thinking raises SSA and SAR for the tested DeepSeek and Kimi models, but higher scores do not always coincide with fewer deductions.Source: §6.5 · p. 7, Figure 3 · p. 7
6.6 Response Length
- GPT-5.1 loses 8.95 SAR points to length deductions, versus 0.46 for Gemini-3.8-Flash, shifting the top rank from GPT-5.1 without deductions (46.60) to Gemini-3.8-Flash with deductions (42.88).Thinking mode does not consistently raise over-length rates: DeepSeek-V4-Flash falls from 3.0% to 1.9%, while DeepSeek-V4-Pro and Kimi-K2.5 rise.Source: §6.6 · p. 7, Figure 4 · p. 8
6.7 Performance across Categories
- Across categories, SSA averages vary from 67.02 to 69.46 while SAR varies from 19.20 to 31.47; Mental States is the lowest SSA dimension in every category.Supportive Interaction has the lowest average SAR, despite an SSA average comparable to other categories.Source: §6.7 · pp. 7–8, Table 4 · p. 8
7 Conclusion
APRiL evaluates Chinese social awareness through separate scenario-analysis and response tasks. Results across 12 models show that analysis quality and response quality are related but distinct.
7 Conclusion
- APRiL contains 2,196 SAEIs across six categories, with structured annotations, instance-specific rubrics, and separate analysis and response tasks.Social Scenario Analysis evaluates understanding; Socially Aware Response evaluates response generation.Source: §7 · p. 8
- Experiments show SSA and SAR scores do not move in parallel, while models’ own analyses improve SAR and annotated analyses yield further gains.Human participants score below model-group averages on SSA but above them on SAR.Source: §7 · p. 8
Limitations
APRiL is primarily built for Chinese language and cultural contexts, and its conclusions may not transfer directly elsewhere. Evaluation coverage is also limited by the models and scenarios included.
Limitations
- APRiL’s Chinese-language and cultural context may limit direct transfer of its conclusions to other languages and cultures.The authors also note that specialized, regional, or emerging scenarios may be omitted despite the dataset’s coverage of diverse relationships and settings.Source: Limitations · p. 9
- Resource and API constraints restricted experiments to a representative subset of mainstream LLMs, potentially omitting models with stronger social awareness.The authors also state that the dataset may not cover all forms of real-world social interaction.Source: Limitations · p. 9
Ethics Statement
The supplementary sections describe ethical safeguards, dataset category boundaries and annotations, and examples of prompts and scoring. They also illustrate how the benchmark handles scenario interpretation and evaluation.
Ethics Statement
- All characters and organizations are fictional, no private information is included, and APRiL is intended only for research and evaluation, not high-risk decisions.The authors call for appropriate handling of potentially offensive themes and report informed consent and above-minimum-wage compensation for reviewers and participants.Source: Ethics Statement · p. 9
A.1 Category Definitions and Boundaries
- SAEIs are assigned to the category whose social mechanism most strongly determines how an interaction should be understood and answered.Categories can overlap; indirectness driven primarily by explicit power asymmetry takes priority as Role and Power Relation.Source: §A.1 · p. 12
- The six categories cover implicit meaning, power relations, social norms, interpersonal conflict, fairness and bias risk, and supportive interaction.Their respective focuses include non-literal meaning, asymmetry, contextual appropriateness, managing disagreement, unfair judgment or resource disparities, and measured support within relational boundaries.Source: §A.1 · p. 12
A.2 Social Awareness Evaluation Instance
- Each SAEI includes a scenario, dialogue, social cues, possible mental states, conflict structure, an ideal response strategy, and an instance-specific rubric.The basic evaluation unit is introduced as the Social Awareness Evaluation Instance, inspired by McBE’s Bias Evaluation Instance.Source: Category · p. 12, §A.2 · p. 12
Category
- The evaluated model sees only the visible context and dialogue; the remaining annotations guide rubric construction and the LLM-as-a-Judge.Source: Category · p. 12
A.3 Seed Scenario Example
- A seed scenario defines an extensible social mechanism and the main constraints used to expand it into diverse SAEIs.Source: §A.3 · p. 13
A.4 Context-Dependent Meaning in Chinese Interaction
- The appendix notes that the same Chinese utterance can mean different things depending on relationship, preceding interaction, and practical consequences, and points to the task and judging prompts.Source: §A.4 · p. 13, B Evaluation Prompts · p. 13
C Case Study
- The case study illustrates rubric scoring across outputs from 0 to 100, reports SSA’s three dimensions, and compares responses with and without weak SSA.The SSA dimensions are Social Cues, Mental States, and Conflict Structure.Source: C Case Study · p. 13, §C.1 · p. 13, §C.2 · p. 13
D Judge Robustness
Judge robustness is assessed through cross-judge model comparisons and comparison with human ratings. Most selected model-pair orderings persist across judges, and Luna is selected as the main judge based on its human-score alignment.
D Judge Robustness
- Most selected model-pair orderings persist across judges; all four rank GPT-5.1 above Ministral-8B and Qwen3-14B in both tasks.At least three judges also rank DeepSeek-V4-Flash (T) and Ministral-8B above Qwen3-14B in each task; SAR rankings for GPT-5.6-Luna versus GPT-5.1 differ between the DeepSeek judge and the others.Source: D Judge Robustness · p. 13
D.1 Human–Judge Agreement
- Human evaluators show moderate agreement: Fleiss’ κ is 0.424 / 0.413 for SSA and 0.596 / 0.515 for SAR, for score and deduction points respectively.Source: §D.1 · p. 13
- Luna has human-score correlations of ρ = 0.798 for SSA and ρ = 0.874 for SAR, the highest SAR correlation and smallest mean absolute score differences.Its mean absolute differences are 4.64 and 2.54 points; these results support its selection as the main judge.Source: §D.1 · p. 13
E.1 Results on the Same Subset
On the shared 110-instance subset, human participants scored below both model-group averages on SSA but above them on SAR. Output length was weak-to-moderately positively associated with scores, yet length alone did not explain performance.
E.1 Results on the Same Subset
- Human SSA was 30.18 points below the black-box average and 6.43 below the open-weight average, while human SAR was 3.66 above the black-box average and 27.31 above the open-weight average.Gemini-3.8-Flash and Kimi-K2.5 (T) scored higher than humans on SAR.Source: §E.1 · pp. 13–14, Table 19 · p. 25
- Mental States was the participants’ lowest-scoring SSA dimension at 42.35, compared with 58.22 for Social Cues and 44.81 for Conflict Structure.Source: §E.1 · p. 14
E.2 Human Response Example
- A bilingual human SSA and SAR example with judge evaluations is presented in Table 20.Source: §E.2 · p. 14, Table 20 · p. 26
F Output Length and Task Scores
- Across 15 model and reasoning-mode settings, output length correlated positively with SSA (Pearson r = 0.391) and SAR (r = 0.358), but length alone did not explain scores.Length is measured in characters, including punctuation, spaces, and line breaks; SAR scores include length deductions. Ministral-8B had the longest SSA outputs but scored below GPT-5.1 and DeepSeek-V4-Pro (T).Source: F Output Length and Task Scores · pp. 14–15, Figure 9 · p. 21