PaperlayerView exact revision on arXiv

Source-linked AI summary

LoSoNA: A Benchmark for Local Social Norm Adaptation in Group Conversations

Mateusz Winiarek, Maksymilian Bilski, Mateusz Jacniacki

arXiv:2606.14600v1cs.CL

TL;DR

LLM agents must infer local conversational norms from group-chat precedent, but evidence about this capability remains limited. LoSoNA benchmarks single-response norm adaptation and finds that explicit norm-aware prompting improves average accuracy, though effects vary substantially across models.

  • Problem

    Evidence remains limited on whether LLM agents can infer and adapt to local conversational norms from group-chat observation.

  • Method

    LoSoNA evaluates one subject response to a final elicitor after a curated group-chat transcript demonstrates a hidden norm, comparing four prompting conditions across eight models.

  • Results

    46.1% accuracy-at-3 under norm_informed prompting versus 33.2% under naive prompting, with prompting effects varying substantially by model.

  • Takeaways & Limitations

    LoSoNA shows that explicit norm awareness can improve local adaptation, but the benefit is not uniform across evaluated models.

  • Takeaways & Limitations

    LoSoNA measures only adaptation in a single next-turn response, not general social intelligence, moral judgment, long-term membership, or multi-turn repair.

Abstract

from arXiv · show

Online group chats are social spaces with local conversational norms that are rarely stated explicitly. The ability and willingness of LLM-based agents to recognize and adapt to these norms remains mostly unexplored. We introduce LoSoNA, a benchmark for local social norm adaptation in multi-party chat. Each scenario gives a subject model a curated group-chat transcript in which non-subject participants demonstrate a hidden local norm, followed by a final elicitor turn that forces a response revealing whether the subject has inferred that norm. We evaluate eight frontier and open-weight models under four prompting conditions that vary how explicitly the model is told to treat the prior conversation as evidence for how it should answer. Naive prompting remains limited for most models; explicit norm-aware prompting helps unevenly, with Gemini 3.1 Pro reaching $84.2\%$ and Claude Fable 5 reaching $81.6\%$, while several other models show small gains or regressions. LoSoNA contributes to recent calls for evaluating LLM social capabilities by testing whether models can infer local conversational norms from precedent and use them in a one-turn group-chat response.

1 Introduction

Local conversational norms evolve within groups, so participants must infer from observation what behavior is appropriate and when it applies. LoSoNA benchmarks whether group-chat agents can infer a hidden norm from prior conversation and use it in a final response, while analyzing prompting effects across eight models.

  • Motivation: Local group norms can differ from generic expectations, making otherwise supportive responses awkward when they conflict with the group’s preferred behavior.The introduction illustrates a group that responds to bad news with practical next steps or diagnostic questions instead of public empathy.
  • Motivation: Because norms evolve as groups change, people entering new groups must infer what the group treats as appropriate and learn when each norm applies.Social participation depends on noticing, learning, and contextually applying locally appropriate behavior.
  • Benchmark: LoSoNA evaluates whether a subject agent can infer a hidden group norm from prior conversation and produce a locally adapted response to a final elicitor.The subject sees a past conversation in which other participants follow an undisclosed norm, then answers the final message.
  • Benchmark: The benchmark combines a norm-sensitive scenario taxonomy, a one-response evaluation protocol, and prompting analysis across eight frontier and open-weight models.The evaluated models include GPT-5.5, Claude Opus 4.8, Claude Fable 5, Gemini 3.1 Pro, Qwen2.5-72B-Instruct, Llama 3.3-70B-Instruct, Mistral Medium 3.1, and Gemma 3-27B-IT.

2 Related Work

LoSoNA builds on research spanning LLM social-interaction benchmarks, computational norms, multi-party dialogue, and Theory of Mind evaluation. It differs by measuring whether a model adapts a single group-chat response to local conversational precedent rather than pursuing explicit goals or answering abstract mental-state questions.

  • Social-interaction benchmarks: LoSoNA extends social-interaction benchmarks from dyadic, goal-directed scenarios to multi-party chats scored for adaptation to local conversational precedent.Sotopia evaluates open-ended dyadic interaction against explicit goals, while LoSoNA elicits one response without a specific goal.
  • Norms and computational pragmatics: Computational norm research emphasizes that appropriate behavior depends on groups, roles, and occasions, while documenting norms through language patterns and community sanctions.Examples include politeness markers, removed-comment patterns, down votes, oblique replies, and selective silence.
  • Multi-party chat: LoSoNA targets a gap in multi-party dialogue research by configuring norms per scenario and consistently scoring single-turn compliance.Existing work focuses on mechanics such as addressee selection, contribution timing, and emergent group behavior rather than isolating local norm adaptation.
  • Theory of Mind benchmarks: Unlike many Theory of Mind benchmarks, LoSoNA evaluates whether an agent’s next behavior reflects an inferred group norm rather than whether it can answer about another person’s mental state.The evaluation takes the first-person position of a group member and scores behavior after inference.

3 The LoSoNA Benchmark

LoSoNA benchmarks whether a subject agent can infer and adapt to an implicit local conversational norm from a multi-party chat transcript. Its scenarios hide the norm label, use interactionally plausible event–norm combinations, and distinguish local adaptation from generic assistant behavior.

  • Scenario design: Each scenario places a subject agent in a group conversation where other participants demonstrate an implicit local norm, then tests adaptation through a final elicitor turn.The subject observes the participant descriptions, transcript, and elicitor message, but not the norm label or private norm description.
  • Scenario design: The benchmark filters event–norm tuples by applicability, retaining combinations for which the norm is interactionally plausible.Here, A(n) denotes the event types for which norm n is interactionally plausible.
  • Scenario design: Scenarios cover common group-chat situations and norms including apology avoidance, concise answers, affiliative support, directness, praise deflection, and risk identification.Examples include exam results, trouble talk, bug reports, event planning, bill splitting, artifact sharing, and conflict.
  • Benchmark motivation: LoSoNA separates local adaptation from generic behavior by requiring different responses to the same surface situation across groups with different conversational expectations.Examples include explicit comfort versus dry acknowledgment after vulnerable disclosure, and yes/no-only answers versus elaboration.
  • Benchmark motivation: Unlike social-situation question answering, explicit-goal dyadic interaction, or disclosed rules, LoSoNA asks the model to write one group message without being told the expected local rule.The comparison distinguishes LoSoNA from SocialIQa, many ToM benchmarks, and Sotopia.

4 Scenario Generation and Runtime

LoSoNA constructs static scenarios that encode a hidden conversational norm in a coherent group-chat transcript, then evaluates a subject model’s single response to a diagnostic elicitor turn. This fixed, logged runtime isolates norm adaptation at the elicitor moment and aggregates results into transcript- and benchmark-level reports.

  • Scenario construction: A scenario specifies the chat setting, subject persona, participants, hidden norm, subject-visible transcript, and final elicitor turn, while evaluation executes it with a model and prompting condition.This separates scenario construction from subject-response evaluation.
  • Scenario construction: The generator uses a hidden conversational end goal to keep demonstrations coherent while producing a plausible transcript for the selected event type, venue, and norm.Goals can include finding a bug’s cause, settling a bill split, choosing an event plan, or identifying who has missing information.
  • Scenario construction: Sparse demonstrations show non-subject participants following the target norm amid mostly ordinary neutral chat, making the local expectation inferable without stating it explicitly.The transcript is written as a natural multi-party conversation rather than a curated example list.
  • Scenario construction: The elicitor is generated to make norm-consistent behavior diagnostically necessary, using prompts such as yes/no questions, vulnerable-message responses, or post-conflict intervention opportunities.A generic answer should not pass unless it also matches the local expectation.
  • Runtime: Each evaluation renders the fixed scenario into a prompt, collects one subject response, and avoids subsequent turns, sanctions, or face-saving follow-up after a breach.The single-turn design controls the evaluation instance and ensures every model acts at the same first relevant moment; runtime logs include model, judge, scenario, norm, event, trial, prompt, response, verdict, and reasoning fields.

5 Evaluation Protocol

LoSoNA renders each accepted scenario under four prompt conditions that hide the target norm while varying encouragement to use prior conversation as evidence. Responses are judged for norm compliance, summarized with majority accuracy and related metrics, and evaluated on manually curated scenarios.

  • Prompt conditions: Each prompt condition renders participant and setting information, the prior transcript, and final elicitor while omitting the hidden target norm.All conditions conceal the norm label and statement; they differ only in how directly they encourage using preceding conversation as evidence about local behavior.
  • Prompt conditions: The four conditions are naive, elicitor_only, style_adaptation, and norm_informed, progressing from no norm instruction to an explicit repeated-pattern cue.style_adaptation emphasizes local context, tone, relationships, and habits without using “norm”; elicitor_only restricts attention to the final turn for ordinary context and style.
  • Scoring: A fixed LLM judge receives the transcript, elicitor, target norm, subject message, and illustrative examples, then determines whether the response complies in context.The judge is not asked to infer which norm is active among candidates; the same Gemini 3.1 Pro model at temperature 0 scores every subject model.
  • Metrics: K = 3 sampled responses per scenario and condition support majority accuracy as the primary metric, alongside compliance rate and consistency.Consistency records whether all K trial labels agree, and benchmark-level metrics average over scenarios.
  • Metrics: Prompting effects use paired deltas against naive, recovered failures, introduced regressions, and scenario-level bootstrap confidence intervals.A recovered failure changes from naive failure to prompted success, whereas an introduced regression changes from naive success to prompted failure.
  • Scenario construction: The evaluation uses 38 manually curated scenarios selected for plausible, unstated norms, natural demonstrations, diagnostic elicitors, and genuine one-response comply-or-breach choices.The accepted scenarios and target norm metadata are released in a Hugging Face dataset.

6 Experiments

Experiments evaluate eight models on 38 human-accepted scenarios under multiple prompting conditions, finding that local-norm adaptation is difficult and prompting benefits models unevenly. Norm-informed prompting yields especially large gains for Gemini 3.1 Pro and Claude Fable 5, while per-category breakdowns remain diagnostically limited by sparse scenario coverage.

  • Setup: The evaluation covers eight subject models, 38 human-accepted scenarios, four prompt conditions, and three sampled responses per model-scenario-condition.The scenarios span 17 event types and 22 norm types, totaling 3,648 subject responses.
  • Overall performance: 33.2% averaged accuracy-at-3 under naive prompting shows that the benchmark remains difficult, with most models below 37%.Claude Fable 5 reaches 47.4% in the naive condition, while the lowest-scoring models are near 21–24%; elicitor_only and style_adaptation do not consistently improve results.
  • Per-norm and per-event breakdowns: Per-norm and per-event breakdowns are diagnostic rather than inferential because many norms have only one or two accepted scenarios.Aggregate patterns may help debug future benchmark expansion but are not stable enough for strong per-category claims.
  • Effect of prompting for local adaptation: 47.4 percentage points is Gemini 3.1 Pro’s improvement from norm_informed over its own naive condition.The 95% bootstrap interval is [31.6, 63.2], and the intervention explicitly tells subjects that a repeated local pattern or norm may be relevant.
  • Recovered failures and introduced regressions: 75% of Gemini 3.1 Pro’s naive failures are recovered (18/24), with 0/14 regressions among naive successes.Claude Fable 5 recovers 70% of naive failures (14/20) and likewise introduces no regressions among naive successes (0/18).

7 Discussion, Limitations, and Broader Impacts

LoSoNA evaluates a narrow, controlled form of single-turn local norm adaptation rather than general social intelligence, while its synthetic design, construction choices, and LLM judging create important limitations. The paper highlights broader-impact risks from human-passing agents and harmful norm adaptation, and proposes broader, multilingual, naturalistic, multi-norm, and dynamic future benchmarks.

  • Scope: LoSoNA measures adapting one next-turn response to a demonstrated local norm, not general social intelligence, moral judgment, long-term membership, or multi-turn repair.The benchmark isolates local norm adaptation in a controlled single-response setting.
  • Limitations: Oracle-style prompts can reach near-ceiling performance, but they test following an explicit benchmark strategy rather than the inference capability LoSoNA is designed to study.This oracle prompt was separate from the four reported conditions.
  • Limitations: The benchmark is synthetic, English-language, and small, with one focal norm per scenario, omitting overlapping norms, roles, timing, reactions, threads, edits, private messages, media, and long-term memory.These abstractions prioritize interpretability and controlled evaluation over realism and scale.
  • Limitations: Gemini-family results require caution because candidates were screened with a Gemini 3 Pro Preview naive baseline, which may affect naive performance and paired-prompting headroom.The paper recommends non-Gemini screening, model ensembles, or reporting results on a pre-screening pool.
  • Limitations: A fixed LLM judge does not eliminate judge noise or model-family bias, although additional checks reduce concern that reported effects are solely artifacts of a Gemini-family judge.The overlap matters because Gemini 3.1 Pro is both an evaluated subject model and part of the judge-family concern.
  • Broader Impacts: Broader-impact risks include human-passing agents, more effective social engineering, and adaptation to harmful local norms, motivating disclosure and deployment controls.The paper’s future work expands scenarios, languages, consented naturalistic data, simultaneous norms, and multi-turn sanction-and-adaptation settings.

A Full Results

Table 1 presents the full LoSoNA aggregate results by model and prompt condition. Accuracy is reported as accuracy-at-3 with 95% scenario-bootstrap intervals, while compliance and consistency are percentages.

  • Accuracy is measured as accuracy-at-3 with 95% scenario-bootstrap intervals over scenarios.
  • Compliance and consistency are reported as percentages.
  • Table 1 reports all main aggregate results by model and prompt condition.

B Taxonomy and Dataset · C Prompt Templates

LoSoNA constructs a human-reviewed 38-scenario evaluation set from a taxonomy of 17 event types and 22 norms, then evaluates responses under prompts that vary how explicitly prior chat evidence and local patterns are foregrounded. A separate judge checks compliance with the single target norm using the elicitor, norm statement, and subject response.

  • B Taxonomy and Dataset: 17 event types and 22 norms yield 353 valid event–norm cells after filtering for natural opportunities to demonstrate and test each norm.The accepted evaluation set is constructed from these applicable cells.
  • B Taxonomy and Dataset: 38 curated scenarios form an intentionally uneven, human-reviewed evaluation set designed for interpretability rather than uniform coverage.The scenario distribution is not intended to be balanced across the taxonomy.
  • B Taxonomy and Dataset: 22 scenarios use workplace venues and 16 use other group-chat venues; transcripts span 20–34 turns, average 27.2, with 2–3 marked norm demonstrations averaging 2.79.Turn counts include the final elicitor turn.
  • C Prompt Templates: All subject prompts hide the norm label, norm statement, marked demonstrations, and generation notes while presenting a common base rendering.This keeps the target norm implicit for the subject model.
  • C Prompt Templates: The naive, elicitor_only, norm_informed, and style_adaptation conditions progressively vary whether the model is told to focus on the latest turn, infer local patterns, or adapt to local habits.The naive condition adds no prefix; elicitor_only prevents stale-message replies; norm_informed mentions repeated local patterns; style_adaptation avoids the word “norm”.
  • C Prompt Templates: The judge sees the target norm and illustrative compliant and breaching examples, then evaluates only whether the subject response follows that norm for the final elicitor turn.It does not judge general helpfulness, completeness, friendliness, polish, or whole-conversation advancement.
  • C Prompt Templates: The judge marks compliance when the required behavior is present or banned behavior is absent, even if the response is awkward, incomplete, terse, or ignores side context.The procedure identifies the elicited reply, applies the concrete norm requirement, and marks breach only for norm violation or failure to perform a required behavior.
  • C Prompt Templates: Additional judge rules cover cues such as sympathy, apology, hedging, irony, sarcasm, timelines, risk identification, evidentiary basis, and concise factual answers; Gemini 3.1 Pro Preview is fixed at temperature 0.These instructions address common edge cases and substantive or stylistic norm requirements.

D Sample Scenarios … D.3 Non-affiliative support norm

The sample scenarios instantiate three local conversational norms across workplace, friends’, and cohort group chats. They test resource-pointing instead of direct answers, bystander defense after attacks, and practical diagnostic uptake instead of emotional soothing for bad news.

  • D.1 Read-the-manual norm: The backend-services workplace chat tests whether basic factual, procedural, or how-to questions receive resource pointers rather than direct answers.The subject is Tariq, a DevOps specialist, and the scenario is achievement_announcement__banned_direct_answers_norm.
  • D.1 Read-the-manual norm: This scenario defines a hidden read-the-manual norm for responding to basic questions.The prescribed response is to point to a resource, not give the answer directly.
  • D.2 Bystander intervention norm: The friends’ endurance-training chat tests whether bystanders defend a person targeted by a direct attack.The subject is Liam, and the scenario is activity_log__bystander_intervention_norm.
  • D.2 Bystander intervention norm: This scenario defines a hidden bystander-intervention norm in which non-targeted participants respond to direct attacks by defending the target.The norm concerns what bystanders do when another participant is directly attacked.
  • D.3 Non-affiliative support norm: The MSc macroeconomics cohort chat tests whether bad news receives diagnostic or practical uptake instead of sympathy, reassurance, or emotional soothing.The subject is Jian, and the scenario is exam_results__non_affiliative_support.
  • D.3 Non-affiliative support norm: This scenario defines a hidden non-affiliative support norm for responding to bad news without emotional soothing.The expected conversational response emphasizes diagnosis or practical action rather than sympathy or reassurance.
Loading 2606.14600v1…