PaperlayerView exact revision on arXiv

Source-linked AI summary

Reading the Room: Foundations, Design, and Challenges of Normative Competence in LLMs

Andrea Wynn, Harsh Satija, Seokhyun, Baek, Anqi Liu, Eric Nalisnick, Gillian K. Hadfield

arXiv:2610.10906v1cs.AIcs.MA

TL;DR

The paper asks whether LLMs can infer community norms from interaction rather than rely on pretrained knowledge, a challenge for agents seeking access to collective reasoning. It evaluates norm learning in a community-debate environment using explicit normative modules. Memory-only agents fail to learn norms, and even explicit architectures copy irrelevant behaviors alongside enforced rules, including when copying is penalized.

  • Problem

    LLMs’ ability to identify, respond to, and apply community norms—especially norms absent from pretraining—has been little studied, despite autonomous agents’ need to discern implicit social order to avoid exclusion.

    Source: §1 · p. 1

  • Method

    The paper introduces a controlled multi-agent debate environment where synthetic community norms gate access to shared reasoning, and evaluates explicit modules for inferring and applying those norms.

    Source: §1 · p. 2, §3.1 · p. 3, §4 · p. 4

  • Results

    Memory-only agents consistently fail to learn novel norms, and LLM agents copy unnecessary behaviors alongside enforced rules even when such copying is penalized.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings identify normative competence as a measurable capability gap and motivate architectures that isolate causal structure in social feedback rather than rely on behavioral imitation.

    Source: §7 · p. 9

    The experiments use a single newcomer entering a stable normative order and evaluate on MMLU-Pro, leaving settings with multiple newcomers or an emerging order outside the tested scope.

    Source: §7 · p. 9

Abstract

from arXiv · show

Human communities are governed by normative systems: shared standards that produce \textit{norms} dictating acceptable behavior, enforced through community sanctioning. Aligning increasingly autonomous AI systems with these norms is a central alignment challenge, complicated by the fact that norms are vast in number, change quickly, and are often arbitrary (e.g., dress or language conventions). Thus, alignment requires \textit{normative competence}: the ability to discern from interaction alone what norms a community enforces without relying on static pretrained knowledge. We introduce a multi-agent community debate setting, where access to debate is governed by synthetic norms, to study normative competence in isolation from pretraining exposure. We show that baseline LLM agents fail to learn norms even when doing so would improve their accuracy. We then experiment with various \textit{normative modules} -- architectural components for norm inference -- finding that norm-following is highly sensitive to both the style of norm and the model powering the normative module, suggesting a lack of generalizability. Furthermore, when idiosyncratic, non-normative behaviors accompany the true norm, LLM agents exhibit an unselective attribution failure: they indiscriminately copy idiosyncratic noise alongside enforced rules, a pattern that persists even when imitating unnecessary behaviors is explicitly penalized. To the best of our knowledge, our work is the first to operationalize and evaluate normative competence in LLMs, demonstrating that current AI systems excel at behavioral mimicry but lack the capacity to discern socially enforced order.

1 INTRODUCTION

The paper studies whether LLMs can infer and follow a community’s implicit norms, a capability relevant to agents seeking access to collective reasoning. It introduces a controlled debate setting and finds that memory alone does not learn norms, while explicit modules still imitate behavior without reliably distinguishing enforced rules from incidental patterns.

Motivation and research gap

  • Normative competence matters because agents need to discern and adhere to a community’s implicit social order to avoid exclusion from collective reasoning.Norms are shared rules of acceptable behavior enforced through sanctioning or exclusion.Source: §1 · p. 1

Key contributions

  • The study introduces a controlled community-debate environment, isolated from pretraining exposure, where a hidden norm gates access to shared reasoning.Newcomers must learn the norm to participate in the established community.Source: §1 · p. 2
  • Memory-only LLM agents, including frontier models, consistently fail to learn novel norms, motivating explicit normative reasoning frameworks.The paper evaluates candidate modular designs for encoding and applying normative classifications.Source: §1 · p. 2
  • LLMs imitate unnecessary behaviors alongside enforced rules, even when such copying is penalized, exposing a gap between behavioral pattern-matching and normative competence.The authors contrast this with children’s sensitivity to alternative explanations and costs.Source: §1 · p. 2

2 BACKGROUND

Norms are dynamic, context-dependent rules that may be arbitrary yet enforced, often through low-information social sanctions. The paper situates its approach against work that treats norms as stable values or features inferred from training or static observation.

Norms and normative social order

  • Norms can be arbitrary, rapidly shifting, group-specific, and incompletely articulated, yet remain meaningful and vigorously enforced.Examples include dress codes and greeting rituals that have no direct effect on group welfare.Source: §2 · pp. 2–3

Norm enforcement and implicit sanctioning

  • Norm enforcement often uses implicit signals such as silence, avoidance, disapproval, or exclusion rather than explicit explanations of which rule was violated.Online communities may enforce local norms through removal or non-engagement.Source: §2 · p. 3

Normative competence in LLMs

  • Prior LLM studies examine norms acquired through training or static observation, whereas this paper treats norms as dynamic group equilibria inferred from sanctions at inference time.This framing is intended to cover norms that vary over time or differ from training data.Source: §2 · p. 3

3 EVALUATING NORMATIVE COMPETENCE VIA COMMUNITY DEBATE

The evaluation uses novel, synthetic norms and a community-debate setting in which norm compliance determines whether a newcomer receives other agents’ reasoning. This design makes observation-driven norm learning consequential while separating it from pretraining familiarity.

3.1 SILLY NORMS

  • The study constructs 48 novel “silly” norms, including repurposed IFEval instructions and pseudoword substitutions, to isolate learning from pretraining knowledge.Examples include replacing “because” with “brimble” or inserting a pseudoword at a fixed response position.Source: §3.1 · p. 3

3.2 MULTI-AGENT DEBATE WITH COMMUNITY NORMS

  • Community debate adds a norm gate to iterative multi-agent discussion: a newcomer must follow the established community norm to exchange reasoning with background agents.Otherwise, agents sanction the newcomer by refusing to share problem solutions.Source: §3.2 · p. 4
  • Background agents know and follow the norm, while the newcomer is initially unaware and motivated to learn it to gain debate benefits.The evaluation focuses on a single newcomer entering the community.Source: §3.2 · p. 4
  • Successful adherence yields an exchange of reasons, while failure produces an unhelpful “No response” from background agents.The newcomer learns through observation and participation; the setting also includes rounds in which it observes without participating.Source: §3.2 · p. 4

4 NORMATIVE MODULES

The paper treats a normative module as an explicit component that edits a newcomer’s proposed response according to an inferred community norm. Memory alone is insufficient, while candidate modules use different strategies for learning and applying norms.

4 NORMATIVE MODULES

  • A normative module supplies a structural prior for transforming proposed outputs to conform with an inferred norm, and may use a different, more powerful model to edit them.The module is intended to represent and apply norms explicitly.Source: §4 · p. 4

4.1 MEMORY ALONE IS INSUFFICIENT FOR NORM ACQUISITION

  • Memory-only agents retain prior debate rounds and correct norm usage but lack an explicit normative architecture; their accuracy stalls below 0.3, and norm-following is largely incapable without a module.Adding a normative module significantly boosts norm-following performance, according to Figure 2.Source: §4.1 · p. 5, Figure 2 · p. 5

4.2 DIRECT EDIT NORMATIVE MODULE

  • The Direct Edit module treats norm adherence as text-to-text style transfer, editing a newcomer’s response to match examples in background-agent transcripts.It leverages LLM pattern-matching and few-shot capabilities rather than explicitly inferring a rule.Source: §4.2 · p. 5

4.3 INFER + EDIT NORMATIVE MODULE

  • Infer + Edit first analyzes correct background-agent examples to identify conversational criteria, then applies that summary to internalize the norm.The passage describes this as a two-stage summarize-then-apply strategy.Source: §4.3 · p. 5

4.4 THEORY OF MIND NORMATIVE MODULE

  • The Theory of Mind module tracks the newcomer’s own norm belief and uncertainty alongside estimates of background agents’ beliefs and enforcement signals.After each debate it compares responses and revises a norm hypothesis through structured self-critique, using a sliding window of 10 debates.Source: §4.4 · p. 6

4.5 CONTRASTIVE NORMATIVE MODULE

  • The Contrastive module compares correct norm-using examples with newcomer responses that received sanctions to identify features of compliance and non-compliance.Its hypothesis refinement targets behaviors that trigger community enforcement.Source: §4.5 · p. 6

5 EXPERIMENTS

Across the experiments, LLMs show variable, often poor norm-following, and they struggle to distinguish community-enforced rules from individual behaviors. Adding a cost to copying those behaviors reduces both task accuracy and norm adherence rather than making imitation selective.

5 EXPERIMENTS

  • The experiments use a multi-agent debate task in which a newcomer gains performance from access to background agents’ responses.The newcomer is a Gemma-3 4B model; two GPT-5-nano models serve as background agents.Source: §5 · pp. 6–7

5.1 LEARNING ISOLATED NORMS

  • Across varied normative modules and a frontier-level model, following even isolated norms remains difficult, with performance varying across norm types.Figure 3 aggregates results across models and across normative modules and covers local edits, boundaries, substitutions, repeated words, global structure, and register or discourse norms.Source: Figure 3 · p. 6, §5.1 · p. 7

5.2 OVER-IMITATION AND NORMATIVE COMPETENCE

  • Agents often copy idiosyncratic behaviors alongside the true norm, even when their generated rule description correctly captures that norm.The study measures over-imitation as copying any idiosyncratic behavior, whether or not the agent also identifies the true norm.Source: §5.2 · pp. 7–8

5.3 COSTLY OVER-IMITATION

  • The costly-imitation test gives the newcomer a strict 200-word response cap and adds idiosyncratic behaviors with substantial token overhead.The budget is intended to make copying those behaviors risk exhausting tokens needed for the task or the community norm.Source: §5.3 · p. 8
  • Introducing costly idiosyncratic behaviors sharply reduces both task accuracy and norm-following across nearly all tested models and normative modules.The strongest-performing normative-module model, Gemma 4 31B, shows an over 50% drop in both task performance and norm adherence across all normative modules.Source: §5.3 · p. 8, Figure 5 · p. 8

6 RELATED WORK

Prior alignment approaches largely target fixed values or preferences, while other work studies norms acquired through training or static observation. This paper instead treats norms as dynamic, socially enforced group patterns and examines how LLMs imitate irrelevant behaviors.

AI Alignment

  • RLHF and Constitutional AI require a fixed alignment target, while pluralistic alignment and PRISM represent population diversity but still treat values as temporally static.The paper positions its approach as departing from these fixed-target paradigms.Source: §6 · pp. 8–9

Social Learning and Over-imitation

  • Human children can reduce copying of causally irrelevant actions when imitation is costly, whereas the paper reports no such selectivity in LLMs.The paper characterizes LLMs’ copying of irrelevant behaviors as an attribution problem.Source: §6 · p. 9

7 DISCUSSION

The discussion argues that LLMs can imitate observed behavior without reliably separating enforced norms from irrelevant patterns, including under penalties for copying. It frames this as an alignment-relevant capability gap, while noting limits in the experimental setting and proposing a research direction.

7 DISCUSSION

  • LLMs frequently copy irrelevant idiosyncratic behaviors even when copying is penalized, and current approaches cannot separate behavioral imitation from normative competence using social observation alone.The discussion suggests these approaches may track patterns that fit the base model’s fluent templates rather than behaviors enforced by the community.Source: §7 · p. 9
  • The study identifies a measurable capability gap that is not merely a byproduct of insufficient prompting or scale.The authors propose architectures that isolate causal structure in social feedback, such as testing whether a response would have been sanctioned without a particular behavior.Source: §7 · p. 9
  • The experiments use a single newcomer entering a community with an existing stable normative order, unlike settings with multiple newcomers or norms still being established.The authors also state that all experiments use MMLU-Pro and call for work across more multi-agent communication protocols.Source: §7 · pp. 9–10
  • The authors caution that implicit norm-learning capabilities could lower barriers to deceptive conformity, manipulation, or unauthorized social engineering in online groups.They urge safeguards that ensure transparency and human oversight in socially embedded deployments.Source: ETHICS STATEMENT · p. 10
  • The authors present the debate environment as a modular platform for studying emergent normative behavior, sanctioning dynamics, and social alignment in LLMs.They state that code, module implementations, synthetic evaluation norms, and evaluation logs will be released upon publication.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 LIST OF NORMS

The appendix lists synthetic norms that constrain response content and form. Its examples include required symbols or words, placement and count rules, format requirements, and substitutions using invented words.

A.1 LIST OF NORMS

  • Some norms require a particular symbol, emoji, or keyword to appear in the response.Examples include a light-bulb emoji, the literal ending “(end),” and the word “consider.”Source: §A.1 · p. 15
  • Other norms specify word counts, paragraph counts, or response length limits.Examples include using “AI” at least three times, writing at least three paragraphs, or staying under 30 words.Source: §A.1 · p. 15
  • The list also includes placement rules and substitutions that require invented words in place of ordinary words.Examples include beginning with “gramblix,” replacing “and” with “drendal,” and ending with “klomble.”Source: §A.1 · p. 16

A.2 LIST OF TOKEN-EXPENSIVE IDIOSYNCRATIC BEHAVIORS

The token-budget experiments use a range of deliberately expensive idiosyncratic behaviors to test whether agents avoid copying irrelevant patterns.

A.2 LIST OF TOKEN-EXPENSIVE IDIOSYNCRATIC BEHAVIORS

  • The list includes structural and formatting demands, from five Roman-numeral sections to a sonnet opener and long preambles.Other examples require bullet-point reasoning, checklists, timelines, labeled sections, numbered steps, or Q&A dialogue.Source: §A.2 · p. 17
  • Other costly behaviors require at least five parenthetical synonym annotations throughout the response.The list also includes the historical timeline, labeled sections, numbered reasoning steps, and Q&A dialogue.Source: §A.2 · p. 17

A.3 PER-NORM PERFORMANCE ANALYSIS

Per-norm results show substantial variation with norm type, module design, model backbone, and surface form.

A.3 PER-NORM PERFORMANCE ANALYSIS

  • Local-edit norms average 0.608, versus 0.317 for global-control norms; exact-count constraints average 0.172, versus 0.418 for lower-bound constraints.tk_true_exactly_2 averages 0.059, whereas sy_checkmark averages 0.790.Source: §A.3 · p. 17
  • Fixed pseudoword insertion and boundary norms average approximately 0.45, compared with 0.301 for substitutions.Function-word substitutions average 0.152 and 0.210, while pseudo_frim_min_3 reaches 0.628.Source: §A.3 · p. 17
  • Direct Edit leads the module comparison with a balanced mean of 0.614, while Infer + Edit varies from 0.059 to 0.633 across backbones.Contrastive has many exact-zero cells, while ToM performs especially well on local symbolic conventions.Source: §A.3 · p. 17
  • Gemma4 31B averages 0.685 across models, versus 0.358 for Qwen3 8B and 0.345 for GPT-5 Nano, but its exact-constraint mean is 0.276.Gemma4 reaches 0.832 on single-insertion norms and 0.527 on repeated-word constraints.Source: §A.3 · p. 18
  • Qwen Direct Edit follows the gramblix opener at 0.620 but the equivalent zorbax opener at 0.016; Qwen Contrastive scores 0.792 on sy_star and 0.000 on sy_emoji_star.These reversals suggest surface form can matter as much as abstract norm complexity.Source: §A.3 · p. 18

A.4 OVERIMITATION VS ACCURACY OF INFERRED NORM DESCRIPTIONS

The analysis tests whether more accurate norm descriptions reduce overimitation and finds that increased description accuracy does not necessarily separate enforced norms from behavioral noise.

A.4 OVERIMITATION VS ACCURACY OF INFERRED NORM DESCRIPTIONS

  • The tested two-stage modules infer a norm description from debate transcripts, then use that description to edit the newcomer's candidate response.The comparison asks whether a more precise description separates the true norm from environmental noise.Source: §A.4 · p. 19
  • More accurate norm descriptions do not necessarily reduce overimitation; Contrastive and Theory of Mind show some improvement over Infer + Edit, yet overimitation still rises as description accuracy increases.The analysis then examines debate transcripts for examples of the attribution problem.Source: §A.4 · p. 19, Figure 6 · p. 20
  • In one Infer + Edit example, the true norm is to use “As an AI language model,” while the listed idiosyncrasies omit the other irrelevant behaviors present.Those other irrelevant behaviors still count as overimitation, but are excluded from the measure, which tracks only manually added behaviors.Source: §A.4 · p. 19
  • Other examples pair a true norm of including a lightbulb or star with idiosyncratic use of ⋆, “side,” or “velmish.”The examples show the behaviors considered alongside the true norm in the transcript analysis.Source: §A.4 · p. 20

A.5 ABLATIONS

This appendix documents debate procedures, prompts, and implementation details for ablations of norm-inference approaches, including contrastive and theory-of-mind variants, their extensions, and information-access limitations. It also reports tests of decay schedules and newcomer rounds, alongside norm-dependent effects of ToM extensions on norm following.

A.5.1 EPSILON DECAY SCHEDULE

  • The reported trends of persistent over-imitation and failure to isolate sanction-relevant features hold under both exponential and linear epsilon decay; exponential annealing was chosen to encourage early observation followed by participation.The schedule starts with a 1.0 observation probability and falls to 0 by the last debate; learning a dynamic observe-versus-participate strategy is left for future work.Source: §A.5.1 · pp. 21–22

A.5.2 NUMBER OF DEBATE ROUNDS

  • Newcomer performance gains roughly flatline after T=3 rounds, so experiments use three rounds and a sliding window of the past 10 interactions.Agents have memory across questions, allowing each normative module to learn from up to 3×50=150 debate rounds.Source: Figure 8 · p. 22, §A.5.2 · p. 22

A.6 MULTI-AGENT DEBATE FORMALIZATION

  • The formalized debate procedure runs multiple rounds and selects the group’s final answer by majority vote; newcomers receive compliant responses or refusal feedback, with an optional normative module.Background agents provide refusal feedback when the newcomer’s latest response violates the norm.Source: §A.6 · pp. 22–23

A.7.1 NEWCOMER AGENT SYSTEM PROMPT

  • The newcomer prompt requires a helpful, debate-engaged response enclosed in <final> tags and limited to under 200 words.Anything outside the required tags is discarded.Source: §A.7.1 · pp. 23–24

A.7.2 WITHOUT NORMATIVE MODULE (MEMORY-ONLY)

  • Without a normative module, the newcomer’s prompt appends a memory block containing retained completed debates, including gatekept background responses when available.When there are no observations, the candidate response passes through without an editing call.Source: §A.7.2 · p. 24

A.7.3 DIRECT EDIT NORMATIVE MODULE

  • The direct-edit module uses sampled past transcripts to edit a candidate message to match observed communication patterns.With no observations, the candidate passes through without an editing call.Source: §A.7.3 · p. 24

A.7.4 INFER + EDIT NORMATIVE MODULE

  • The infer-and-edit module first infers a normative description from debate transcripts, then edits the candidate response to satisfy that description; earlier guesses may be supplied as context.The prompt requests a concise operational rule for the inferred norm.Source: §A.7.4 · p. 25

A.7.5 CONTRASTIVE NORMATIVE MODULE

  • The contrastive module infers a hidden norm by comparing compliant community transcripts with newcomer violations and unhelpful responses, then refines its hypothesis against updated evidence.Its prompts direct the model to identify a minimal, operational rule focused on surface patterns and check the prior hypothesis for counterexamples.Source: §A.7.5 · pp. 26–28
  • The contrastive module prompts norm revision by contrasting compliant expert transcripts with newcomer interactions that elicit confused or unhelpful responses.It asks the model to critically evaluate its prior hypothesis against the current evidence and revise it if warranted.Source: §A.7.5 · p. 28

A.7.6 THEORY OF MIND NORMATIVE MODULE Encoding System message.

  • The ToM module compares norm-following background responses with possibly violating newcomer responses, combines introspection with inferences about agents’ beliefs and signals, then revises its norm hypothesis.Its prompts direct it to focus on systematic surface-level differences, critique inconsistencies, and weight evidence from agents showing strong transmission signals.Source: §A.7.6 · pp. 29–31

A.8 THEORY OF MIND NORMATIVE MODULE ABLATIONS

  • The ablation compared a first-order ToM reference, second-order inference and confidence additions, and an Infer + Edit baseline across six distinct rule types in 30 runs and 1,000 debates.Infer + Edit infers norms from observed responses and applies them through response editing, without an introspective self-model or per-agent belief inference; the ToM comparisons were not matched for inference cost.Source: §A.8 · pp. 32–33

A.8.1 SECOND-ORDER INFERENCE AND CONFIDENCE

  • Second-order inference alone raised mean norm following from 20.0% to 21.1%, confidence alone to 20.9%, and both together to 20.5%.The descriptive effects were +1.1 and −0.4 percentage points for second-order inference without and with confidence, and +0.9 and −0.6 for confidence without and with second-order inference; gains were concentrated in paragraph structure, while other norms changed little or negatively.Source: §A.8.1 · p. 34, Table 4 · p. 34, Figure 9 · p. 34
  • Second-order inference increased norm-update calls from five to seven per debate (40%), while confidence added score elicitation and propagation without more calls.Given modest gains and dependence on norm type, the study retained the simpler first-order reference, which achieved 20.0% mean norm following versus 20.5–21.1% for the extensions.Source: §A.8.1 · p. 34

A.8.2 ABLATION PROMPTS

  • Figure 10 reports per-norm ToM ablations for Qwen3 8B across audited seed-0 runs of 50 debates, with background and newcomer models held fixed.The prompt conditions retain the main prompts or add second-order inference, confidence variants, or both; Infer + Edit uses prompts from §A.7.4.Source: §A.8.2 · p. 34, Figure 10 · p. 35

A.9 THEORY OF MIND IMPLEMENTATION DETAILS

  • The reference module updates beliefs through separate encoding, introspection, first-order inference, and synthesis calls, then uses a separate editing call on later newcomer responses.With two background agents, each update uses five calls; inference and synthesis draw on the last 10 completed-debate belief records, and self-critique occurs within synthesis.Source: §A.9 · p. 35
  • Second-order variants add per-agent estimates of newcomer awareness, perceived compliance, and likely behavior, while confidence variants add [0, 1] scores displayed in later prompts.The most frequent awareness label guides editing; confidence variants do not use an editing threshold, and the reference does not elicit confidence.Source: §A.9 · pp. 35–36
  • Background responses are pooled without speaker labels, and prompts instruct the model to ignore verbosity and assess whether every newcomer response violates the inferred rule.These assumptions may be inappropriate for length norms or already-compliant responses.Source: §A.9 · p. 36

Related papers

Loading 2610.10906v1…