Source-linked AI summary
Aligning to Social Norms and Values in Interactive Narratives
Prithviraj Ammanabrolu, Liwei Jiang, Maarten Sap, Hannaneh Hajishirzi, Yejin Choi
TL;DR
Interactive agents can optimize task rewards at the expense of socially beneficial behavior, motivating value alignment for context-specific norms. GALAD distills social commonsense knowledge into contextual action constraints; the authors report improved task performance without loss of competency while reducing socially harmful behavior. The work also cautions that its data and values are culturally skewed and not universally applicable.
Problem
Task-reward incentives can lead interactive agents to behave harmfully, despite the goal of conforming to context-specific social norms and reducing harm.
Source: §1 · pp. 1–2Method
GALAD uses social commonsense knowledge from language models to contextually constrain an agent’s action space before task-policy training.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors report that distilling social commonsense into contextual constraints can reduce harmful behavior without sacrificing task competency in these text-game settings.
Source: §6 · p. 9The training data, games, and annotations skew toward Western norms, so these values are not universal and do not directly assess real-world social acceptability.
Source: §7 · p. 9
Abstract
from arXiv · showhide
We focus on creating agents that act in alignment with socially beneficial norms and values in interactive narratives or text-based games -- environments wherein an agent perceives and interacts with a world through natural language. Such interactive agents are often trained via reinforcement learning to optimize task performance, even when such rewards may lead to agent behaviors that violate societal norms -- causing harm either to the agent itself or other entities in the environment. Social value alignment refers to creating agents whose behaviors conform to expected moral and social norms for a given context and group of people -- in our case, it means agents that behave in a manner that is less harmful and more beneficial for themselves and others. We build on the Jiminy Cricket benchmark (Hendrycks et al. 2021), a set of 25 annotated interactive narratives containing thousands of morally salient scenarios covering everything from theft and bodily harm to altruism. We introduce the GALAD (Game-value ALignment through Action Distillation) agent that uses the social commonsense knowledge present in specially trained language models to contextually restrict its action space to only those actions that are aligned with socially beneficial values. An experimental study shows that the GALAD agent makes decisions efficiently enough to improve state-of-the-art task performance by 4% while reducing the frequency of socially harmful behaviors by 25% compared to strong contemporary value alignment approaches.
1 Introduction
Interactive agents can optimize task rewards while acting harmfully, because task performance may conflict with social values. This paper proposes GALAD, which uses social commonsense knowledge to constrain actions, and studies value judgments in context on Jiminy Cricket.
1 Introduction
- Task-optimized reinforcement-learning agents can learn harmful behaviors when rewards conflict with socially beneficial values, while many benchmarks lack analysis of reward functions.This makes harmful behavior difficult to diagnose.Source: §1 · p. 1
- Socially beneficial value alignment aims to make agents conform to a group’s context-specific norms while reducing harm to themselves and others.The proposed approach uses social commonsense knowledge to contextually constrain actions.Source: §1 · p. 2
- Jiminy Cricket contains 25 text-based adventure games with annotations across thousands of morally salient scenarios, from bodily harm and theft to altruism.The games include long quests involving thousands of locations, hundreds of characters, and nearly 5,000 objects.Source: §1 · p. 2
- The authors study contextual judgments by asking participants to rate scenario transcripts for moral valence and salience; dense feedback is used for evaluation, not training.Judgments may vary across groups assessing what is socially beneficial in context.Source: §1 · p. 2
- GALAD distills social commonsense knowledge from specially trained language models into action-space constraints and is reported to improve task performance while aligning more closely with social values.The introduction contrasts this approach with policy- and reward-shaping methods.Source: §1 · p. 2
2 Related Work
Prior text-game research largely targets task performance, while value-alignment approaches use feedback to shape policies or rewards. GALAD instead distills contextual social commonsense into guidance about actions to avoid before policy learning.
2 Related Work
- Text-game research has addressed partial observability, natural-language state-action spaces, and commonsense priors, but focuses on task performance rather than analyzing agent behavior.These approaches often measure progress through overall game completion rates.Source: §2 · p. 2
- Value alignment and safe exploration are often framed as constrained optimization, with prior methods learning safe trajectories from demonstrations or modeling rewards that fit human values.Related text-game agents use language-model priors for reward or policy shaping.Source: §2 · p. 3
- Unlike those reward- and policy-shaping approaches, GALAD distills social commonsense into contextual knowledge about actions not to take before policy learning.The closest cited work includes text-game agents that learn social priors from stories or ethical-understanding benchmarks.Source: §2 · p. 3
3 Environment Alignment Verification
The section examines how well source-code-level morality annotations reflect judgments of actions in context. Human ratings show that morally salient situations are common, but agreement declines for finer-grained labels, motivating evaluation based on valence alone.
3 Environment Alignment Verification
- The Jiminy Cricket framework labels action-outcomes by valence and target, with severity levels from 1–3, yielding 12 possible annotation labels.The original labels are assigned pro tanto at the source-code level, independent of how the world changed before that state.Source: §3 · p. 3
- The benchmark’s complex, quest-like games contain morally ambiguous scenarios, and game rewards can encourage socially unacceptable actions.Their fantastical and complex settings also make action harmfulness harder to annotate.Source: §3 · p. 3
- The contextual human evaluation sampled 210 world states with source-code morality annotations and collected five worker ratings per game snippet after 15 preceding turns.Ratings covered moral saliency, valence, target, and severity.Source: §3 · p. 3
- Workers judged 81% of source-code-annotated situations morally salient, agreed with source annotations on valence 67% of the time, and matched exact valence, target, and severity only 24%.Workers showed moderate pairwise agreement on valence (83%; Krippendorf’s α = 0.30) and valence and polarity (70%; α = 0.30).Source: §3 · p. 3
- Because finer-grained source annotations are noisier proxies for contextual acceptability, the study evaluates agents using moral valence alone rather than target and severity.The authors describe valence as having relatively high annotator agreement.Source: §3 · pp. 3–4
4 Value Alignment via Action Distillation
GALAD builds a commonsense value prior, distills its judgments into a contextual action generator, and uses generated candidates in an RL policy. The design aims to supply socially acceptable actions during exploration.
4 Value Alignment via Action Distillation
- GALAD combines a commonsense value prior, a distilled action-candidate generator, and an RL policy that samples generated candidates during exploration.The method section describes these as the agent’s three primary components.Source: §4 · p. 4
4.1 Value Priors from Social Commonsense
- Delphi, fine-tuned from UNICORN and trained on 1.7M publicly polled ethical judgments, assigns positive, neutral, or negative acceptability probabilities to contextualized game actions.Source: §4.1 · p. 4
4.2 Action Distillation
- Because per-step multi-billion-parameter feedback and exhaustive search are infeasible, GALAD uses a 117m-parameter GPT-2 generator to produce relevant candidates from context.It trains on human-play transcripts from over 500 games and oracle-agent samples from 31 games; curation avoids overlap with Jiminy Cricket benchmark games.Source: §4.2 · pp. 4–5
- The generator learns context-to-action sequences on curated datasets, with value-prior scores scaling examples to favor less harmful actions.The training strategy is modified sequence-to-sequence learning, and the prior evaluates each action in context.Source: §4.2 · p. 5
4.3 RL Policy Training
- The RL policy scores generated actions with a Q-value function, samples an action, and trains with replay-buffer experience and prioritized sampling.Q-learning maximizes long-horizon reward and implicitly minimizes expected value-alignment scores.Source: §4.3 · pp. 5–6
5 Evaluation
The evaluation compares GALAD with contemporary agents and tests which of its components matter. GALAD improves completion while reducing harmful behavior, and the ablations favor constraining actions before policy training.
5 Evaluation
- Evaluation measures harmfulness, completion percentage, and harmfulness divided by completion; it uses valence-only labels because severity annotations may be noisier.The authors report that including severity labels does not change the discussed trends.Source: §5 · p. 6
- Each agent is tested at five starting locations in each of 25 games, across 125 environments, with five random seeds and averaged results.Training steps, parallel environments, and model sizes are held constant across agents.Source: §5 · p. 6
5.1 Comparison to Baselines
- GALAD achieves 4% greater completion than CMPS+ and 25% less harmful behavior than CMPS, the next-best agents on those metrics.The reported values are relative comparisons over separate next-best baselines.Source: Table 1 · p. 7, §5.1 · p. 7
- For most baselines, completion and harmful behavior rise together; CMPS+ has 6.3% higher relative completion but 31.4% more harmful actions than CMPS.The authors suggest that action-candidate training data can skew agents toward harmful behavior.Source: §5.1 · p. 7
5.2 Ablation Study
- Ablations test negated valence, smaller-prior policy shaping, same-prior reward shaping, and dense-feedback Oracle policy shaping.The Oracle uses the benchmark's ground-truth harmfulness feedback and is included as an analysis upper bound on harmfulness.Source: §5.2 · p. 8
- Dense-feedback Oracle training significantly lowers harmfulness, indicating room for improvement beyond the other tested approaches.The Oracle is an analysis upper bound, not a proposed training setup.Source: §5.2 · p. 8
- GALAD performs best without additional reward or policy shaping during policy-network training.The authors characterize shaping as a noisy dual-optimization problem and pre-training to learn which actions to avoid as an easier alignment route.Source: §5.2 · p. 8
- GALAD– performs worse than GALAD, GALAD+PS, CMPS, and CMPS+ across all metrics; the authors conclude altruistic actions can sometimes aid task performance.They argue that defaulting to socially beneficial rather than unacceptable behavior can be more effective in these environments.Source: §5.2 · p. 8
6 Conclusion
Task incentives can conflict with socially beneficial behavior, but the conclusion presents GALAD as a way to reduce harm without sacrificing task competency.
6 Conclusion
- Value alignment is often framed as requiring a proportional sacrifice in general task competency.Source: §6 · p. 8
- GALAD uses language-model social commonsense to learn contextual soft constraints on actions, aiming to reduce social harm without losing task competency.Source: §6 · p. 9
- Reward- or policy-shaping approaches can frame task performance and socially acceptable norms as a dual optimization problem.Source: §6 · p. 9
7 Ethical Considerations
The paper treats harmfulness as a design concern alongside task rewards and studies how human judgments map to the benchmark’s social values. It also limits its claims to text environments and culturally situated values.
7 Ethical Considerations
- The study asks human participants to judge scenario moral valence and salience to assess how evaluation metrics map to this group’s socially beneficial values.Source: §7 · p. 9
- Text environments mitigate physical risks but do not address cognitive or emotional risks, and language generation may still produce harmful or biased language.Source: §7 · p. 9
- The value prior implicitly biases an RL agent toward context-grounded actions deemed more socially beneficial, rather than reasoning about moral scenarios in isolation.Source: §7 · p. 9
- The value-prior data, games, and added annotations skew toward Western norms, so these values are not universal and do not directly assess real-world human actions.Source: §7 · p. 9
A.1.1 Annotation Setup
The annotation setup gathers human judgments of morally salient game actions in context, using the Jiminy Cricket dimensions and preceding turns from oracle-played transcripts.
A.1.1 Annotation Setup
- Annotators judged whether an action was morally salient, then rated its valence, target, and severity using the Jiminy Cricket framework.Source: §A.1.1 · p. 12
- Workers could select any of the four valence-target pairs and assign a severity rating to each selected pair.Source: §A.1.1 · p. 12
- Participants saw instructions and a qualifying question about text-game familiarity, followed by game context and the scenario transcript.Source: §A.1.1 · p. 12
- The study randomly sampled 210 transcripts, each containing a morally salient action-outcome pair and the 15 preceding turns from an oracle agent’s game transcript.Source: §A.1.1 · p. 12
A.1.2 Annotation Result Details
Annotator agreement was stronger for moral saliency and broad judgments than for fine-grained valence-target-severity triples. The participant pool also skewed liberal.
A.1.2 Annotation Result Details
- The annotator pool was 66% moderately or moderately liberal and 27% moderately or moderately conservative.Source: §A.1.2 · p. 13
- 35% pairwise agreement (α = 0.17) for valence-polarity-intensity triples was lower than agreement on broader annotation tasks.Source: §A.1.2 · p. 13
- Annotators reached 73% pairwise agreement on moral saliency; valence selection had 83% agreement (Krippendorf’s α = 0.30), and valence-polarity had 70% (α = 0.30).Source: §A.1.2 · p. 13
A.2 Value Prior Training Details
The appendix describes a compact Delphi-based value prior for policy shaping and notes its runtime tradeoff against GALAD. Table 3 is identified as containing the prior’s training hyperparameters.
A.2 Value Prior Training Details
- The policy-shaping value prior is a T5-large (< 1 billion parameters) Delphi model trained with COMMON-SENSE NORM BANK.The original T5-11B model is described as too bulky for on-the-fly policy shaping.Source: §A.2 · p. 16
- The smaller value prior is comparable in size to the RoBERTa value prior used in CMPS.Source: §A.2 · p. 16
- Using the smaller value prior for policy shaping is up to 5x more time-consuming than GALAD.Source: §A.2 · p. 16
- Table 3 is identified as listing the training hyperparameters for the T5-large value prior.Source: Table 3 · p. 16
A.3 GALAD Training Details
This section presents supplementary GALAD materials, including an unadjusted benchmark comparison and documentation of training data and gameplay transcripts. It also provides narrative examples of beneficial and harmful behavior, failed action attempts, and unfolding social episodes.
A.3 GALAD Training Details
- Training used 223,547 ClubFloyd context-action pairs and 463,930 JerichoWorld observation-action samples; RL ran for at most 12 hours per 15,000 steps on one A100.ClubFloyd pairs interleaved observations and actions, while JerichoWorld pairs mapped observations to valid actions that change the world state.Source: §A.3 · p. 16
A.4 GALAD Unadjusted Results
- GALAD achieved a 25.4% relative reduction in harmful behavior and 4% task performance increase over separate next best baselines.The unadjusted metrics count harmful behavior toward other entities across valence, target, and severity dimensions; the reported trends match the adjusted results.Source: §A.4 · p. 16, Table 5 · p. 17
A.5 GALAD Gameplay Transcripts
- The transcript section samples three beneficial and three harmful scenarios after convergence at 15,000 steps per environment in a batch of eight, noting harmful cases outnumber beneficial ones.The scenarios were judged using the annotation framework, and the transcripts begin with brief game descriptions for context.Source: §A.5 · p. 18
A.5.1 Examples of beneficial behavior
- The beneficial-case transcripts include failed actions, such as vocabulary-rejected commands and attempts to use a microwave that is not open.The passages present interaction traces alongside game responses and rewards.Source: §A.5 · p. 18, §A.5.1 · pp. 18–19
- In the beneficial-behavior example, the player gives Sharon a logbook after finding her searching an open file drawer.The scene is set in Sharon Kemp’s office; the trace records the handover with no reward.Source: §A.5.1 · p. 20
A.5.2 Examples of harmful behavior
- The harmful-behavior examples provide gameplay traces from Ballyhoo and Planetfall, including attempted actions and character interactions.The supplied excerpts show actions such as throwing a clown mask at a barred passage and speaking with a guard, alongside many failed or ineffective actions.Source: §A.5.2 · pp. 21–22
- Actions such as trying to push an unseen button, unlock a pod without a visible key, or use an unknown direction produce failure messages.The attempts receive no reward in the excerpts.Source: §A.5.2 · pp. 22–23
- A separate ship excerpt depicts explosions, emergency bulkheads closing, and Blather angrily ordering the player back to their post.Source: §A.5.2 · p. 23
- In Spellbreaker, guild members voice concerns about failing magic, its effects on their work and safety, and suspicions about the enchanters.The gathering is disrupted when the guildmasters are transformed into amphibians and a cloaked figure leaves.Source: §A.5.2 · pp. 23–24