PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Behavior in Relational Moral Dilemmas: Moral Rightness, Predicted Human Behavior, and Model Decisions

Jiseon Kim, Jea Kwon, Luiz Felipe Vecchietti, Wenchao Dong, Jaehong Kim, Meeyoung Cha

arXiv:2604.21871v1cs.CL

TL;DR

The paper asks whether LLMs capture context-dependent moral nuance, a question made important by their growing role in social contexts. It varies severity and relational closeness in the Whistleblower’s Dilemma and compares moral rightness, predicted human behavior, and model decisions. It finds that model decisions align with moral-rightness judgments even as predicted human behavior becomes more loyalty-sensitive with closeness.

  • Problem

    Prior work often treats model morality as static, leaving underexamined whether LLMs capture context-dependent moral nuance in socially situated dilemmas.

    Source: §1 · p. 1

  • Method

    The study varies crime severity and relational closeness in the Whistleblower’s Dilemma and evaluates moral rightness, predicted human behavior, and model decisions across six LLMs.

    Source: §1 · p. 2

  • Results

    Model decisions align more closely with moral-rightness judgments than predicted human behavior, which reflects greater loyalty as relational closeness increases.

    Source: §6 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    Evaluating LLMs across multiple perspectives can reveal deployment-relevant differences between advising what is right and predicting what people would do.

    Source: §6 · p. 8, §7 · p. 9

    The study includes no human responses for its scenarios, so it cannot directly assess the accuracy of predicted human behavior or quantify alignment with human decisions.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Human moral judgment is context-dependent and modulated by interpersonal relationships. As large language models (LLMs) increasingly function as decision-support systems, determining whether they encode these social nuances is critical. We characterize machine behavior using the Whistleblower's Dilemma by varying two experimental dimensions: crime severity and relational closeness. Our study evaluates three distinct perspectives: (1) moral rightness (prescriptive norms), (2) predicted human behavior (descriptive social expectations), and (3) autonomous model decision-making. By analyzing the reasoning processes, we identify a clear cross-perspective divergence: while moral rightness remains consistently fairness-oriented, predicted human behavior shifts significantly toward loyalty as relational closeness increases. Crucially, model decisions align with moral rightness judgments rather than their own behavioral predictions. This inconsistency suggests that LLM decision-making prioritizes rigid, prescriptive rules over the social sensitivity present in their internal world-modeling, which poses a gap that may lead to significant misalignments in real-world deployments.

1 Introduction

The paper asks whether LLMs capture context-dependent moral nuance, which prior evaluations often treated as static. It varies severity and relational closeness in the Whistleblower’s Dilemma and finds that moral judgments, predicted behavior, and model decisions can diverge.

1 Introduction

  • Prior evaluations often treat model morality as monolithic and static, overlooking context-dependent human moral reasoning and the role of close relationships.The paper frames this as an important gap because LLMs increasingly influence normative reasoning in social contexts.Source: §1 · p. 1
  • LLM outputs shift with relational context and evaluative perspective, while their sensitivity to social nuance is not applied consistently across perspectives.The authors characterize this structural inconsistency as a source of unstable outputs and motivate context-aware, multi-perspective evaluation.Source: §1 · p. 2
  • The framework varies relational closeness and crime severity in the Whistleblower’s Dilemma to evaluate AI-generated moral choices.The study evaluates 1,296 prompt instances across six LLMs, examining moral rightness, predicted human behavior, and autonomous model decisions.Source: §1 · pp. 1–2
  • The paper identifies discrepancies among models’ moral-rightness judgments, predictions of human behavior, and their own decisions.It also analyzes how fairness and loyalty shift across these evaluative perspectives.Source: §1 · p. 2

2 Related Work

Human moral judgment varies with relationships and with whether a question concerns what should be done or what people will do. Existing LLM morality benchmarks often use static, binary evaluations, leaving these contextual and perspective shifts underexamined.

Contextuality and Perspective in Morality

  • Human moral judgment can shift toward loyalty and care in close relationships, and prescriptive judgments can diverge from predictions of what people will do.The Whistleblower Dilemma provides a setting to examine competing values such as fairness and loyalty, which may be reconfigured by severity and relational closeness.Source: Contextuality and Perspective in… · p. 2
  • Prior LLM evaluations have expanded from Moral Machine scenarios to multilingual, persona-based, and everyday-dilemma settings.The cited work includes Moral Machine studies and later extensions to broader contexts.Source: Contextuality and Perspective in… · p. 2
  • Prior LLM morality benchmarks often use static, binary responses in isolated settings, missing shifts across prescriptive, descriptive, and decisional perspectives.This work varies relational closeness and crime severity to examine context sensitivity and cross-perspective inconsistency.Source: Contextuality and Perspective in… · p. 2

3 Whistleblower’s Dilemma Framework

The framework adapts the Whistleblower’s Dilemma to measure how reporting judgments vary by crime severity, relational closeness, and evaluative perspective. It uses a shared scenario structure to compare normative judgments, predictions of human behavior, and model decisions.

3 Whistleblower’s Dilemma Framework

  • The Whistleblower’s Dilemma is used to measure moral judgments across contextual conditions and perspectives, with each prompt pairing a scenario and a perspective-specific question.The prompt template asks whether someone witnessing wrongdoing should report, with the crime severity, relationship, and question systematically varied.Source: §3 · p. 3, §3.1 · p. 3

3.1 Template

  • Each condition combines an evaluative perspective, severity level, and relational closeness level.The template instantiates these as variables p, s, and r.Source: §3.1 · p. 3

3.2 Severity and Closeness Levels

  • Severity is graded across minor, moderate, major, and critical harms, using three crime types and scenarios generated with GPT-4 and manually validated.Closeness ranges from stranger to acquaintance, friend, and family, with three wording variants at each level.Source: §3.2 · p. 3

3.3 Perspective-based LLM Behavior

  • The three perspectives ask whether reporting is morally right, whether people would report, and whether the model itself would report.These distinguish prescriptive moral rightness, descriptive expectations of human behavior, and the model’s own behavioral intention; comparing them reveals inconsistencies and whether decisions align with stated rightness judgments.Source: §3.3 · pp. 3–4

4 Experiment Setting

The experiments evaluate six LLMs on a dataset spanning three perspectives, four severity levels, and four closeness levels. Reporting ratios are aggregated across repeated runs, while moral-foundation terms and a pilot human annotation study support analysis of reasoning and predicted behavior.

4 Experiment Setting

  • The dataset comprises 1,296 prompts across 3 perspectives, 4 severity levels, and 4 closeness levels, yielding 432 prompts per perspective and 27 prompts per condition.Source: §4 · p. 4
  • Reporting ratios are calculated separately by perspective, severity, and closeness, and results are averaged across three independently seeded runs.Each prompt yields a binary report/no-report decision, and the ratio summarizes decisions for a condition.Source: §4 · p. 4
  • The study evaluates six LLMs, including Claude 3.5 Haiku, Gemini 2.5 Pro, GPT-5 mini, o3-mini, Qwen3 30B A3B Thinking, and DeepSeek-V3.-1.Model usage details are provided in Appendix B.3.Source: §4 · p. 4
  • The Moral Foundations Dictionary maps rationale vocabulary to moral foundations, allowing the authors to measure the proportion of words associated with each foundation.The main analysis focuses on Fairness, Loyalty, Authority, and Care; Sanctity terms are rare and excluded.Source: §4 · p. 4
  • A pilot annotation study uses 80 scenario instances and four annotators to assess reportability and applicable moral foundations.Structured human annotations are converted to proportional ratios for quantitative comparison with machine predictions.Source: §4 · p. 4

5 Result

Across contexts, reporting rose with crime severity and fell with closeness, but the perspectives diverged: moral-rightness judgments remained high while predicted behavior was more context-sensitive. Model decisions tracked moral rightness more closely, and predicted reporting decisions aligned with human reporting annotations.

5.1 Contextual Sensitivity

  • Across all six models and perspectives, reporting rose with crime severity and fell with interpersonal closeness, though contextual effects differed by perspective.Source: §5.1 · p. 5

5.3 Moral Dynamics

  • Fairness and Authority ratios tended to accompany reporting, while Loyalty was more associated with silence; predicted human behavior emphasized Loyalty and reported less.Moral rightness and model decisions had near-ceiling report rates, averaging above 0.9, while predicted human behavior reported around 0.5.Source: §5.3 · pp. 5–6
  • Closeness generally increased Loyalty’s share while severity decreased it; closeness effects were larger, especially for predicted behavior, where Fairness also fell.Source: §5.3 · pp. 6–7

5.2 Perspective Inconsistencies

  • Moral-rightness reporting ratios had medians above 0.9, while predicted-human-behavior ratios were lower and widely dispersed; model decisions tracked moral rightness more closely.Figure 3 aggregates reporting ratios across severity and closeness levels.Source: §5.2 · p. 5, Figure 3 · p. 7

Human annotation Agreement and Alignment

6 Discussion

Models predict greater loyalty in human behavior as relationships grow closer, but their own decisions remain closer to moral-rightness judgments. The discussion argues that evaluating these distinct perspectives matters for deployment and value-sensitive steering.

6 Discussion

  • Predicted human behavior gives Loyalty greater weight as closeness increases, while model decisions remain closer to moral-rightness judgments.The authors interpret this as recognizing relational obligations without reflecting them in the models’ own normative judgments.Source: §6 · p. 7
  • A policy that systematically down-weights Loyalty may be internally coherent yet culturally selective, especially where relational duties are normatively salient.The discussion connects this concern to context-specific duties and social position.Source: §6 · p. 8
  • Evaluating only one perspective can miss differences between advising what is right and predicting what people would do.The authors recommend evaluating models across perspectives tied to distinct deployment objectives, without attributing the divergences to cognitive identity.Source: §6 · p. 8
  • When asked to decide, models tend to give morally consistent “should” answers even when these differ from their predictions of typical human behavior.The authors describe this as extending prior observations of idealized generations to moral decision-making.Source: §6 · p. 8
  • Diagnosing which moral values models emphasize can inform prompt or policy adjustments for the intended purpose in a given setting.The authors present value extraction as a practical basis for steering model behavior.Source: §6 · p. 8
  • The mechanisms behind context-dependent differences remain unknown, and studying them in commercial systems is constrained by limited access to internal representations.The authors suggest mechanistic interpretability methods may be more tractable in open-source models.Source: §6 · p. 9

7 Conclusion

The conclusion describes machine moral reasoning as contingent on context and evaluative perspective. Models’ decisions track prescriptive moral-rightness judgments rather than their predictions of human behavior, creating a practical concern for social deployment.

7 Conclusion

  • Machine moral reasoning is contingent on situational context and evaluative perspective rather than stable or monolithic.The conclusion reports fairness-based norms in prescriptive judgments alongside capacity to model loyalty-driven nuances of human behavior.Source: §7 · p. 9
  • Models’ decisions align with prescriptive moral-rightness judgments rather than what they expect humans would do.The authors describe this as a systematic gap between what machines represent and how they choose to act.Source: §7 · p. 9
  • When machines predict one pattern of behavior but recommend another, their advice can feel detached from how people reason and act.The conclusion calls for evaluations that surface shifts in moral judgments across context, relationship, and framing.Source: §7 · p. 9

Limitations

The study examines divergences among model outputs but lacks human responses for its scenarios, limiting direct assessment of behavioral prediction accuracy. Its demographic scope, binary prompts, and focus on one dilemma also constrain how broadly findings can be interpreted.

Limitations

  • Without human responses to these scenarios, the study cannot directly test predicted human behavior against real behavior or quantify alignment with human decisions.The observed response patterns broadly align with trends reported in prior studies, but that does not provide a direct comparison for these scenarios.Source: Limitations · p. 10
  • The prompts specify no demographic reference group, and the evaluation uses fixed linguistic framing, leaving demographic differences and population generalizability untested.The authors report preliminary demographic-group differences in model outputs but leave systematic investigation for future work.Source: Limitations · p. 10
  • Binary responses and prompt design may affect robustness; prior work shows that prompt modifications can change alignment results.The authors note that additional prompt variations and dilemma modeling could strengthen the benchmark.Source: Limitations · p. 10
  • Because the study focuses on the Whistleblower’s Dilemma, its patterns and alignment with human preferences may not generalize to other dilemma families.The authors identify expanding the range of dilemma types as a way to support more reliable moral benchmarks.Source: Limitations · p. 10

Ethical considerations

The work is a diagnostic, not a prescription for converging models on one normative answer, and its framing is predominantly Western. The supplementary prompt and severity examples preserve core scenario variables while varying surface wording and crime category.

Ethical considerations

  • The study characterizes responses without endorsing a resolution, and cautions that single-answer alignment may suppress relational and culturally situated tradeoffs.The Moral Foundations Dictionary, English-language prompts, synthetic scenarios, and team-run structured pilot annotation further define the study’s scope.Source: Ethical considerations · p. 10

A.1 Template

  • Alternative prompt templates vary surface phrasing and formatting while preserving severity, closeness, perspective-specific questions, and binary Yes/No responses.These variants are intended to check that observed patterns are not artifacts of one wording choice.Source: §A.1 · p. 12

A.2 Severity Level Examples Across Crimes

  • Fraud and burglary are each represented at four severity levels, enabling comparisons across severity while holding crime category fixed.The levels are minor, moderate, major, and critical.Source: §A.2 · p. 12

B Experiment Setting

Across models, reporting ratios rise with crime severity and fall with relational closeness, while moral-language ratios in rationales are used to characterize how emphasized considerations vary across contexts and decisions.

B.1 Response Rates Across Models

  • Valid response rates exceeded 95% across models, and the authors report that observed reporting patterns are unlikely to reflect selective non-response or safety refusals.Table 4 summarizes average valid response rates by model and perspective.Source: §B.1 · p. 12, Table 4 · p. 13

B.2 Moral Value Extraction

  • Moral emphasis is measured as normalized shares of care, fairness, loyalty, and authority words, excluding sanctity and renormalizing the remaining ratios.Matched words are identified with the Moral Foundations Dictionary; the table of unique mapped words and average matched-word counts characterize lexical coverage and frequency.Source: §B.2 · p. 12

B.3 Model Configuration

  • The study queried six named models through public inference APIs, generally using provider-default inference settings and one deterministic pass per condition.The gpt-5-mini model additionally used low verbosity; no repeated sampling was conducted per prompt.Source: §B.3 · pp. 12–13

C.1 Reporting Ratios Across Context Levels

  • Reporting ratios increase with crime severity and decrease with relational closeness across models and perspectives.The directions of both trends remain stable despite differences in overall reporting propensity and gradient magnitude.Source: §C.1 · p. 13

C.2 Contextual Sensitivity of Moral Values

  • Contextual sensitivity analyses jointly regress each moral ratio on closeness and severity, estimating per-level shifts while holding the other factor constant.The analysis uses HC3-robust standard errors and 95% confidence intervals; moral foundations are descriptive signals, not asserted drivers of decisions.Source: §C.2 · p. 14

C.3 Moral Value Distribution with Reporting

  • Higher fairness and authority ratios tend to accompany Report responses, while higher loyalty ratios tend to accompany Not report responses.Care is the most prominent value across perspectives and appears in both response types; the associations vary by perspective.Source: Figure 7 · p. 13, §C.3 · p. 15

C.4 Mformer Moral-Level Analysis

An alternative classifier produced patterns consistent with the dictionary-based analysis, while model decisions aligned strongly with human reporting annotations but foundation-level alignment varied.

C.4 Mformer Moral-Level Analysis

  • Mformer analyses of moral-value distributions, reporting decisions, and contextual sensitivity show patterns consistent with MFD-based ratios, suggesting findings are robust to scoring method and less likely driven by verbosity differences.Mformer estimates foundation likelihoods at sentence or document level and is less sensitive to absolute response length than dictionary counts.Source: §C.4 · p. 15

C.5 Human Alignment Details

  • Models show strong decision-level alignment with human reporting judgments, but moral-foundation alignment varies, with Loyalty positively correlated and Care negatively correlated on average.Table 7 reports model-wise Spearman correlation, Pearson correlation, and MAE for reporting decisions.Source: §C.5 · p. 15

D.1 Post-Training Regimes and Perspective Divergence

A controlled comparison of post-trained OLMo variants found that tuning regime and model scale were associated with differences in divergence between prescriptive and predicted responses, though patterns were not uniform.

D.1 Post-Training Regimes and Perspective Divergence

  • Post-training choices and model scale are associated with varying divergence between OLMo models’ prescriptive and predicted responses, with some configurations less aligned with predicted human behavior.The comparison covers instruction-tuned and reasoning-tuned variants at 7B and 32B scales using identical prompts and matched response extraction; patterns are not uniform across settings.Source: §D.1 · p. 15
Loading 2604.21871v1…