Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs

1 Introduction

• The paper motivates this approach with real-world deployments where LLMs must make frequent ethical decisions across domains, cultures, and heterogeneous users.Examples include moderating an online forum for HIV+ youths in Africa and advising Indian farmers about pesticides.Source: §1 · pp. 1–2
• LLMs should reason from a situation and an externally specified moral stance, resolving dilemmas when possible or requesting clarification when the stance is insufficient.The authors argue against fixed value alignment and for value alignment at application development or end-user stages.Source: §1 · pp. 1–2
• The framework combines moral dilemmas with ethical policies spanning interpersonal, professional, social, and cultural conflicts, while supporting deontological, virtue, and consequentialist approaches.Policies can be specified at different abstraction levels.Source: §1 · p. 2
• GPT-4 exhibits nearly perfect ethical reasoning, while ethical reasoning generally improves with model size across the evaluated GPT-x models.GPT-3 and ChatGPT show strong internal value bias associated with poorer reasoning.Source: §1 · p. 2
• Most evaluated models, including GPT-4, favor democratic and self-expression values associated with Western and English-speaking societies over traditional and survival values.The comparison is framed against values characteristic of Global South and Islamic cultures.Source: §1 · p. 2
• The work contributes an argument for decoupling ethical policies from LLM training, an extensible framework for assessing ethical reasoning, and a dataset-based assessment revealing gaps and biases.The dataset is shared in the appendix.Source: §1 · p. 2

2.1 Ethics: Theories and Definitions

• Normative ethics comprises deontology, virtue ethics, and consequentialism, which respectively prioritize duties or rules, agent character, and action outcomes.These distinctions provide the normative alternatives used by the paper’s framework.Source: §2.1 · p. 3
• Ethical dilemmas involve conflicting moral values, and resolving them may require new value judgments within an otherwise consistent ethical framework.The Aisha-Monica authorship case illustrates actions that cannot both be performed simultaneously.Source: §2.1 · p. 3
• Value pluralism holds that multiple values can be equally correct yet conflict, so different individuals or cultures may reach distinct ethically sound resolutions.The paper maps cultures along survival/self-expression and tradition-based/democratic value dimensions.Source: §2.1 · p. 3
• Pluralist approaches reject a universal principle or strict hierarchy capable of resolving every moral conflict across situations and individuals.Rossian pluralism weighs moral pros and cons, whereas particularism makes them situation-dependent.Source: §2.1 · p. 3

2.2 Ethics Frameworks in NLP

• Much NLP ethics work assumes developer-defined deontological rules, but such systems are not readily generalizable to diverse applications and contexts.This limitation is especially consequential for LLMs intended for varied downstream uses.Source: §2.2 · p. 3
• Prior frameworks explore reflective equilibrium, stakeholder interactions, value pluralism, Global South recontextualization, and ethical questions in language technologies.The cited work includes CRE, multi-scale human-machine frameworks, and stakeholder involvement approaches.Source: §2.2 · p. 3
• The paper’s overview treats ethical policies as moral principles with preference orderings and presents system-level factors affecting their definition, operationalization, and implementation.Related work also applies Rawlsian or prioritarian model selection to address unfairness toward low-resource languages.Source: Figure 1 · p. 4, §2.2 · p. 4

3.1 A Critique of Ethical Alignment

• Universal alignment cannot generalize across applications, geographies, laws, and communities because LLMs serve heterogeneous stakeholders and no universally consistent ethical system exists.Source: §3.1 · p. 4
• Alignment datasets can overrepresent particular values, including clear-cut moral judgments and predominantly United States social norms, underrepresenting Global South and minority perspectives.Source: §3.1 · p. 4
• Fixed alignment can conflict with application needs, such as ending customer-service conversations whenever users express toxic or rude behavior.Source: §3.1 · p. 4
• The proposed alternative is a value-neutral, sound ethical reasoner whose ethical alignment is introduced at the application or user-interaction level.Source: §3.1 · p. 4

3.2 Implementing Flexible Ethical Policies

• External classifiers improve predictability but require less powerful post-processing and fresh application-specific datasets, creating scalability challenges.Source: §3.2 · p. 4
• Prompting preserves the LLM’s capabilities but may produce uncertain or unsound reasoning and remain vulnerable to jailbreaks, while fine-tuning or RLHF requires policy-specific data and engineering effort.Source: §3.2 · p. 5

3.3 A Framework for ‘in-context’ Ethical Policies

• The framework composes a task definition, ethical policy, and user input into a prompt, and defines ethical consistency when the generated output validly resolves the task under that policy.Source: §3.3 · p. 5
• When a policy is underspecified and supports incompatible resolutions, the model should return ϕ rather than choose arbitrarily.Source: §3.3 · p. 5

4.1 Experimental Setup

• The experiment compares five GPT-x models on baseline prompts without policies and policy-based prompts requiring one of three responses: should, should not, or cannot decide.Option permutations are used to mitigate ordering bias, producing 24 baseline prompts and 432 policy-based prompts per model.Source: §4 · p. 6, §4.1 · pp. 6–7
• The dataset contains four dilemmas covering interpersonal, professional, societal, cultural, community, and personal value conflicts, each evaluated under explicit policy resolutions.The authors construct 18 policies per dilemma from three normative ethics branches and three abstraction levels, including complementary policies.Source: §4.1 · p. 6

4.2 Experimental Results

• Baseline behavior reveals that GPT-3 tends to choose affirmative responses, whereas GPT-4 favors individualism, self-expression, and professional ethics over interpersonal, societal, and cultural values.These results expose internal moral stances before policy conditioning.Source: §4.2 · p. 7
• GPT-4 achieves 93.29% average policy-based accuracy, compared with 70% for the best individual annotator and 80% for majority human judgments.GPT-3 achieves close to 50% accuracy and seldom changes its baseline prediction when policies are introduced.Source: §4.2 · p. 7, Table 2 · p. 7
• Accuracy drops by around 15% for Level 2 policies, while Level 0 and Level 1 perform similarly; deontological policies generally outperform virtue and consequentialist policies.Performance trends vary by dilemma, supporting theory-neutral and potentially hybrid policy statements.Source: §4.2 · pp. 7–8, Table 3 · p. 8

5 Discussion and Conclusion

• GPT-4 shows substantially lower bias than GPT-3, but all models display high bias on Rajesh’s dilemma involving community values versus individualism and self-expression.The authors interpret GPT-4’s policy-inconsistent response as evidence of a gap in cultural understanding.Source: §5 · p. 8, Figure 2 · p. 8
• The paper argues for decoupling ethical policies from LLM training so application developers and users can specify policies suited to their contexts.This position responds to the difficulty of generalizing one moral alignment across applications, geographies, laws, and communities.Source: §5 · p. 8
• Baseline patterns indicate a preference for individualism, self-expression, and secular democratic values over community and tradition-based values associated with the Global South or Islamic cultures.Figure 3 represents the models as aligned with Western and English-speaking value systems.Source: §5 · p. 8, Figure 3 · p. 8
• Future work must investigate complex policies with simultaneous partial orderings, expand dilemmas across cultures and topics, evaluate more models, and study crosslingual ethical reasoning.The authors also identify unresolved questions about regulation and accountability when models fail to follow application-defined policies.Source: §5 · p. 8
• Broader ethical soundness requires answering who is responsible when an LLM fails to adhere to an ethical policy supplied by application developers.The paper identifies this as a societal question rather than an experimentally resolved issue.Source: §5 · pp. 8–9

• Only the latest models, such as GPT-3.5 and GPT-4 series models, currently exhibit the ethical-reasoning capacity required for in-context ethical policies.The authors expect future language models to build further on this capacity.Source: Limitations · p. 9
• Except for Heinz’s dilemma, the authors constructed the dilemmas, policies, and ideal resolutions, creating potential bias from an ethnically homogeneous design group.The authors describe the dataset as extensible and intend to broaden its representation.Source: Limitations · p. 9
• The policy levels were tailored to this probing experiment and may not match the complexity and scale required by real-life systems.The framework focuses on normative ethics but is described as extensible to other ethical statements.Source: Limitations · p. 9

• The authors maintain that ethical value alignment should occur at the application level, not directly within LLMs.Source: Impact statement · p. 9
• The paper acknowledges that completely unfiltered models could propagate harms and allows alignment to a small set of broadly accepted ethical values.Source: Impact statement · p. 9
• LLMs should have knowledge of diverse ethical principles and their applicability available without having most ethical values injected into the model.Source: Impact statement · p. 9

A Dilemmas and Value Statements

• The section lists the dilemmas, including Heinz’s dilemma and author-created cases, together with policies using different levels and moral types.Source: A Dilemmas and Value Statements · p. 12
• Figures 4–7 present Heinz’s, Monica’s, Rajesh’s, and Timmy’s dilemmas as the core cases used in the framework.Source: Figure 4 · p. 13, Figure 5 · p. 14, Figure 6 · p. 15
• The prompt asks the model to choose among three answers for a situation and statement, selecting the option most consistent with the statement.Source: A Dilemmas and Value Statements · p. 17

B Prompt Structure

• Figure 8 depicts the prompt structure, with unchanged lines in black, changeable lines in blue, and model responses in red.The same prompt structure was maintained across all experiments.Source: Figure 8 · p. 17, B Prompt Structure · p. 17

C.1 Bias and Confusion

• GPT-4 has the least bias and confusion across dilemmas, while GPT-3 has the highest scores on both measures.Source: §C.1 · p. 17
• Model selection probabilities for options O1, O2, and O3 remain close to 33%, indicating no significant positional bias and making drastic effects from option shuffling unlikely.Source: §C.1 · p. 17
• The bias and confusion values are computed using a defined procedure introduced before the corresponding model comparisons.Source: §C.1 · p. 17

C.2 Moral Level Wise Comparison

• GPT-4 performs best across all levels of ethical reasoning capability, whereas GPT-3 performs worst.Source: §C.2 · p. 17

C.3 Moral Framework Wise Comparison

• Figure 9 examines model bias across different dilemmas, while Figure 10 examines confusion across those dilemmas.Source: Figure 9 · p. 18, Figure 10 · p. 18
• Figure 11 reports model performance across different levels of ethical reasoning capability.Source: Figure 11 · p. 18
• Figure 12 reports model performance across virtue, consequentialist, and deontological moral approaches.Source: Figure 12 · p. 18
• Table 4 reports Heinz’s dilemma accuracy for policies varying in type and abstraction level, including overall average accuracy.Source: Table 4 · p. 18
• Models perform best under a deontological moral framework compared with virtue and consequentialist frameworks.Source: §C.3 · pp. 17–18

C.4 Dilemma-wise Views

• GPT-4 achieves 100% accuracy on all listed dilemmas except Rajesh’s dilemma under consequentialist and virtue moral frameworks.Source: §C.4 · p. 18
• Table 5 reports Monica’s dilemma accuracy for policies of different types and abstraction levels.Source: Table 5 · p. 19
• Table 6 reports Rajesh’s dilemma accuracy for policies of different types and abstraction levels.Source: Table 6 · p. 19
• Table 7 reports Timmy’s dilemma accuracy for policies of different types and abstraction levels.Source: Table 7 · p. 19