PaperlayerView exact revision on arXiv

Source-linked AI summary

OMIT the Action: Measuring Framing-Invariant Omission Bias under Philosophical Disagreement

Sihyeon Lee, Jihun Song, Chanwoo Kim, Jiwoo Kum, Chanjun Park

arXiv:2610.07847v1cs.CL

TL;DR

Omission bias in LLM evaluation remains underexplored, with prior work limited in conflict coverage and scale. This paper introduces OMIT, a benchmark built from paired frames and disagreement among five philosophical personas, and evaluates eight LLMs and inference-time interventions. Omission bias appears across evaluated models, while principle-focused prompts reduce it but can also coincide with action-biased responses.

  • Problem

    Omission bias remains comparatively underexplored in LLM research, and prior evaluations are narrow in conflict type, vignette count, and model coverage, with no mitigation analysis.

    Source: §1 · p. 1

  • Method

    OMIT uses 218 paired-frame scenarios across 10 conflict types derived from five-perspective philosophical disagreement to evaluate framing-invariant omission bias and inference-time prompting across LLMs.

    Source: §1 · p. 2

  • Results

    Omission bias appears across evaluated models and tends to decrease with model size within families; principle-focused prompts reduce omission bias and increase frame-consistent responses, but some responses shift toward action bias.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Mitigation should be assessed through omission-bias, action-bias, and frame-consistency outcomes together, rather than OBR alone.

    Source: §6 · p. 9

    The intervention analysis covers inference-time prompting but does not examine how omission bias changes during training or whether training-time interventions reduce it.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As LLMs increasingly assist in moral reasoning, omission bias, the tendency to prefer inaction even when equivalent framings reverse substantive outcomes, poses a significant risk of skewed decision-making. Yet omission bias remains underexplored in LLM evaluation, with the few existing studies limited in scale and focused largely on utilitarian-deontological conflicts. To address this gap, we introduce OMIT, a benchmark consisting of 218 paired-frame scenarios across 10 conflict types, constructed by leveraging disagreement patterns from an LLM-based, five-perspective philosophical persona panel (utilitarianism, deontology, virtue ethics, care ethics, and contractualism). Evaluating eight LLMs, we find that omission bias is pervasive but inversely correlates with model size within families. We further evaluate four inference-time interventions and find that interventions encouraging models to consider moral principles before committing to a yes/no answer reduce omission bias and increase frame-consistent responses, although lower omission bias rates can also coincide with shifts toward action-biased responses. Ultimately, this work contributes not only the OMIT benchmark, but also a methodology for using diverse philosophical disagreement signals to evaluate framing-sensitive inaction preferences and the distributional effects of mitigation attempts in LLMs under complex moral conflicts.

1 Introduction

OMIT evaluates framing-invariant omission bias, a preference for inaction across equivalent action and omission framings, in a broader set of philosophical conflicts than prior LLM studies. Its benchmark and model evaluation show omission bias across evaluated models and that principle-focused prompts can reduce it, though reductions may coincide with action-biased shifts.

Motivation and gap

  • Omission bias is underexplored in LLM research despite its relevance to moral decision-making.It describes judging harm from direct action more harshly than equivalent harm from inaction.Source: §1 · p. 1
  • Prior LLM omission-bias evaluations focus largely on utilitarian–deontological conflicts, use only 13 paired vignettes and few models, and do not test mitigation.The paper identifies these constraints as limitations of prior work.Source: §1 · pp. 1–2

Benchmark and contributions

  • 218 paired-frame scenarios span 10 conflict types derived from disagreement among five normative-ethics personas.The benchmark evaluates eight LLMs and measures framing-invariant Omission Bias Rate (OBR).Source: §1 · p. 2

Findings

  • Principle-focused inference-time prompts reduce omission bias and increase frame-consistent responses, but lower OBR can coincide with action-biased shifts.The authors recommend evaluating OBR alongside changes in action bias and frame consistency.Source: §1 · p. 2

2 Related Works

Prior work establishes omission bias as a moral framing effect and has begun measuring it in LLMs, but existing studies and benchmarks leave gaps in scale and conflict coverage. OMIT combines paired action–omission framing with scenario-level moral-conflict labels, using philosophical personas as construction signals rather than evaluation-time conditioning.

2.1 Cognitive Bias and Omission Bias in LLMs

  • Prior LLM omission-bias work uses one utilitarian–deontological axis, 13 paired vignettes, few model families, and no mitigation strategy.OMIT instead uses a five-philosophy persona panel to type conflicts and studies cross-vendor models and mitigation.Source: §2.1 · p. 3

2.2 Moral Benchmarks for LLMs

  • ETHICS, MoralChoice, and Delphi cover moral domains, ambiguity, or open-ended judgments, but none combines paired action–omission frames with scenario-level conflict typing.OMIT uses high-ambiguity MoralChoice scenarios, reconstructs paired mirror frames, and assigns conflict labels from persona disagreement.Source: §2.2 · p. 3

2.3 Persona Prompting

  • Earlier persona work uses roles and normative viewpoints to condition model outputs; OMIT uses philosophy personas only to identify contested scenarios and evaluates unconditioned models.Prior studies report that persona cues can affect reasoning, social simulations, safety-related generations, and moral or social outputs.Source: §2.3 · p. 3

3 Benchmark Construction

OMIT constructs paired action–omission scenarios and labels them by disagreement among five philosophical personas. Its measure isolates repeated agent-level inaction across frames, and the construction procedure retains scenarios where personas favor opposing substantive outcomes.

3 Benchmark Construction

  • OMIT is a benchmark of paired action–omission mirror scenarios, each typed by disagreement patterns from a five-persona moral-philosophy panel.The panel uses five perspectives: utilitarianism, deontology, virtue ethics, care ethics, and contractualism.Source: §3 · p. 3, §3.3 · p. 4

3.1 Omission bias definition

  • Framing-invariant omission bias is a model’s recommendation of agent-level inaction in both frames, even though the frames reverse the outcome of inaction.Agent-level inaction means not changing the default trajectory; outcome-level inaction refers to the described action not occurring.Source: §3.1 · p. 3, Table 1 · p. 4

3.2 Mirror framing

  • Mirror framing preserves the two substantive outcomes while reversing which outcome follows from agent-level inaction.MoralChoice’s single-frame scenarios are reconstructed as an action frame asking whether to initiate an outcome and an omission frame asking whether to stop it.Source: §3.2 · p. 4

3.3 Philosophy Persona Panel

  • The panel retains scenarios with at least one (Y, N) and one (N, Y) persona response, then labels each cross-group pair as a philosophical conflict.A scenario can receive multiple labels when several persona pairs disagree.Source: §3.3 · p. 5
  • 218 omission-bias test scenarios remain from 657 valid mirror-framed scenarios after preprocessing.The five-persona panel represents utilitarianism, deontology, virtue ethics, care ethics, and contractualism; it is a construction-time probe, not moral ground truth.Source: §3.3 · pp. 4–5

4 Experimental Setup

The evaluation covers eight LLMs across model-size and weight-availability tiers. It classifies paired-frame responses into omission bias, action bias, and frame-consistent outcomes, then reports their rates over successfully parsed pairs.

4.1 Models

  • Eight models span small and large open-weight tiers and a closed-model tier, covering variation in model size, family, and weight availability.The small open-weight tier includes models with at most 12B parameters, the large tier includes models with at least 27B, and the closed tier contains API-based models.Source: §4.1 · p. 5

4.2 Metrics

  • The four paired-frame response combinations classify (N, N) as omission bias, (Y, Y) as action bias, and (Y, N) or (N, Y) as frame-consistent outcomes.A pair is evaluated only when both frame responses are successfully parsed.Source: §4.2 · p. 5
  • OBR, ABR, and FCR are computed over successfully parsed paired responses, and the three rates sum to one; OBR is reported with Wilson 95% confidence intervals.The denominator n is the number of scenarios for which both frame responses are successfully parsed.Source: §4.2 · pp. 5–6

5 Results and Analysis

The results validate persona-disagreement filtering as a way to select scenarios with higher omission bias, while showing substantial variation across models and intervention conditions. Principle-mediated deliberation reduces omission bias and increases frame-consistent responses, but can also coincide with more action-biased responses.

5.2 Per-Model Analysis of Omission Bias

  • Across the eight models, OBR ranges from 0.063 to 0.550, with Gemma-3-12b-it highest at 0.550 and Gemini-2.0-flash lowest at 0.063.The low OBR for Gemini-2.0-flash coincides with the highest ABR, at 0.227.Source: Table 2 · p. 6, §5.2 · pp. 6–7

5.1 Validating the Philosophy Persona Panel

  • Filtered, random-full, and random-complement sets average approximately 0.30, 0.17, and 0.12 OBR, respectively, with this ordering for every model.The filtered set also yields higher OBR than both random baselines across five additional models.Source: §5.1 · p. 6, Figure 2 · p. 7
  • Mean OBR increases monotonically across no, partial, and strong persona disagreement, operationalized as one, two, and three distinct response types.This graded relationship supports the construct validity of the panel labels as more than a binary filtering signal.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 Per-Model Analysis of Omission Bias

  • The larger variant has lower OBR in all three within-family pairs; the Llama change is ∆= −0.381 and the Qwen change is ∆= −0.109, but scale alone does not eliminate bias.The authors caution that training data, instruction tuning, and alignment may also contribute; Llama-3.1-70b-instruct retains OBR of 0.110.Source: §5.2 · p. 7, Table 3 · p. 7

Closed models exhibit divergent bias profiles

  • Closed models diverge: GPT-4o-mini has high OBR comparable to small open models, whereas Gemini-2.0-flash has the lowest OBR and highest ABR.The paper also reports that aggregate model-level differences are broadly preserved across conflict types, though they vary by conflict category.Source: Closed models exhibit divergent bias… · p. 7

5.3 Bias Shifts under Inference-Time Interventions

  • M1 and M4 lower OBR and raise FCR more reliably than M2 and M3, but also raise ABR, so these shifts are not clean bias removal.The authors recommend assessing intervention success through the joint movement of OBR, ABR, and FCR.Source: §5.3 · pp. 8–9

6 Conclusion

OMIT measures framing-invariant omission bias under moral-philosophical conflict using disagreement patterns from five philosophical perspectives. The evaluation finds omission bias across models, while principle-focused prompts reduce it but may shift responses toward action bias.

6 Conclusion

  • OMIT measures framing-invariant omission bias under moral-philosophical conflict using a five-perspective philosophy persona panel.Source: §6 · p. 9
  • Omission bias tends to decrease with model size within evaluated model families, but no model is entirely free from it.Source: §6 · p. 9
  • Prompts encouraging consideration of moral principles before a yes/no answer reduce omission bias and increase frame-consistent responses, but some responses shift toward action-biased patterns.Source: §6 · p. 9

Limitations

The evaluation leaves several boundaries on the conclusions: the model-size analysis covers only evaluated families and sizes, and interventions are limited to inference-time prompting. The benchmark is an initial compact set, and the mechanisms behind shifts between omission and action bias remain unexamined.

Limitations

  • The paper does not analyze mechanisms or conditions under which omission bias shifts into action bias.Source: Limitations · p. 9
  • Within-family scaling conclusions apply only to evaluated model families and sizes; 70B-scale evaluation includes only llama-3.1-70b-instruct.Source: Limitations · p. 9
  • Intervention analysis covers inference-time prompting, not how bias changes during training or whether training-time interventions reduce it.Source: Limitations · p. 9
  • OMIT prioritizes a compact, carefully typed initial test set over exhaustive scenario coverage because the construction pipeline is costly.Source: Limitations · p. 9

Ethical statements

This section documents the ethical safeguards, prompts, and procedures used to construct and evaluate OMIT. It describes the benchmark’s mirror-frame design and construction, including filtering and conflict-label distribution, and summarizes four inference-time prompt designs.

Ethical statements

  • The study used no human subjects or crowdworkers; philosophical personas are not authoritative traditions, and OBR does not imply that action or inaction is morally superior.OMIT uses MoralChoice’s high-ambiguity subset, released under a permissive license, and the benchmark, construction pipeline, and evaluation harness will be released under a permissive license.Source: Ethical statements · p. 10

A Mirror-Framing Example

  • The deer example shows that both frames preserve the same two outcomes while reversing the default: no leaves suffering in one frame and quick death in the other.In the action frame, no means the deer continues to suffer; in the omission frame, no lets the prepared euthanasia proceed.Source: A Mirror-Framing Example · p. 12, Table 4 · p. 12

B Prompts

  • The appendix reports prompts for frame construction, persona labeling, forced-binary evaluation, and mitigation, with angle brackets marking model-filled slots and braces marking call-time inputs.It also introduces the reframer system prompt and a representative user template.Source: B Prompts · p. 12, §B.1 · p. 12

B.1 Mirror-Frame Reframer

  • The reframer constructs paired prompts that preserve the same world and outcomes while reversing the default state and the yes/no question.The procedure constrains actors to plausible pre-completion or external-party setups and avoids authority figures that could introduce an obedience confound.Source: §B.1 · pp. 12–13

B.2 Philosophy-Persona Panel

  • The construction-time panel uses five philosophical personas, each returning a forced yes/no answer with a one- or two-sentence justification.The shared output format prohibits refusal, abstention, hedging, or choosing both.Source: §B.2 · p. 13

B.3 Evaluation System for M0 Baseline

  • The M0 baseline evaluates action and omission frames in separate calls using a forced-binary prompt and disallows rationale generation to preserve the measured construct.Each of the eight evaluation models receives the system prompt with either frame.Source: §B.3 · p. 14

B.4 Inference-Time Intervention Prompts

  • Inference-time interventions reuse the baseline anti-refusal header, vary only the output structure, and are applied separately to action and omission frames.The appendix introduces M1 as a generic chain-of-thought condition.Source: §B.4 · p. 14
  • M1 uses stepwise reasoning, M2 argues for the opposite tentative answer, M3 evaluates a reversed-default mirror, and M4 states a general moral principle before answering.M4’s Stage 1 asks for a principle without the yes/no wording or a specific action verb, aiming to reduce commitment to the question’s surface form.Source: §B.4 · pp. 14–15

C OMIT Construction Statistics and Compute

C.1 Construction Funnel

  • 218 paired scenarios form the frozen OMIT benchmark after filtering: 256 of 657 are removed at Stage 1, and 218 of the remaining 401 meet Stage 2’s two-sided disagreement requirement.Construction begins with 680 scenarios; 19 without a clean lexical inaction anchor are excluded before mirror-frame reconstruction, and four malformed records are removed.Source: §C.1 · p. 15, Table 5 · p. 16

C.2 Conflict-Label Distribution

  • Across 218 conflict-labeled scenarios, the average number of pairwise conflict labels is 3.47, ranging from 1 to 6 across the 10 possible persona pairs.A scenario can receive multiple labels because multiple personas may fall on either side of the YN/NY split.Source: §C.2 · p. 15, Table 6 · p. 16

C.3 Construction Compute and Cost

OMIT’s construction stages use LLM-assisted labeling, mirror-frame reconstruction, and philosophy-persona annotation; its frozen release contains 218 conflict-labeled paired records.

C.3 Construction Compute and Cost

  • The construction pipeline has three LLM-assisted stages: inaction-side labeling, mirror-frame reconstruction, and philosophy-persona panel annotation.The inaction-labeling stage uses LLM fallback calls; regex labeling requires no LLM calls.Source: §C.3 · p. 16
  • Mirror-frame reconstruction uses one GPT-5 call per scenario, while the GPT-4.1-mini panel uses one call per scenario, persona, and frame.Evaluation costs for the eight models are separate from the construction costs.Source: §C.3 · p. 16

C.4 Release Artifact

  • The frozen OMIT v1 artifact contains 218 conflict-labeled paired records with both frame prompts, persona response pairs, conflict labels, and construction metadata.Source: §C.4 · p. 16

D Panel Selection Is Not a Trivial Surface Proxy

The paper tests whether philosophy-panel disagreement selects scenarios that elicit omission bias beyond what simple surface features capture. Panel disagreement produces the clearest upward trend in OBR, unlike scenario length or harm-feature count.

D Panel Selection Is Not a Trivial Surface Proxy

E Additional Frontier-Model Evaluation

The appendix extends evaluation to five additional models and examines panel filtering, conflict-level patterns, persona alignment, and intervention outcome distributions. Omission bias remains observable across the additional models, while panel filtering yields higher OBR than random baselines for each.

E Additional Frontier-Model Evaluation

E.2 Philosophy Persona Panel Validation

  • 0.166 mean OBR on the filtered set exceeds 0.075 for random-full and 0.057 for random-complement, with filtered OBR higher for every additional model.Source: §E.2 · p. 17, Table 10 · p. 18

F Per-Conflict OBR Heatmap

  • Across conflict types, higher-aggregate-OBR models tend to remain higher and lower-OBR models lower, while utilitarian-pole conflicts often show higher OBR among omission-prone models.The authors treat these patterns descriptively because some conflict types have few scenarios; Figure 5 organizes conflicts by persona-pair labels and model tier.Source: F Per-Conflict OBR Heatmap · p. 17, Figure 5 · p. 19

G Model Moral Fingerprint

  • Models differ in their frame-level yes/no alignment with the five philosophy personas, a descriptive profile that does not define ground-truth moral labels.Figure 6 reports Cohen’s κ for each model-persona pairing; positive and negative values indicate above- and below-chance alignment, respectively.Source: G Model Moral Fingerprint · p. 17, Figure 6 · p. 19

E.1 Omission Bias across Additional Models

  • 0.060–0.252 is the OBR range across the five additional models; four fall within the larger open-weight models’ 0.110–0.298 range, while GPT-4o falls below it.These comparisons extend model coverage but do not establish a scaling relationship across model families.Source: §E.1 · p. 17, Table 9 · p. 18

H Full Outcome Rates under Inference-Time Interventions

  • Table 11 reports per-model OBR, ABR, and FCR under the baseline and four interventions, with valid-response-pair denominators varying from 204 to 218.OBR is both-frame inaction, ABR both-frame intervention, and FCR frame-consistent outcomes.Source: H Full Outcome Rates under… · p. 17, Table 11 · p. 18

I.1 Representative M1 Trace

A representative trace shows M1 can organize a stable ethical judgment before mapping it to frame-specific answers. An audit finds that principle-like reasoning alone does not ensure debiasing, so interventions should be assessed across the full outcome distribution.

I.1 Representative M1 Trace

  • In one llama-3.1-70b-instruct case, M1 shifted baseline (N, N) to frame-consistent (N, Y) by weighing similar ethical considerations across both frames.The model favored preserving the medicine flow, mapping that stable judgment to different surface answers because the frames have opposite yes/no polarities.Source: §I.1 · p. 20

I.2 Principle-Reasoning Audit

  • 517 of 522 baseline (N, N) pairs were analyzed after five lacked M1 responses because of HTTP 403 API access errors.The table caption defines records as model–scenario pairs and says percentages are calculated within each transition row.Source: Table 12 · p. 21
  • Principle-like reasoning appeared in M1 traces but was not sufficient for successful debiasing, supporting evaluation across the full OBR/ABR/FCR outcome distribution.Named frameworks appeared in transitions to frame-consistent responses as well as cases that remained (N, N) or shifted to (Y, Y).Source: §I.2 · p. 21
Loading 2610.07847v1…