PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Probabilities, Different Advice: Gain-Loss Framing Effects in High-Stakes LLM Decision Support

Yuyao Zhang, Yakun Cui, Han Zhu, Jinsong Lu, Yike Guo, Sirui Han

arXiv:2610.29294v1cs.AIcs.CL

TL;DR

The paper asks whether equivalent probabilities expressed through gain and loss frames change high-stakes LLM advice. It introduces a controlled benchmark and finds recommendation-category changes across models, with probability canonicalization substantially reducing sensitivity. These results identify probability representation as a practical consistency target, while decision correctness remains outside what consistency alone establishes.

  • Problem

    It remains unclear whether probability-equivalent gain-loss framing systematically changes high-stakes recommendations beyond ordinary generation variability, despite known prompt sensitivity and decision biases.

    Source: §1 · p. 1

  • Method

    The authors evaluate seven LLMs on a controlled benchmark of 2,880 prompts from 240 scenarios and test human judgments, conversational paraphrases, repeated generations, and inference-time interventions.

    Source: §1 · p. 2, §3.3 · p. 3, §3.2 · p. 3

  • Results

    32.4% of complete groups changed recommendation category under gain-loss framing, most changes involved uncertainty, and canonicalization reduced category changes to 6.7%.

    Source: §1 · p. 2, §6 · pp. 7–8

  • Takeaways & Limitations

    Probability representation is a practical intervention target for consistency, but greater recommendation consistency does not establish greater decision quality.

    Source: §7 · p. 8, §6 · p. 8

    Most scenarios lack complete utility functions, user preference models, or domain-specific normative answers, limiting conclusions about decision correctness.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to support consequential decisions, yet their reliability under equivalent representations of uncertainty remains unclear. Existing work has documented prompt sensitivity, cognitive biases, and confidence miscalibration, but it is still unknown whether high-stakes advice changes when the same probability is expressed through complementary gain and loss frames. We introduce FrameBench-HPS, a controlled benchmark of 2,880 prompts derived from 240 high-stakes scenarios across healthcare, finance, education/career, and legal/consumer-rights domains. Across seven LLMs, gain-loss framing changes the recommendation category in 32.4% of complete groups, with most changes involving uncertainty rather than direct reversals. Human evaluation confirms corresponding shifts in perceived risk and directiveness. Robustness experiments show that framing effects persist under conversational paraphrases and exceed ordinary repeated-generation variability. Probability canonicalization substantially reduces framing sensitivity, identifying probability representation as a practical target for improving consistency in high-stakes LLM decision support.

1 Introduction

Equivalent probabilities expressed through gain and loss frames may produce different high-stakes recommendations, a reliability question not resolved by broader evidence of prompt sensitivity and decision biases. FRAMEBENCH-HPS tests this across seven LLMs and finds framing-related recommendation changes, with canonicalization reducing sensitivity.

1 Introduction

  • Prior evidence of prompt sensitivity and decision biases leaves unclear whether equivalent probability descriptions systematically alter high-stakes recommendations beyond ordinary generation variability.The introduction also asks whether lightweight inference-time interventions can reduce this sensitivity.Source: §1 · pp. 1–2
  • FRAMEBENCH-HPS comprises 2,880 prompts from 240 high-stakes scenarios across healthcare, finance, education/career, and legal/consumer-rights domains.The benchmark examines probability-equivalent framing and its robustness and mitigation.Source: §1 · pp. 1–2
  • 32.4% of complete groups changed recommendation category under gain-loss framing, while direct yes-no reversals remained rare.Most changes involved uncertainty rather than direct reversals.Source: §1 · p. 2
  • The study evaluates seven LLMs, validates response properties with three annotators, and tests conversational paraphrases, repeated generations, and inference-time interventions.The interventions include probability canonicalization and frame-robust prompting.Source: §1 · p. 2
  • Probability canonicalization reduces matched-repeat category changes to 6.7% and nearly removes the mean framing-related risk shift.The authors identify probability representation as a practical intervention point.Source: §1 · p. 2

2 Related Work

Related work covers consequential LLM decisions, framing and cognitive biases, and prompt robustness. FRAMEBENCH-HPS focuses on complementary probability framing in high-stakes advice and compares its effects with repeated-generation variability.

LLMs for high-stakes decision support

  • Prior studies examine LLMs in consequential finance, legal, and healthcare settings, including abstention under uncertainty and confidence calibration (Hu and Zhao, 2026; Hu et al., 2026; He et al., 2026; Machcha et al., 2026; Cocchieri et al., 2026; Lovering et al., 2025; Zhang et al., 2026).Source: §2 · p. 2

Cognitive biases and framing in LLMs

Prompt sensitivity, robustness, and consistency

3 FRAMEBENCH-HPS Dataset

FRAMEBENCH-HPS isolates recommendation changes induced by probability-equivalent gain-loss framing using shared scenario abstractions, paired probability conditions, and quality-controlled prompts. Its construction spans four high-stakes domains and 2,880 prompts.

3 FRAMEBENCH-HPS Dataset

  • The benchmark is designed to isolate decision-support changes induced by probability-equivalent gain-loss framing.Source: §3 · p. 2

3.1 Scenario Abstraction

  • Each scenario represents a consequential decision across healthcare, finance, education/career, or legal/consumer-rights domains, with structured fields separating scenario meaning from surface wording.The abstraction fixes the decision target, benefits, downsides, probability-sensitive outcome, expected decision type, and high-stakes factor across framing variants.Source: §3.1 · p. 3

3.2 Probability and Framing Conditions

  • Each base scenario uses positive-outcome probabilities of 20%, 40%, 60%, and 80%, paired with neutral, gain, and loss conditions.Gain frames state the favorable-outcome probability; loss frames state its complement, preserving the same binary probability structure.Source: §3.2 · p. 3

3.3 Prompt Generation and Quality Control

  • Human screening selected 240 scenarios from 288 candidates, then human semantic review and structural checks validated the paired formulations before inclusion in the 2,880-prompt benchmark.The review checked equivalent decision problems, complementary probabilities, and unchanged decision targets, context, and consequences.Source: §3.3 · p. 3

4 Experimental Setup

The setup evaluates seven LLMs on paired gain- and loss-framed inputs using structured recommendation and scalar outputs. It defines framing-comparison measures and applies scenario-level validation and statistical analysis.

4 Experimental Setup

  • Seven LLMs answer independently issued benchmark prompts with a recommendation, risk, actionability, confidence, and brief explanation; temperature is 0.The requested recommendation categories are yes, no, or uncertain, and scalar fields use integer values from 1 to 5.Source: §4 · p. 4
  • Responses are checked for valid recommendation categories, 1–5 scalar values, and an explanation; incomplete structured answers are not retained.For outputs with visible reasoning, a complete JSON object must appear after the closing </think> marker.Source: §4 · p. 4
  • Framing effects are measured as paired gain–loss differences, with scenario-level bootstrap intervals and sign-flip tests for signed scalar shifts.Positive risk shifts indicate higher model-assigned risk under loss framing, while positive actionability shifts indicate higher actionability under gain framing; the fields are treated as structured outputs, not validated psychological measures.Source: §4 · p. 4

5 Results and Analysis

Across the seven-model evaluation, probability-equivalent gain–loss framing often changes recommendation categories, primarily through uncertainty, and shifts structured and human-rated responses. The sensitivity persists across conversational reformulations and exceeds same-frame generation variability, while probability canonicalization substantially reduces it.

5.1 RQ1 Framing Primarily Shifts Recommendations Through Uncertainty

  • 32.41% of complete groups changed recommendation category, versus 0.63% with direct yes–no reversals; 98.1% of changes involved uncertain.The sensitivity analysis using all valid gain–loss pairs yielded a similar 32.44% category-change rate.Source: Table 1 · p. 5, §5.1 · p. 5
  • Loss framing raised model-assigned risk by 0.285 points on average, while the gain-minus-loss actionability difference was −0.230 and mean absolute confidence difference was 0.702.All seven models showed positive mean risk shifts; the reported signed differences were supported by scenario-level sign-flip tests.Source: §5.1 · p. 5
  • Human ratings increased from 3.24 to 3.51 for perceived risk and from 3.23 to 3.55 for directiveness under loss framing; expressed certainty showed no significant directional difference.The reported differences were 0.27 for risk and 0.32 for directiveness; certainty difference was 0.08 (pHolm = .312).Source: §5.1 · p. 5

5.2 RQ2 Framing Sensitivity Persists Beyond Fixed Prompt Templates

  • Recommendation-category change rates varied across the seven models, from 63.33% for GPT-4o-mini to 21.58% for Qwen3-VL-235B; risk shifts decreased as positive-outcome probability increased.Mean loss-minus-gain risk shifts were 0.535, 0.287, 0.209, and 0.109 at 20%, 40%, 60%, and 80% probabilities, respectively.Source: §5.2 · p. 6, Figure 2 · p. 7
  • Conversational prompts retained framing sensitivity: category changes were 34.20% versus 36.81% for original templates, while cross-frame disagreement reached 34.32% versus 7.12% same-frame disagreement.The conversational-minus-original category-change difference was −2.60 percentage points, with a scenario-bootstrap interval including zero.Source: §5.2 · p. 6
  • Under conversational prompting, category-change rates ranged from 56.5% for GPT-4o-mini to 14.1% for Gemini 2.5 Pro among the three selected models.Claude Sonnet 4.6 had a 32.0% matched-repeat category-change rate.Source: §5.2 · p. 6

5.3 RQ3 Probability Canonicalization Substantially Reduces Framing Sensitivity

  • 6.68% category changes under probability canonicalization compared with 34.20% under conversational prompting; the mean risk shift fell from 0.363 to −0.010.The canonicalized category-change rate was close to the 7.12% same-frame repeated-generation disagreement under conversational prompting; the gain-minus-loss actionability shift was 0.015.Source: §5.3 · pp. 6–7, Table 2 · p. 7
  • Frame-robust prompting reduced category changes to 22.83%, a smaller reduction than canonicalization; direct yes–no reversals remained uncommon across all four conditions.The category-change reduction from the conversational baseline was 11.37 percentage points; the authors do not interpret the 1.56% reversal rate under frame-robust prompting as a reliable increase.Source: §5.3 · p. 7, Table 2 · p. 7

6 Discussion

Probability-equivalent gain-loss framing changes recommendations mainly by shifting models into or out of uncertainty, while direct yes-no reversals remain rare. The effects vary across models and probabilities, persist under conversational reformulations, and are substantially reduced by probability canonicalization.

Framing sensitivity and recommendation uncertainty

  • 32.41% of complete groups change recommendation category, but only 0.63% show direct yes-no reversals; 98.1% of 2,153 changes involve uncertainty.The dominant shift is between recommending, withholding a recommendation, and advising against an action, so reversal-only evaluations miss most observed sensitivity.Source: §6 · p. 7
  • Loss framing increases model-assigned risk alongside higher human ratings of risk presentation and directiveness, but structured fields do not uniformly track perceived properties.Risk has a relatively strong association between structured and perceived measures (ρ = .704), while directiveness and actionability are less closely associated (ρ = .307).Source: §6 · pp. 7–8

Robustness beyond prompt templates

  • Category-change rates span 21.58% for Qwen3-VL-235B to 63.33% for GPT-4o-mini, while risk shifts decrease from 0.535 at 20% probability to 0.109 at 80%.Framing sensitivity is heterogeneous across evaluated models and probability levels, rather than a uniform effect.Source: §6 · p. 8
  • Under conversational prompts, category-change rates are similar to original templates (34.20% versus 36.81%), while cross-frame disagreement (34.32%) exceeds same-frame repeated-generation disagreement (7.12%).This separation indicates the effect cannot be attributed solely to ordinary repeated-generation variability.Source: §6 · p. 8

Probability representation as an intervention

  • Canonicalization lowers conversational category change from 34.20% to 6.68%, near the 7.12% same-frame baseline, and reduces the mean risk shift from 0.363 to −0.010.Frame-robust prompting also improves consistency, but its category-change rate remains 22.83%, with residual differences in risk, actionability, and confidence.Source: §6 · p. 8
  • Canonicalization makes complementary probabilities explicit, but greater consistency does not establish better decision quality or justify ignoring preferences and contextual information.Probability-framing robustness is one component of reliable decision support alongside correctness, calibration, preference alignment, and uncertainty communication.Source: §6 · p. 8

7 Conclusion

The paper finds that probability-equivalent gain-loss framing produces substantial variation in high-stakes LLM decision support, mostly through shifts into or out of uncertainty rather than direct reversals. Framing effects persist across conversational reformulations and exceed repeated-generation variability, while probability canonicalization reduces disagreement toward the repeated-generation baseline.

7 Conclusion

  • Gain-loss framing produces substantial recommendation variation across models and high-stakes domains, primarily by shifting recommendations into or out of uncertainty.Direct yes-no reversals remain rare.Source: §7 · p. 8
  • Framing effects persist under conversational reformulations and substantially exceed same-frame repeated-generation variability.Source: §7 · p. 8
  • Probability canonicalization reduces recommendation disagreement to approximately the repeated-generation baseline and largely removes the corresponding risk shift.Source: §7 · p. 8

Limitations

The evaluation is bounded by synthetic scenarios, selected probability levels, and controlled framing templates; it does not establish decision correctness or downstream user benefit.

Benchmark scope

  • The benchmark uses synthetic scenarios, four probability levels, and controlled gain-loss templates; multi-turn conversations and other contextual uncertainty remain unevaluated.Conversational robustness covers 32 held-out scenarios and three model families.Source: Limitations · p. 9

Decision quality

  • Most scenarios lack complete utility functions, user-preference models, or normative answers, and domain correctness requires expert assessment.Source: Limitations · p. 9

Intervention scope

  • The mitigation evaluation does not establish domain-expert correctness or downstream user benefit, and its consistency gain applies to the tested normalization procedure.Canonicalization makes both complementary probabilities explicit.Source: Limitations · p. 9

Ethical Considerations

The study uses synthetic scenarios without personal records or demographic profiling and evaluates behavioral consistency, not real-world decision instructions or safety.

Ethical Considerations

A Dataset Examples

The benchmark varies probabilistic wording while holding the underlying scenario and decision target fixed, then evaluates model responses and human judgments across sampling, annotation, and robustness procedures.

B Query Prompt Template

  • Each prompt uses the same instruction format, and each frame variant is queried independently without showing the model its group’s other variants.The query combines a framed scenario with a fixed JSON-output instruction.Source: B Query Prompt Template · p. 11

B.1 Frame-Specific Scenario Templates

  • The frame templates distinguish a stated positive-outcome probability, a favorable-outcome chance, and the complementary chance that the outcome will not occur.The placeholders define {P} as the positive-outcome probability and {100-P} as its complementary negative-outcome probability.Source: §B.1 · p. 11

B.2 Output Instruction

  • The fixed output instruction requests recommendation, risk level, actionability, confidence, and a brief explanation in JSON format.The recommendation is constrained to yes, no, or uncertain; the scalar fields use 1–5 values.Source: §B.2 · p. 11

A Dataset Examples

  • Table 3 examples preserve each scenario’s actor, decision target, and probability structure while varying the probabilistic wording.The examples illustrate a 60% positive-outcome probability; the full dataset uses 20%, 40%, 60%, and 80%.Source: A Dataset Examples · p. 11, Table 3 · p. 12

B.3 Field Definitions

  • The protocol uses structured output fields for recommendations and scalar responses, with Table 4 identified as their definitions.The supplied passage identifies the table’s scope but does not provide the field definitions themselves.Source: §B.2 · p. 11, Table 4 · p. 12

C.1 Sampling

  • The human-evaluation sample contains 140 gain-loss pairs, balanced across four domains, four probability levels, and seven models, with distinct base scenarios.Gain and loss response order is randomized and balanced at 70 pairs in each response position.Source: §C.1 · p. 12

C.2 Annotation Procedure

  • Three blinded annotators independently rate both responses on risk presentation, directiveness, and expressed certainty using 1–5 scales.Response-level scores are the median of the three ratings, and gain-loss medians are paired by scenario; the human evaluation comprises 280 response-level units.Source: §C.2 · p. 12, §C.3 · p. 13

D Robustness and Mitigation Prompt Construction

  • Conversational rewrites preserve scenario facts and numeric values, while canonicalization states both complementary probabilities and the frame-robust instruction directs advice to rely on probabilities and scenario facts.The rewrites must not add emotion, recommendation, evaluative cues, or rationale.Source: D Robustness and Mitigation Prompt… · p. 13

E Additional Benchmark Results

The additional-results section identifies confidence perturbation results, while the reproducibility details specify retained benchmark responses, robustness-test sampling, and inference settings.

E Additional Benchmark Results

  • Table 6 reports probability-level self-rated confidence perturbation, with intervals resampling base scenarios and absolute differences indicating magnitude rather than direction or internal uncertainty.The supplied passages identify the table’s scope but do not provide its cell values.Source: Table 6 · p. 13, E Additional Benchmark Results · p. 13

F Statistical and Reproducibility Details

  • Strict validation retains 20,079 responses, 6,644 complete neutral-gain-loss groups, and 6,658 valid gain-loss pairs.The earlier stored-field parsing policy reproduces 6,664 complete groups; the strict audit excludes 21 Qwen and 11 Gemini responses.Source: F Statistical and Reproducibility… · p. 13
  • Each robustness-test condition contains 2,304 responses and 1,152 matched-repeat gain-loss pairs, with intervals resampling 32 held-out scenarios.The full three-model evaluation contains 9,216 test responses and 768 development responses; all 9,984 pass the predefined schema checks.Source: F Statistical and Reproducibility… · p. 13

G Model and Inference Configurations

Loading 2610.29294v1…