FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect

1 Introduction

• The benchmark varies framing at the root, propositional, and global levels while holding factual evidence fixed.It also includes plain and negated forms, so the gold response can flip while the named option stays fixed.Source: §1 · pp. 1–2
• FramingQA evaluates framing effects across law, medicine, finance, and robotics, addressing a gap in benchmarks that focus on social pressure or context added outside the question’s semantic structure.The paper distinguishes these effects from sycophancy, which it describes as socially driven prioritization of user approval over accuracy.Source: §1 · pp. 1–2
• Models can show strong average accuracy yet near-zero compositional robustness across differently framed questions.The benchmark is designed to expose cases where per-question accuracy does not ensure consistent answers to the same facts.Source: §1 · p. 2

2 Compositionality in framing effect

• The framing effect is a response change under question reframing while the underlying evidence and the fact it determines remain fixed.The paper distinguishes framing—the manipulation of phrasing—from the framing effect, the measurable outcome change induced by that variation.Source: §2 · p. 2

2.1 Semantic Systematicity

• Semantic systematicity is tested through polarity consistency and paraphrase invariance under meaning-preserving transformations.Textual domains use polarity reversal and candidate substitution; paraphrases vary question form at fixed stance, or premise form at the propositional level.Source: §2.1 · pp. 3–4

2.2 Structural Interpretation

• Root bias presupposes a candidate conclusion, propositional bias supplies a rationale as grounds for a neutral question, and global bias combines both.For binary yes/no questions, propositional and global levels are inherently indistinguishable; the benchmark separates them by whether the rationale is grounds for the claim or background to it.Source: §2.2 · p. 4

3 The FramingQA Benchmark

• The benchmark contains 756 contexts and 54,094 biased variants across law, finance, medicine, and robotics.The textual and open-ended robotic tasks use different question formats, with no neutral reference for binary yes/no questions.Source: §3 · p. 4

3.1 FramingQAlaw: Legal Reasoning

• Legal tasks cover issue-spotting and hearsay rule application, with hearsay exceptions deliberately excluded to isolate sensitivity to prompt framing.Issue-spotting focuses on consumer, court, employment, and tort categories; the hearsay task uses core definitional criteria for consistency across jurisdictions.Source: §3.1 · p. 5

3.2 FramingQAfin: Financial Reasoning

• Finance cases derive from public Personal Finance & Money Stack Exchange posts, shortened and redacted before assignment to six categories.The category assignment is automatic, and snippets are truncated to 120 words.Source: §3.2 · p. 5

3.3 FramingQAmd: Medical Reasoning

• Medical cases are challenging clinical reports framed by foregrounding selected diagnostic explanations while keeping the case evidence fixed.The cases retain diagnostic ambiguity, including overlapping symptoms and test results.Source: §3.3 · pp. 5–6

3.4 FramingQArobo: Robotic Reasoning

• Robotic instruction-following tasks use household activities in simulation to test behavior under biased prompt stances with the task specification unchanged.Completion depends on semantic understanding, planning order, spatial grounding, and temporal dependencies.Source: §3.4 · p. 6

3.5 Annotation

• Domain experts set gold labels and rationales, while an LLM generates candidate phrasings that experts accept or reject.The model does not determine correctness; practitioners or domain-experienced annotators support the benchmark’s domain-specific review.Source: §3.5 · p. 6

4.1 Setup

• Nine open-source models spanning four families are evaluated across law, medicine, finance, and robotics.The models range from 3.8B to 70B parameters.Source: §4.1 · p. 6
• Textual tasks use binary or four-class decisions grounded in fixed contexts, while robotics tests instruction following in simulation.Source: §4.1 · p. 6
• Models use greedy decoding, one turn per question, and no in-context examples.Source: §4.1 · p. 6

4.2 Evaluation Metric

• Textual exact-match accuracy includes mean scores across questions and strict scores requiring every variant in a set to be correct.Strict accuracy is zero for a context after any variant fails; expected accuracy estimates strict accuracy under independent variant errors.Source: §4.2 · p. 6
• Expected accuracy is computed by multiplying per-question accuracies within each context and averaging across contexts; observed strict accuracy above it indicates errors concentrate in the same contexts.Source: §4.2 · p. 6
• Robotic task completion is the fraction of instruction propositions satisfied within the episode horizon, averaged across bias conditions.The score ranges from 0 to 1 and is computed per episode.Source: §4.2 · p. 7
• Robotic evaluations also report steps as a measure of plan efficiency and replans as a measure of interpretation or execution instability.Both metrics have mean and strict scores for each condition.Source: §4.2 · p. 7

5 Results

• Strict accuracy under incorrect-bias framing rose with model scale within each family in both settings, most clearly in multiple choice.The binary trend was weaker and most visible through negation; gold-No accuracy tended to rise despite inconsistencies.Source: §5 · p. 7
• Despite moderate mean accuracy (binary 52.4–61.0%; multiple choice 48.0–60.3%), binary strict (all) accuracy was 0.0% for every model.Binary strict (pair) accuracy was 1.0–8.3%, while multiple-choice strict (pair) was 11.1–43.9% and strict (all) ranged from 1.4% to 35.2%.Source: §5 · p. 7
• In multiple choice, correct-bias mean accuracy was 70–90% versus 55–62% neutral, while incorrect-bias accuracy was 48–62%; binary gold-Yes accuracy exceeded gold-No under both framing directions, despite Phi-model exceptions under incorrect-bias framing.Binary has no neutral reference; strict multiple-choice accuracy was 54–77% for correct-bias versus 11–45% for incorrect-bias.Source: §5 · p. 7
• Observed strict accuracy exceeded independent-error expectations almost everywhere, by four to nine orders of magnitude in aggregate All conditions.In multiple choice, the gap was roughly 35–135× under incorrect-bias framing versus 2–14× under correct-bias framing.Source: §5 · p. 8, Table 4 · p. 9
• Propositional-level scores stayed comparatively robust under incorrect bias, while root and global scores fell sharply, indicating that global framing carries root-level effects rather than adding levels independently.The propositional effect contributes almost nothing once the question is also framed, although it is not weak in isolation.Source: §5 · pp. 8–9, Table 5 · p. 9

5.1 Results in Robotic Simulations

• In robotic simulations, correct-bias instructions completed more of the task than incorrect-bias instructions at every structural level (53–58% versus 49–53%).The mean gap peaked at 6 percentage points under global framing, was 5 points under root framing, and was 1 point under propositional framing; strict completion was 14% correct-bias versus 8.6% incorrect-bias.Source: §5.1 · p. 9, Table 6 · p. 10
• Across constrained robotic episodes, framing gaps persisted, while unconstrained episodes had nearly identical mean completion but separated under strict scoring.For unconstrained episodes, mean completion was 62% neutral, 62% correct-bias, and 63% incorrect-bias; strict completion was 17% correct, 13% incorrect, and 5.6% for all variants.Source: Table 6 · p. 10, §5.1 · p. 10
• Correct-bias instructions required fewer replans across structural levels under both mean and strict scoring; under strict scoring, incorrect-bias instructions required more steps to reach the same goal.The authors interpret the extra steps as detours rather than outright failures in the long-horizon embodied setting examined.Source: §5.1 · p. 10

• FramingQA is a benchmark for evaluating compositional framing effects in large language models across high-stakes domains.Source: §6 · p. 10
• Moderate mean accuracy can mask near-zero consistency, as models may fail under a different bias framing or phrasing of the same framing.Source: §6 · p. 10
• FramingQA covers English single-turn law, medicine, and finance tasks plus simulated robotic instruction following, using open models up to 70B with greedy decoding.The evaluation focuses on direct answers; task definitions are simplified to isolate framing sensitivity from expert reasoning.Source: §6 · p. 10

A Dataset Construction

• Each domain starts from a factually verified context and gold label, then generates biased variants plus an unframed question in MC and open formats.Source: A Dataset Construction · p. 12
• Textual contexts yield 54 + 2E binary and 28 + E MC questions; robotics uses six biased variants per episode plus a neutral instruction.Source: A Dataset Construction · p. 12

B Question variant templates

• At root and global levels, templates vary question phrasing; at propositional level, the question stays fixed while the premise varies.Source: Table 7 · p. 12

• 200 contexts span four LegalBench-aligned issue-spotting categories—consumer, courts, employment, and torts—with 50 contexts each.Source: §C.1 · p. 13
• Five mutually exclusive hearsay slices contain 14 statement-made-in-court, 19 nonassertive-conduct, 29 standard-hearsay, 12 non-verbal-hearsay, and 20 not-introduced-to-prove-truth instances.Source: §C.1 · p. 13

• Finance narratives are seeded from public help-post-style questions on Personal Finance & Money Stack Exchange, retrieved via its API.Source: §C.2 · p. 13
• Debt and repayment (34), banking and payments (43), and fraud and scams (25) form three finance narrative categories.Source: §C.2 · p. 13
• Credit reporting and score (17), housing finance (34), and investing and financial advice (47) are the other finance narrative categories.Source: §C.2 · p. 13

C.3 Medicine

• Medicine uses case-specific candidate diagnoses rather than a fixed label set; each gold diagnosis is the source report’s final diagnosis, with three plausible distractors across 100 cases.The distractors share presenting features with the gold diagnosis, so symptom overlap alone cannot resolve the case.Source: §C.3 · p. 13

D.1 Law

• A relationship-and-debt scenario asks whether the issue is consumer-related (+) or employment-related (−).Source: §D.1 · p. 14

D.2 Finance

• A bankruptcy question about one spouse’s debt is framed as debt and repayment (+) versus credit reporting and score (−).Source: §D.2 · p. 14

D.3 Medicine

• One cardiac case describes a mobile LVOT element with uncertain anatomical relation, then contrasts accessory mitral valve tissue (+) with papillary fibroelastoma (−).Source: §D.3 · p. 15

D.4 Robotics

• Robotics keeps the placement command fixed while root, propositional, and global framings cast the goal as decorating (+) or clearing (−) the living-room table.Source: §D.4 · p. 15