CoBe: A Benchmark For Conversational Counterfactual Text Editing

1 INTRODUCTION

• The task requires inferring which narrative facts remain fixed, applying the hypothetical intervention, and revising its commonsense downstream effects without explicit causal instructions.Counterfactual reasoning changes events downstream of the intervention while keeping causally prior background factors fixed.Source: §1 · pp. 2, 4
• Associational generation can change background facts to match statistical correlations, risking downstream errors in safety-critical workflows such as healthcare.The paper also notes that naive counterfactual data augmentation can introduce new biases through spurious correlations.Source: §1 · p. 3
• CoBe contains 2,000 samples, including 200 public examples, pairing short scenarios with single-turn colloquial prompts for counterfactual edits.The scenarios span domains including science, engineering, business, everyday life, healthcare, and legal, policy, and HR settings.Source: §1 · pp. 3–4
• Frontier models average 58.05%, with Gemini-3.1-Pro favoring correlational consistency and GPT-5.4-Pro struggling to incorporate scenario background.The benchmark reports these as model-specific error profiles.Source: §1 · p. 4
• CoBe is presented as the first testbed for zero-shot conversational counterfactual edits grounded in formal causal semantics, with criteria allowing multiple valid responses.The authors say its criteria and error profiles could support integration as a training signal in fine-tuning and inference-time pipelines.Source: §1 · p. 4

General benchmarks

• General benchmarks sometimes test counterfactual reasoning indirectly, leaving a gap in rigorous tests of formally grounded, open-ended text edits.The comparison table is labeled as covering prior benchmarks.Source: §2 · p. 4

Counterfactual story rewriting

• TimeTravel and PASTA use short, linear ROCstories and crowd judgments or similarity metrics, while CoBe uses expert-verified criteria that permit multiple valid rewrites.Prior work notes that similarity metrics correspond only weakly with counterfactual validity; Mu and Li assume invariant story parts are pre-specified.Source: §2 · p. 4

Other causal reasoning tasks

• CLadder and CounterBench query causal structures stated explicitly, whereas CoBe tests narrative revision without a supplied graph or explicit editing instructions.Other text-based causal benchmarks address complementary tasks, including multiple-choice counterfactuals and causal-relation assessment.Source: §2 · p. 5

3 COBE: A DATASET FOR COUNTERFACTUAL TEXT EDITING

• CoBe began with 106 core examples, generated 25 variations per core into 2,650 augmented examples, then author review and expert feedback removed 650, leaving 2,000 retained samples.Source: §3 · p. 5
• Each scenario has three conversational query phrasings, and CoBe varies the causal structures connecting story events.The authors average responses across phrasings to reduce ambiguity about the intended counterfactual interpretation.Source: §3 · p. 6

3.1 EVALUATION METHODOLOGY

• Per-story criteria check whether unaffected facts remain, required changes occur, and causal connectors fit the rewritten scenario.The judge marks a response correct only when all three checks pass.Source: §3.1 · pp. 6–7, Figure 5 · p. 7
• The criteria allow multiple valid rewrites rather than requiring reference matching, and the judge agrees with expert raters at Cohen’s κ = 0.812.The authors report that E2 accepts removal as well as modification of a listed fact, so an incomplete rewrite can pass; they do not claim robustness to deliberate omission.Source: §3.1 · p. 7

Overall results

• 61.34% is the highest accuracy across models averaged over three query phrasings, and the authors characterize overall performance as weak.Table 5 reports model performance averaged across the three queries.Source: Table 5 · p. 8, §4 · p. 8

Model scale

• Llama-70B performs close to larger models, including a reported 6.5% gap with Claude, so parameter scale does not consistently correspond to stronger counterfactual reasoning.The authors argue that gains from scaling may not improve this capability to the same extent.Source: §4 · p. 8

Performance across evaluation criteria

• Incorrectly editing non-descendant facts is the most consistent failure pattern; failures to update downstream facts or use appropriate connectors are generally lower.The authors suggest mistaken correlations may contribute to non-descendant edits.Source: §4 · p. 8

Performance across graph types

• Models show lower accuracy on Diamond-like and Fork-like graphs, where intervention effects must propagate across downstream branches.The authors also report that models tend to misread correlated graphs as causal structures.Source: §4 · p. 8, Figure 7 · p. 9

Performance across query phrasings

• Accuracy differs negligibly across the three paraphrased query phrasings.The authors interpret this as consistent with limited native conversational counterfactual reasoning.Source: §4 · p. 8, Figure 8 · p. 9

Model biases

• Model error profiles differ: GPT struggles with numerical updates, Claude with latent-state invariance, and Gemini with avoiding upstream changes.The authors also report that Gemini maintains grammatical structure especially well.Source: §4 · p. 9

Prompt ablations

• Frontier models improve by around 3% with a simple explicit causal instruction, yet score 81.3% and 76.1% under a detailed step-by-step recipe.Additional conversational prompts improve frontier-model scores by around 5% on average but lower scores for smaller models.Source: §4 · p. 9

5 CONCLUSION

• CoBe provides a causally grounded benchmark with diverse scenarios, graph structures, and novel failure modes to help researchers investigate model blind spots.The dataset is structured for use in developing better systems.Source: §5 · p. 9
• CoBe performance should not be taken as evidence of general counterfactual reasoning, and the authors identify possible author bias in sample curation despite diversity and human quality checks.Extending the benchmark to domains such as vision is a future direction.Source: §5 · p. 10

AI USE STATEMENT

• LLMs supported scenario variation, benchmark evaluation, coding, debugging, and editing; authors reviewed AI-assisted material and independently checked the code and results.The authors state that they did not rely on generative AI to draw the paper’s insights and conclusions.Source: AI USE STATEMENT · p. 10

A.1 STRUCTURAL CAUSAL MODELS AND THE PEARL CAUSAL HIERARCHY

• SCMs formalize counterfactual edits by overriding the intervened variable’s mechanism while leaving other mechanisms and the exogenous distribution unchanged.Factual and counterfactual scenarios share background context; changes to descendants depend on structural mechanisms and context.Source: §A.1 · p. 16

A.2 SUBJUNCTIVE CONDITIONALS AND CONVERSATIONAL COUNTERFACTUALS

• Conversational subjunctive questions are intended to signal counterfactual reasoning: preserve background factors and prior events, changing only downstream effects without requiring formal instructions.The paper distinguishes these from indicative associational questions.Source: §A.2 · p. 17

B.1 EXAMPLES

• The examples illustrate criteria for preserving unaffected facts, revising intervention-dependent facts, and maintaining causal connectors; the illustrative answers are not scored or unique.Color highlights distinguish facts to preserve, the intervention, appropriate revisions, and errors.Source: §B.1 · p. 18

B.1.1 SAMPLE FAILURE 1: SPATIAL CONSEQUENCE

• In the spatial-consequence example, the rubric keeps Sato’s direction and the skyline’s relative position fixed while changing the sun’s position after the time shift.It does not require sunlight to remain visible at 6 PM.Source: §B.1.1 · p. 18

B.1.2 SAMPLE FAILURE 2: MISATTRIBUTING AN UNCHANGED EVENT

• Claude Opus 4.7 made the scandal ruin Vera’s reputation, changing the original fact that it ruined her opponent’s.The illustrative answer preserves the opponent’s scandal while stating that Vera lost.Source: §B.1.2 · p. 18
• The query stipulates both Vera’s charisma and her defeat, so the opponent’s scandal remains unchanged background rather than a cause to rewrite.The author-annotated reasoning says the query does not ask the model to predict the election from charisma.Source: §B.1.2 · p. 18
• Reassigning the scandal to Vera rationalizes the stipulated defeat but alters a fact the evaluation criteria say should not change.The criteria mark the opponent’s ruined reputation as unchanged.Source: §B.1.2 · p. 18

B.1.3 SAMPLE FAILURE 3: RETAINING AN INAPPLICABLE CONSEQUENCE

• The squash rewrite transfers the rain-related soccer cancellation to the squash session, although the outdoor field need not become dry and other reasons for cancellation remain possible.The intended indoor-squash interpretation makes the wet outdoor field irrelevant to the squash practice.Source: §B.1.3 · p. 19

B.1.4 SAMPLE FAILURE 4: UNSUPPORTED CAUSAL CONNECTOR

• The building rewrite preserves the council’s decision but wrongly uses the collapse risk as its reason, although the risk and beam fissure remain unchanged.The example requires no particular replacement connector and does not require a collapse or injury to occur.Source: §B.1.4 · p. 19

B.1.5 SAMPLE FAILURE 5: OVERLOOKING AN UNCHANGED PURPOSE

• Removing the rain does not remove the repair errand or the need to bring the umbrella, but the response treats rain as the only reason for carrying it.The umbrella is still needed for the stated repair purpose.Source: §B.1.5 · pp. 19–20

B.1.6 SAMPLE FAILURE 6: ALTERING A HISTORICAL FACT

• Changing Hari’s favorite color to blue does not change the red dress in the previously unseen photograph of his girlfriend.The preference changes, but the photograph records an unaffected historical fact.Source: §B.1.6 · p. 20

B.1.7 SAMPLE FAILURE 7: VIOLATING A QUANTITATIVE CONSTRAINT

• 5 gallons per minute inflow versus 8 gallons per minute outflow yields a 3-gallon-per-minute net loss, so the water level falls rather than rises.The faucet’s inflow remains fixed while the intervention adds a greater outflow.Source: §B.1.7 · p. 20

B.1.8 SAMPLE SUCCESS: MARKED CORRECT DESPITE MULTIPLE REWRITES

• Three alternative rewrites switch Lisa from reading to watching a movie while preserving her terminal location and three-hour flight delay.Headphones, pausing for announcements, and charging are optional elaborations, not required consequences; the bookmark may remain in the closed book.Source: §B.1.8 · p. 21

B.2 SAMPLE JSON

• The schema separates facts to preserve from facts to revise, permits multiple valid answers, and illustrates a falling water level when an 8-gallon-per-minute drain exceeds the 5-gallon-per-minute faucet inflow.The example’s stated flows imply a net loss of 3 gallons per minute while those rates hold.Source: §B.2 · p. 22

C QUALITATIVE FAILURE MODES

• The failure taxonomy covers overlooked reasons for actions, altered background facts, inapplicable consequences, and violations of physical or quantitative constraints.The examples include removing an umbrella despite an unchanged repair errand and retaining a rising water level despite greater outflow than inflow.Source: C QUALITATIVE FAILURE MODES · p. 23
• Other failures include inconsistent spatial consequences, unsupported causal connectors, and changing historical facts to match a new preference.The examples include facing a setting sun while walking east, assigning collapse risk as a reason to keep a building open, and changing a photographed dress color.Source: C QUALITATIVE FAILURE MODES · p. 23

D EVALUATION DETAILS

• The evaluation-details appendix points readers to infrastructure, evaluation prompts, human agreement, and additional results.These topics are referenced as material in Appendices D.1, D.2, D.4, and D.5.Source: D EVALUATION DETAILS · p. 23

D.1 INFRASTRUCTURE

• Open-source responses used 1–4 NVIDIA H100 GPUs; frontier-model responses used APIs, with available reasoning modes set to medium or the applicable default.The setup describes how responses were generated across the two model groups.Source: §D.1 · p. 23

D.2 LLM-JUDGE PROMPTS

• The LLM judge checks connector usage and the rewrite against preservation and change criteria, accommodating paraphrases, additional text, and multiple valid rewrites.The passage reports that agreement with human judgments is in Appendix D.4.Source: §D.2 · p. 23

D.3.1 SAMPLE CURATION AND AUGMENTATION

• Generated variations and their evaluation criteria were reviewed before inclusion, while development spot-checks did not determine whether examples were retained.Selection prioritized scenario quality, domain and causal-structure diversity, and clarity of evaluation criteria.Source: §D.3.1 · p. 26

D.3.2 EXTERNAL EXPERT QUALITY AUDIT

• Three independent causal-inference experts rated 100 randomly sampled candidates on coherence, grammar, counterfactual validity, and collection diversity using a five-point scale.They were not shown the preservation and revision criteria; their audit complemented the authors’ separate checks of those criteria.Source: §D.3.2 · p. 27
• Reviewers identified contrived causal relationships in augmented examples, prompting further exclusions; altogether, 650 of 2,650 candidates were removed.The final dataset contains 2,000 scenarios.Source: §D.3.2 · p. 27, §D.3.1 · p. 27
• Experts rated the audited candidates 4.85/5 for coherence, 4.94/5 for grammar, 4.71/5 for counterfactual-query validity, and 5/5 for diversity.These ratings cover 100 audited examples, including some later excluded from the final dataset.Source: §D.3.2 · p. 27

D.4.1 SETUP

• Three expert annotators independently judged 250 model responses, and their majority judgment served as the reference for evaluating the LLM judge.Annotators saw the story, query, and response, but not the story-specific evaluation criteria.Source: §D.4.1 · p. 28

D.4.2 LLM JUDGE ALIGNMENT AND STATISTICAL SIGNIFICANCE

• 92.1% Accuracy and κ = 0.812 against human majority judgments supported using the LLM judge across the benchmark.The same comparison reported 82.5% Precision, 91.7% Recall, and 86.81% F1, with failure as the positive class.Source: §D.4.2 · p. 29

• Models struggle to keep scientific laws invariant when editing scenarios, whether those laws are stated explicitly or implicitly.Figure 9 presents accuracy across different failure modes.Source: §D.5 · p. 29, Figure 9 · p. 30
• Models often mistake correlational relationships for causal ones, with substantial failure rates when scenarios invite that misinterpretation.Source: §D.5 · p. 30
• Models also sometimes fail to propagate intervention effects to downstream variables, consistent with above-average failure on Diamond- and Fork-like graphs in Fig. 7.Source: §D.5 · p. 30

E PROMPT ABLATION EXPERIMENTS

• The four tiers range from original conversational prompts to progressively explicit guidance; Tiers 1–2 leave editing rules implicit, while Tiers 3–4 provide them.Tiers 3–4 are instruction-assisted controls outside the primary task setting.Source: E PROMPT ABLATION EXPERIMENTS · p. 30
• All four tiers were evaluated on the same 500 randomly sampled examples for GPT-5.4-Pro, Gemini-3.1-Pro, and Qwen-2.5-7B.Scores were averaged over the corresponding templates for Tiers 1 and 2.Source: E PROMPT ABLATION EXPERIMENTS · p. 30, Table 7 · p. 31

E.1 MORE EXPLICIT CONVERSATIONAL PROMPTS (TIER 2)

• Averaging over D–G, GPT improves from 66.2% to 71.7% (+5.5 percentage points) and Gemini from 58.7% to 64.4% (+5.7 points), while Qwen declines from 36.2% to 33.3% (−2.9 points).Stronger conversational cues therefore do not consistently improve performance, and substantial failure rates persist.Source: §E.1 · p. 31, Table 7 · p. 31

E.2 EXPLICIT PRESERVATION AND UPDATING INSTRUCTIONS (TIER 3)

• Relative to Tier 1, H1 raises GPT’s and Gemini’s pass rates by 7.3 and 8.4 percentage points, respectively, but Qwen falls to 25.0%.The additional gains over Tier 2 are 1.8 and 2.7 points for GPT and Gemini; explicit editing rules do not uniformly resolve failures.Source: §E.2 · p. 31, Table 7 · p. 31

E.3 A DETAILED CAUSAL RECIPE (TIER 4)

• With the detailed causal recipe, GPT reaches 81.3% and Gemini reaches 76.1%, while Qwen reaches 32.0%.These are increases of 7.8, 9.0, and 7.0 percentage points over H1, respectively; GPT’s result still leaves 18.7% of responses failing benchmark checks.Source: Table 7 · p. 31, §E.3 · p. 32
• Across tiers, supplying a causal procedure can substantially improve performance for some models, but it changes the assistance level from the primary conversational task.Source: §E.3 · p. 32