PaperlayerView exact revision on arXiv

Source-linked AI summary

ROCK: Causal Inference Principles for Reasoning about Commonsense Causality

Jiayao Zhang, Hongming Zhang, Weijie J. Su, Dan Roth

arXiv:2202.00436v2cs.CLcs.AIcs.LGstat.AP

TL;DR

Commonsense causality reasoning lacks a well-posed theoretical framework and may be confounded by co-occurrences. ROCK applies a potential-outcomes framework, using temporal propensities to balance confounding; its balanced scores generally outperform temporal and unadjusted estimators across datasets.

  • Problem

    CCR lacks a well-posed theoretical framework, and defining study units, treatments, and outcomes for natural-language descriptions is challenging.

    Source: §1 · p. 1, §2.2 · p. 2

  • Method

    ROCK formulates CCR with potential outcomes and balances preceding-event covariates by matching temporal propensities, using a modular zero-shot pipeline.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 4

  • Results

    Balanced scores generally outperform the temporal and unadjusted estimators across datasets, while temporality remains vulnerable to spurious correlations without balancing.

    Source: §1 · p. 2, §5.3.2 · p. 8

  • Takeaways & Limitations

    ROCK demonstrates the potential of a causal-inference perspective for zero-shot CCR and is on par with methods using external causal knowledge on some datasets.

    Source: §6 · p. 8

    The temporal-propensity estimator can remain biased even under perfect matching, and the authors’ argument for a reasonable estimate assumes ϱ ≫ 0.

    Source: §3.5 · pp. 4–5

Abstract

from arXiv · show

Commonsense causality reasoning (CCR) aims at identifying plausible causes and effects in natural language descriptions that are deemed reasonable by an average person. Although being of great academic and practical interest, this problem is still shadowed by the lack of a well-posed theoretical framework; existing work usually relies on deep language models wholeheartedly, and is potentially susceptible to confounding co-occurrences. Motivated by classical causal principles, we articulate the central question of CCR and draw parallels between human subjects in observational studies and natural languages to adopt CCR to the potential-outcomes framework, which is the first such attempt for commonsense tasks. We propose a novel framework, ROCK, to Reason O(A)bout Commonsense K(C)ausality, which utilizes temporal signals as incidental supervision, and balances confounding effects using temporal propensities that are analogous to propensity scores. The ROCK implementation is modular and zero-shot, and demonstrates good CCR capabilities.

1. Introduction

The introduction frames commonsense causality reasoning (CCR) as identifying plausible causes and effects in natural-language descriptions, while noting that temporal order alone can be confounded. It formulates CCR through causal inference and introduces ROCK’s temporal-propensity balancing approach.

1. Introduction

  • CCR asks whether one event plausibly causes another; temporal precedence alone can mislead when a preceding event such as hunger confounds the association.The introduction frames the task around ordinary reasonable judgments and illustrates how conditioning on a comparable intervention can shrink the estimated effect.Source: §1 · p. 1, Figure 1 · p. 1
  • The paper formulates CCR as estimating how intervening on E1 changes the likelihood of E2, and proposes balancing preceding-event covariates with temporal propensities.The introduction notes that temporal signals are essential but vulnerable to spurious correlations when covariates are not balanced.Source: §1 · pp. 1–2
  • ROCK is presented as a novel, modular, zero-shot framework for CCR that mitigates confounding by matching temporal propensities.The authors report that the framework demonstrates effectiveness, without specifying results in these passages.Source: §1 · p. 2

2. Background

The background connects CCR to established causal-inference frameworks but highlights the difficulty of defining study units, treatments, and outcomes in natural-language descriptions. ROCK adopts temporal principles and balanced covariates to address these challenges.

2. Background

  • Causal inference has been studied across human-population research, while textual causal inference often treats text as data; CCR instead uses language to convey meanings, complicating the definition of units, treatments, and outcomes.Related work also uses textual data as covariates or study units, unlike CCR’s natural-language descriptions.Source: §2 · p. 2

2.1. Existing Approaches

  • Prior CCR approaches use causal cues, conditional probabilities, language-model engineering, or other causal perspectives, but the authors claim theirs is the first causal perspective for CCR.Related examples include Granger causality, point-process event modeling, and simulation-based physical reasoning.Source: §2.1 · p. 2

2.2. Challenges of CCR

  • CCR methods can achieve strong empirical performance, but dataset artifacts and unresolved causal roles motivate a well-founded theoretical framework.The paper identifies study units, treatments, outcomes, and the meaning and stability of interventions as open challenges.Source: §2.2 · p. 2

2.3. Principles of the ROCK Framework

  • ROCK uses precedence as a temporal signal and compares events with balanced pretreatment covariates; its pipeline samples prior events and interventions, matches temporal propensities, then estimates the effect.The paper presents temporal signals as incidental supervision and notes that reasoning about temporality has made progress.Source: §2.3 · p. 2, Figure 2 · p. 3

3. The ROCK Framework

ROCK frames commonsense causality reasoning (CCR) as a potential-outcomes problem about how an event changes the temporal likelihood of another, then estimates that change using matched temporal propensities. Its framework treats preceding events as covariates and allows multiple plausible interventions, while acknowledging limits to the estimator.

3. The ROCK Framework

  • ROCK samples covariates and interventions, selects a matched intervention subset using temporal propensities, and estimates the causal score from that subset.Source: §3 · p. 3

3.1. The Central Question of CCR

  • CCR asks how intervening on E1 changes the likelihood that E2 follows, with the estimand interpreted as an average treatment effect.Source: §3.1 · p. 3

3.2. The Potential-Outcomes Framework

  • A thought experiment treats each person’s complete timeline as a study unit, while interventions include plausible counterfactual events beyond negation and may have multiple versions.Source: §3.2 · pp. 3–4

3.3. Balancing Covariates

  • Estimating the effect requires balancing pretreatment covariates under strong ignorability, using events before E1 but not events between E1 and E2.Source: §3.3 · p. 4

3.4. Matching Temporal Propensities

  • Because the true covariates are unknown, ROCK samples preceding events with large language models and uses temporal propensities as surrogate propensity scores for matching.Source: §3.4 · p. 4

3.5. Discussions on Temporal Propensity Matching

  • The estimator can remain biased even with perfect matching because temporal propensity vectors use one-dimensional marginals rather than the full joint distribution.Source: §3.5 · pp. 4–5
  • The proposed error analysis assumes dependence between potential outcomes and treatment assignment remains nonzero after conditioning on covariates, which the authors consider likely with diverse covariates.Source: §3.5 · p. 5

4. Implementation of ROCK

ROCK is presented as a modular implementation that estimates the causal effect of one event on another through event sampling, temporal prediction, intervention generation, and scoring. The implementation uses language models for these components and fine-tunes a temporal predictor on news-derived event pairs.

4.1. Components of ROCK

  • ROCK represents events as triples and takes E1 and E2 as inputs, returning an estimate of their causal effect through four pipeline components.Source: §4.1 · p. 5

4.2. Implementation Details

  • The event sampler prompts GPT-J to generate 100 events likely to precede E1, cropping each sequence at its first stop token.Source: §4.2 · p. 5
  • The temporal predictor estimates event order with masked language modeling and symmetrizes scores for the two possible temporal directions.Source: §4.2 · p. 5
  • To address low coverage in pretrained temporal prediction, the authors fine-tune RoBERTa on 397174 event pairs extracted from New York Times articles from 1987–2007.Source: §4.2 · p. 5
  • The intervention generator creates alternatives by masking event components and filling them with PolyJuice generations controlled by intervention types.Source: §4.2 · p. 6
  • The scorer estimates temporal propensities from predictor scores for sampled preceding events and candidate interventions, then forms vectors for the causal-effect estimate.Source: §4.2 · p. 6

5. Empirical Studies

The empirical studies evaluate ROCK on COPA and GLUCOSE-D1, finding that temporal-propensity-balanced estimators generally outperform unbalanced and temporal estimators. Ablations indicate that temporal fine-tuning helps, while normalization gains are usually marginal.

5.1. Setup and Details

  • Evaluation uses COPA-DEV, COPA-TEST, and a 153-instance GLUCOSE-D1 dataset, with each task selecting the higher-scoring of two choices.COPA has 100 development and 500 test instances; GLUCOSE-D1 is constructed from its test set.Source: §5.1 · p. 6

5.2. Design Choices and Normalizations

  • Design variants include temporal pre-filtering, score and propensity normalization, co-occurrence stabilization, and estimand normalization; normalization benefits vary by dataset.The pre-filter retains covariates judged to precede E1; the complete design-choice ablations are reported in Section 5.4.Source: §5.2 · pp. 6–7

5.3.2. DISCUSSION

  • Balanced estimators with temporal propensities generally outperform unbalanced and temporal estimators across the evaluated datasets.The paper reports best zero-shot results; without temporal fine-tuning, performance generally degrades.Source: Table 1 · p. 7, §5.3.2 · p. 8

5.3.1. A CONCRETE EXAMPLE

  • In one COPA-DEV example, the balanced score selects the correct choice, although other estimators also succeed there and balanced scores sometimes fail.On this instance, the unadjusted score is approximately −0.010 and the balanced score approximately 0.002; balanced scores dominate on average.Source: §5.3.1 · p. 7

5.3.2. DISCUSSION

  • ROCK outperforms self-talk but trails reported COPA results from instruction fine-tuning and n-gram ranking methods.The authors emphasize that ROCK uses vanilla-LM temporal information without task-specific fine-tuning and is more interpretable.Source: §5.3.2 · p. 8

5.4. Ablation Studies

  • Without temporality fine-tuning, almost all estimators perform poorly; the authors conclude that fine-tuning helps LMs extract temporal knowledge useful for CCR.The ablation uses pretrained RoBERTa-BASE without temporal fine-tuning and increases k to 30.Source: §5.4 · p. 8
  • Increasing covariate-set size generally improves performance when ϵ is reasonable, but added covariates can add noise when ϵ is too large.When ϵ is too small, added covariates may have little impact.Source: §5.4 · p. 8
  • Normalization improvements are generally marginal, although some normalizations benefit particular datasets.The ablation removes single normalization components; direct matching and temporality pre-filtering are reported as making no change.Source: §5.4 · p. 8, Table 2 · p. 8

6. Discussions and Open Problems

The paper presents ROCK as a novel zero-shot CCR framework that applies causal-inference ideas to commonsense reasoning. It identifies LM bias and propensity estimation as open challenges, alongside opportunities for improved prompting.

6. Discussions and Open Problems

  • ROCK is presented as a novel zero-shot CCR framework and the first attempt to incorporate causal-inference frameworks into commonsense reasoning.The authors report potential across datasets and performance on par with methods using external causal knowledge on some datasets.Source: §6 · p. 8
  • Implicit events and reporting biases in training data may bias LM outputs, while exact propensity computation would require new methods for extracting many-event temporal relations.The authors also identify prompt engineering and investigation of implicit biases as future work.Source: §6 · p. 8

A. Miscellaneous Proofs

The proof appendix restates a proposition about expected L2 error under perfect matching and invokes conditional variance decomposition.

A. Miscellaneous Proofs

  • Proposition 3.2 concerns expected L2 error under perfect matching and defines r as the difference between potential outcomes r1 and r0.The proof passage says the result follows from conditional variance decomposition.Source: A. Miscellaneous Proofs · p. 13
  • The proof uses the bound Var(r) ≤ 1, given that r lies in [0, 1].Source: A. Miscellaneous Proofs · p. 13

B. Additional Experiment Details

The appendix recommends small ϵ values as starting points, with the choice depending on ROCK implementation details and the CCR domain.

B.1. Rule-of-Thumb for Choosing ϵ

  • For a starting ϵ range, use (0.01, 0.1) when p = 1 or (0.005, 0.05) when p = 2.The optimal value depends on the ROCK components and CCR domain.Source: §B.1 · p. 13

B.2. Further Discussions on Temporality Fine-Tuning

Without temporality fine-tuning, balanced estimators reached a higher best score, but the difference was not statistically significant; estimand normalization may explain the pattern and can behave differently across datasets.

B.2. Further Discussions on Temporality Fine-Tuning

  • 0.58 without temporality fine-tuning exceeded 0.564 with fine-tuning for the best balanced estimators, but the gap was within one standard deviation of the random baseline (0.022).The authors therefore draw no statistically significant conclusion and suggest pretrained language models may already encode temporality.Source: §B.2 · p. 13
  • Estimand normalization (E) accounted for the unusually strong result without fine-tuning, yet was detrimental on GLUCOSE-D1.The authors therefore caution against including E when applying ROCK to a new dataset.Source: §B.2 · p. 13

B.3. Full Ablation on Normalizations

The appendix describes six normalization choices for ROCK, spanning temporal scores, propensity estimates, co-occurrences, and the estimand; some alter the target or stabilize estimates.

B.3. Full Ablation on Normalizations

  • Direct matching (D) matches vectors of temporal probabilities instead of forming propensity vectors from conditional probabilities.The authors describe this choice as weakly motivated but potentially easier to compute in some circumstances.Source: §B.3 · p. 13
  • Temporality pre-filtering (F) removes sampled covariates that the temporal predictor does not judge to precede the event being treated.The stated criterion retains X when f(X, E1) > f(S, E1).Source: §B.3 · p. 13
  • Score normalization (S) can scale the temporal score using the reverse-direction and null-event scores, with the null event represented by an empty string.The authors say this can stabilize estimates when both directional scores are near zero, though it usually differs little from omitting the null event.Source: §B.3 · p. 13
  • Propensity normalization (Q) normalizes probability estimates before forming temporal propensity vectors, using relative frequencies over events and event-intervention pairs.The passage defines P(X(0)) and P(X(0), A(1)) using those respective relative frequencies.Source: §B.3 · p. 13
  • Co-occurrence stabilization (C) averages the two event-order probabilities, yielding an alternative estimand based on co-occurrence rather than precedence.The authors characterize this as a weaker notion of causation in commonsense causality reasoning.Source: §B.3 · pp. 13–14
  • Estimand normalization (E) divides the temporal probability by the sum of the two cross-order joint probabilities.This normalization is applied to the temporal probability in the estimand.Source: §B.3 · p. 14

B.3.1. ABLATION RESULTS

Normalization changes generally yield marginal improvements, while estimand normalization can produce dataset-specific gains or losses; the authors offer cautious defaults for choosing among them.

B.3.1. ABLATION RESULTS

  • Normalization gains are generally marginal, including relative to using no normalizations, except for GLUCOSE-D1 where estimand normalization drives a larger gap.Without estimand normalization, the worst GLUCOSE-D1 result is 0.594 (+Q or +FQ).Source: §B.3.1 · p. 14
  • Estimand normalization has little or no effect on most datasets, but can help COPA-TEST without temporal fine-tuning and hurt GLUCOSE-D1 with fine-tuning.The authors recommend disabling E by default because its effect varies across datasets.Source: §B.3.1 · p. 14
  • The authors recommend enabling temporal score normalization, forming propensity vectors properly rather than using direct matching, and starting with no normalizations when uncertain.Pre-filtering and propensity normalization generally have little effect; co-occurrence stabilization may help when a weaker causation notion is desired.Source: §B.3.1 · p. 14

B.4. Full Examples

The appendix gives three worked ROCK examples, showing sampled covariates, generated interventions, propensity distances, and temporal probabilities for plausible event pairs.

B.4. Full Examples

  • Each worked example tabulates 50 sampled covariates, generated interventions, propensity-vector distances, and temporal probabilities.The appendix presents these as examples from the ROCK implementation.Source: §B.4 · p. 14
  • For the 72nd COPA-DEV instance, the example examines whether assigned homework plausibly precedes students passing notes or groaning.Separate tables give the full inference for each event pair, with matched interventions highlighted.Source: §B.4 · p. 14, Table B.6 · p. 17
  • For the 63rd COPA-DEV instance, two candidate causes—preparing to wash hands or clean the bathroom—are paired with putting on rubber gloves.The appendix provides separate full tables for the two candidate event pairs.Source: §B.4 · p. 15, Table B.8 · p. 19
  • For the 79th COPA-DEV instance, the examples compare a pocket filled with coins and a sewn pocket hole as possible causes of jingling while walking.Separate tables show the full inference for each event pair, with matched interventions highlighted.Source: §B.4 · p. 15, Table B.10 · p. 21
Loading 2202.00436v2…