Source-linked AI summary
Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates
Katherine A. Keith, David Jensen, Brendan O'Connor
TL;DR
Observational causal estimates can be biased by confounders, motivating efforts to measure potential confounders from text. This review gathers and categorizes applications and methods for text-based adjustment, while highlighting unresolved representation and evaluation challenges.
Problem
Confounders can bias causal estimates, while work on using text to address them is scattered across communities.
Source: §1 · pp. 1–2Method
The review gathers and categorizes text-as-confounder applications and synthesizes approaches to measuring confounders, adjusting estimates, and evaluating methods.
Source: §1 · p. 2Results
Takeaways & Limitations
Text-based adjustment requires attention to causal identification, representation choices, and evaluation rather than assuming that available text captures confounders.
Source: §3.2 · p. 5, §4 · p. 6Without a positive identification strategy, causal effects cannot be estimated, making text measurement a secondary concern.
Source: §3.2 · p. 5
Abstract
from arXiv · showhide
Many applications of computational social science aim to infer causal conclusions from non-experimental data. Such observational data often contains confounders, variables that influence both potential causes and potential effects. Unmeasured or latent confounders can bias causal estimates, and this has motivated interest in measuring potential confounders from observed text. For example, an individual's entire history of social media posts or the content of a news article could provide a rich measurement of multiple confounders. Yet, methods and applications for this problem are scattered across different communities and evaluation practices are inconsistent. This review is the first to gather and categorize these examples and provide a guide to data-processing and evaluation decisions. Despite increased attention on adjusting for confounding using text, there are still many open problems, which we highlight in this paper.
1 Introduction
Confounding can bias estimates from observational data, and text may provide measurements of potential confounders that are otherwise unavailable. This review gathers work across communities and guides researchers through methods and evaluation choices for using text in causal adjustment.
1 Introduction
- Unaccounted confounders can produce strongly biased estimates and invalid causal conclusions.Source: §1 · p. 1
- Text may serve as a surrogate for potential confounders, or language itself may confound treatment and outcome.For example, social media posts may help measure occupation when structured records are unavailable.Source: Figure 1 · p. 1, §1 · p. 2
- Text is high-dimensional and must be transformed into lower-dimensional representations, while meaningful representations and causal evaluation remain open challenges.Possible representations include filtered word counts, lexicon indicators, topic models, and embeddings.Source: §1 · p. 2
- The review focuses on text used as a causal confounder, rather than text as mediator, treatment, outcome, or a source for causal discovery.Source: §1 · p. 2
- The review gathers and categorizes applications, offers a decision flowchart, and summarizes causal estimators and evaluation approaches for researchers across NLP and causal inference.It covers matching, propensity-score weighting, regression adjustment, doubly robust methods, and causally driven representation learning.Source: §1 · p. 2
2 Applications
The review collects applications using text to adjust for confounding, including text as a proxy for unobserved variables and language as a confounder. Examples illustrate different choices of text representation and adjustment, while open problems include NLP disparities and ethical concerns.
2 Applications
- The review gathers applications using text to adjust for potential confounding, encompassing text as a surrogate and language itself as a confounder.Table 1 is explicitly not exhaustive; its authors describe a seed-paper and citation-based search process.Source: §2 · p. 2
- Kiciman et al. use prior Twitter posts, word counts, and stratified propensity-score matching to adjust for confounding when demographic data are unavailable.Their study reports decreased mentions of study habits and positive emotions, and increased mentions of potentially risky behaviors, associated with alcohol use.Source: §2 · pp. 2–3
- Some studies target specific confounders such as ideology or mood, while others condition on available text and assume low-dimensional summaries capture all possible confounders.Source: §2 · p. 3
- NLP systems can be inaccurate for low-resource languages and exhibit racial and gender disparities; how to mitigate these issues in text-derived confounders remains unclear.The ethical implications of predicting psychological indicators such as mental-health status from text are also questioned.Source: §2 · p. 3
- Figure 2 guides analysts from causal-diagram assumptions to NLP measurement and adjustment, with evaluation including sensitivity analysis, human assessment when appropriate, and recovery of true effects.Source: Figure 2 · p. 4
3 Estimating causal effects
Potential-outcomes and structural-causal-model frameworks offer complementary ways to reason about and estimate causal effects. Both foreground the challenge of confounding, while the structural framework uses graphs to guide identification and adjustment.
3 Estimating causal effects
- The two complementary frameworks emphasize, respectively, conceptualizing causal relationships and estimating the size or strength of causal effects.Source: §3 · p. 4
3.1 Potential outcomes framework
- Because the same unit’s treated and untreated outcomes cannot both be observed, the average treatment effect is biased by a naive difference in means when confounders affect treatment and outcome.Source: §3.1 · p. 4
3.2 Structural causal models framework
- Structural causal models represent variables as graph nodes and direct causal dependencies as directed edges, with interventions on treatment removing its incoming arrows.Source: §3.2 · p. 4
- The backdoor criterion identifies an adjustment set that blocks every backdoor path without including treatment descendants; without identification, causal effects cannot be estimated.Source: §3.2 · p. 5
- Causal graphs distinguish confounders, colliders, and mediators: conditioning on confounders can block bias, while conditioning on colliders can introduce it.Mediators require different adjustments to estimate the natural direct effect; the review asks whether text can encode confounders and colliders simultaneously and whether adjustment can isolate confounders.Source: §3.2 · p. 5, Figure 3 · p. 5
4 Measuring confounders via text
After drawing a causal graph, researchers can measure text confounders by specifying variables in advance or learning a representation inductively. Both routes raise unresolved questions about measurement error and the sensitivity of causal estimates to modeling choices.
4 Measuring confounders via text
- Researchers can either measure pre-specified confounders from text or learn a low-dimensional representation inductively for subsequent causal adjustment.Source: §4 · p. 5
- Pre-specified confounders can be measured with lexicons or supervised classifiers trained on annotated examples.Classifier-based adjustment is valid under the stated conditions that mental states are accurately recovered and no additional latent confounders remain.Source: §4 · p. 5
- Open measurement problems include error from imperfect text methods, proxy variables, other representations, calibration, and prevalence estimation.The review notes that effect restoration has been applied to text classifiers, while other representation and measurement-error issues remain open.Source: §4 · pp. 5–6
- Inductive approaches use representations such as bag-of-words, topic distributions, embeddings, or fine-tuned language models as confounders, though encoding all aspects of language remains difficult.Source: §4 · p. 6
- Causal estimates may change with representation, preprocessing, and hyperparameter choices, motivating sensitivity analyses across modeling specifications.Source: §4 · p. 6
5 Adjusting for confounding bias
The section reviews ways to adjust for confounding when conditioning on all confounders is impractical, including methods that use text-derived measures. It outlines assumptions and open questions for these approaches.
5 Adjusting for confounding bias
- Backdoor adjustment estimates the causal quantity of interest using variables that satisfy the backdoor criterion, but conditioning on all confounders can be impractical in high-dimensional text settings.The review's method overview is not exhaustive; it directs readers to more extensive guides.Source: §5 · p. 6
5.1 Propensity scores
- Propensity scores estimate treatment probability from possible confounders, and IPTW weights units to calculate the average treatment effect.IPTW emphasizes treated units who were initially unlikely to receive treatment.Source: §5.1 · pp. 6–7
5.2 Matching and stratification
- Matching seeks treatment and control groups with similar confounder assignments, while propensity-score stratification compares units within score intervals.Text matching requires choosing a representation, distance metric, and algorithm; estimates may be sensitive to such choices as calipers and the number of controls.Source: §5.2 · p. 7
5.3 Regression adjustment
- Regression adjustment models conditional outcomes and uses them to predict counterfactual outcomes under treatment and control.The predictions are used in the regression-adjustment estimate.Source: §5.3 · p. 7
5.4 Doubly-robust methods
- Doubly robust methods model both treatment and outcome, and the effect estimate is unbiased if either model is unbiased.A-IPTW combines propensity scores and conditional outcomes; TMLE updates the conditional outcome estimate using a regression on propensity weights and the outcome estimate.Source: §5.4 · p. 7
5.5 Causal-driven representation learning
- Causal-driven representation methods update text representations using treatment and other causal information, but remain un compared on common benchmarks and may be sensitive to modeling choices.Examples combine representation learning with outcome prediction, propensity scores, matching, or causal estimators.Source: §5.5 · pp. 7–8
6 Human evaluation of intermediate steps
The section recommends human-interpretable checks of intermediate steps in text-based causal estimation, especially balance diagnostics and judgments of treatment propensity.
6 Human evaluation of intermediate steps
- When possible, practitioners should evaluate intermediate pipeline steps with interpretable balance metrics and/or human judgments of treatment propensity.In matched samples, balance ideally makes the confounder distributions alike between treatment groups.Source: §6 · p. 8, §6.1 · p. 8
6.1 Interpretable balance metrics
- Balance can be checked with standardized differences in means, topic-level comparisons, string-kernel similarity, or known structured confounders.Human evaluation can rate treatment propensity for individual items or assess similarity within matched pairs; the section identifies standardization of such experiments as an open problem.Source: §6.1 · p. 8, §6.2 · p. 8
7 Evaluation of causal methods
Because real-world causal effects are typically unknown, the section presents evaluation as an open challenge and discusses constructed observational studies and semi-synthetic data as possible approaches.
7 Evaluation of causal methods
- Real-world causal effects are typically unknown, and synthetic evaluations can lack unknown unknowns and allow many researcher degrees of freedom.The review encourages constructed observational studies or semi-synthetic datasets, while noting that latent text confounders make realistic evaluation data difficult to create.Source: §7 · pp. 8–9
7.1 Constructed observational studies
- Constructed observational studies compare observational estimates with randomized estimates from similar participants and settings.For text extensions, the review proposes collecting participants’ essays or public posts, or using observational data with covariates to assess estimates made from text alone.Source: §7.1 · p. 9
7.2 Semi-synthetic datasets
- Semi-synthetic datasets use real covariates and synthetic treatment and outcome, but researchers may choose metadata their methods can recover.The review proposes competition-style evaluation in which one group creates text-confounder data and others evaluate methods without seeing the data-generating process.Source: §7.2 · p. 9
8 Discussion and Conclusion
The conclusion identifies opportunities for causal observational studies using text while warning against adjustment with all available text absent human validation or supervision.
8 Discussion and Conclusion
- The authors caution against using all available text for causal adjustment without human validation or supervision, because potential errors cannot otherwise be diagnosed.They describe the lack of ground truth in causal applications as making it difficult to distinguish modeling errors and forking paths from true causal effects.Source: §8 · p. 9