How to Make Causal Inferences Using Texts

Naoki Egami; Christian J. Fong; Justin Grimmer; Margaret E. Roberts; Brandon M. Stewart

Source-linked AI summary

How to Make Causal Inferences Using Texts

Naoki Egami, Christian J. Fong, Justin Grimmer, Margaret E. Roberts, Brandon M. Stewart

arXiv:1802.02163v1Feb 6, 2018stat.MLcs.CLstat.ME

TL;DR

Text-based causal inference depends on discovering latent representations that make high-dimensional documents usable as treatments or outcomes, but discovery can compromise validity. The paper develops a conceptual framework and split-sample procedure to separate discovery from estimation, identifies analyst-induced SUTVA violations and overfitting as key risks, and applies the framework to text-as-outcome and text-as-treatment settings.

Problem
Text-based causal inference requires discovering latent measures from complex documents, but using the same data for discovery and estimation can create analyst-induced SUTVA violations and overfitting.
Method
The paper develops a conceptual framework centered on the codebook function g and uses a train/test split to discover g in one sample and estimate causal effects in another.
Results
The framework identifies analyst-induced SUTVA violations and overfitting as threats and demonstrates a procedure for causal inference with text as treatment or outcome.
Takeaways & Limitations
Sequential research designs can reserve separate data for discovering theoretically relevant measures and estimating their causal effects.
Takeaways & Limitations
The train/test split reduces efficiency, and the appropriate training and test-set sizes are difficult to determine in advance.

Abstract

from arXiv · show

New text as data techniques offer a great promise: the ability to inductively discover measures that are useful for testing social science theories of interest from large collections of text. We introduce a conceptual framework for making causal inferences with discovered measures as a treatment or outcome. Our framework enables researchers to discover high-dimensional textual interventions and estimate the ways that observed treatments affect text-based outcomes. We argue that nearly all text-based causal inferences depend upon a latent representation of the text and we provide a framework to learn the latent representation. But estimating this latent representation, we show, creates new risks: we may introduce an identification problem or overfit. To address these risks we describe a split-sample framework and apply it to estimate causal effects from an experiment on immigration attitudes and a study on bureaucratic response. Our work provides a rigorous foundation for text-based causal inferences.

1 Introduction

Text-based causal inference expands the evidence available to social scientists but requires explicit attention to how textual measures are discovered. The paper identifies identification and estimation risks from learning measures and proposes split-sample procedures to address them.

Text-based methods increasingly measure quantities from judicial opinions, propaganda, social media, and government documents, but causal applications often leave identification assumptions unstated.
The codebook function g compresses complex, high-dimensional documents into categories, topics, or dimensions used as measures of interest.
Learning g from the same data used for causal estimation can create dependence across units because treatments and responses influence the discovered measure.
Using the same documents for discovery and estimation can also overfit, mistaking sample-specific noise for a robust causal effect.
A train/test split separates discovery from estimation, addressing these identification and overfitting concerns while supporting applications with text as either outcome or intervention.

2 The Central Role of g, The Codebook Function

Causal inference with text requires compressing high-dimensional documents into a lower-dimensional representation. The paper calls the mapping g a codebook function because it links raw text to the latent property of theoretical interest.

The codebook function g compresses high-dimensional text into a low-dimensional measure because raw documents are typically unsuitable for social science inference.
Researchers may use g to represent latent properties such as occupational background when estimating how candidate biographies affect popularity.
The representation space Z may contain discrete categories, proportions, or continuous variables, which researchers often label for interpretation.
g connects raw text to the underlying property of interest and can be learned through manual coding or machine-learning methods.
Because coding schemes are commonly developed through iteration between coding rules and documents, the discovery process itself requires explicit attention.

1) We can (and often do) learn g from the data.

Researchers can learn g from data through reading samples, supervised classification, or unsupervised learning. The chosen representation should then be evaluated against the research question and its intended use.

Researchers can learn g by reading a sample of documents to determine how text should map into categories.
Supervised methods estimate g from hand-coded or otherwise labeled documents, while unsupervised methods discover low-dimensional representations from the data.
After defining a research question, analysts can evaluate g by labeling latent features, assessing measured scales, and examining classification accuracy.

2) There is no single correct g.

There is no application-independent correct g; useful representations instead balance interpretability, theoretical interest, fidelity to the text, and tractability. These properties can conflict, especially when broader labels improve theoretical relevance but reduce replicability.

The paper identifies interpretability, theoretical interest, label fidelity, and tractability as useful properties of g once a research question is specified.
Interpretability requires communicating what a discovered measure captures, because data-driven representations are not necessarily substantively interpretable.
Theoretical interest means that g operationalizes theoretical concepts and identifies causal effects bearing on observable implications.
Label fidelity requires evidence that the latent variable accurately captures the property implied by its label.
The four properties can conflict: broader labels may increase theoretical relevance while lowering fidelity and making results harder to replicate.

3 The Problem of Causal Inference with g

Causal inference with text requires compressing high-dimensional documents through a codebook function g, but discovering g from the same data used for estimation creates identification and estimation risks. The section develops these risks through SUTVA-based notation and stylized randomization examples.

Representing text: The codebook function g maps high-dimensional text into lower-dimensional categories, topics, dimensions, or feature vectors used as treatments or outcomes.Possible forms include hand-coded categories, latent scales, dictionary counts, and crowd-sourced content measures.
Causal assumptions: Standard causal inference assumes SUTVA, random treatment assignment, and positivity, with proper randomization guaranteeing the latter two assumptions.SUTVA means each unit’s response depends only on its own treatment, without interference or hidden treatment versions.
Causal estimands: ATE_k = E[g(Y_i(1))_k − g(Y_i(0))_k] expresses the average treatment effect for category k after text outcomes are mapped through g.The equivalent representation is E[z_i,k(1) − z_i,k(0)], where z_i,k denotes category k for unit i.
Identification concerns: When documents used to discover g overlap with documents used for estimation, treatment assignments can make g depend on multiple units and violate SUTVA.This analyst-induced dependence is called an Analyst Induced SUTVA Violation because the discovery process creates it even when the original experiment satisfies SUTVA.
Identification concerns: Re-randomization can produce different discovered categories, making category stability unavailable when classifications are estimated from data rather than fixed before the experiment.The stylized example contrasts categories such as Candidate Morals and Polarization with Taxes and Immigration under different treatment assignments.
Estimation concerns: Using the same documents to discover g and estimate effects can overfit, mistaking sample-specific effects for effects that hold in the population.The risk increases because text permits flexible searches over codebooks and can resemble recoding variables until statistical significance appears.

4 A Train/Test Split Procedure for Valid Causal Inference with Text

The paper recommends separating discovery of the text codebook g from causal-effect estimation with a random training/test split. This design addresses analyst-induced SUTVA violations and overfitting while preserving inductive measurement and enabling causal estimands for text outcomes or treatments.

Predefining g avoids dependence between discovery and effect estimation but can miss concepts expressed in interventions or responses.
The preferred procedure randomly divides units into a training set J for discovering g and a non-overlapping test set I for estimating effects.The split explicitly separates codebook creation from codebook application.
Because g does not depend on test-set randomization, each test-set response depends only on its assigned treatment status, avoiding an analyst-induced SUTVA violation.The remaining dependence on training observations is treated as analogous to shaping the object of inquiry after a pre-test.
Exploration in training data followed by one-time test-set verification reduces overfitting and supports patterns that are more likely to replicate.The split permits considering several models without accidentally p-hacking, while locking g before test-set evaluation.
4.2 Text as treatment: For text as treatment, g can map documents to multiple binary latent treatments, supporting marginal and interaction questions that topic-model simplices do not straightforwardly permit.The AMCE summarizes one binary component’s effect while averaging over the distribution of other components.
4.3 Procedure: Validation for label fidelity should use close reading in the training set before the test set is opened.

5 Applications

The paper applies its text-based causal-inference framework to immigration attitudes and bureaucratic responses, using discovered textual outcomes or treatments with split-sample designs. Across the immigration experiments, criminal-history information consistently shifted respondents toward more severe punishment or deportation, while complaint features predicted whether the CFPB responded promptly.

The authors demonstrate the framework in two applications: an immigration survey experiment with text as outcome and a CFPB study with text as treatment.They describe the discovery process and emphasize that the procedure generalizes beyond the specific STM and sIBP models used.
5.1 Text as outcome: an experiment on immigration: The immigration application analyzes open-ended explanations of punishment preferences after respondents evaluated identical immigration offenses with or without prior criminal history.The initial telephone survey included 1,300 respondents; later experiments modified who answered the open-ended question, question wording, and survey order.
5.1 Text as outcome: an experiment on immigration: Across three experiments, the considerations Americans raised were surprisingly stable, while some new categories emerged.The authors report all three experiments to make the iterative research process transparent and reduce selective reporting based on results.
5.1 Text as outcome: an experiment on immigration: In Experiment 3, prior criminal history increased writing about maximal punishment, deportation, and sending the person back to their country of origin.The control group more often advocated allowing the person to stay or making punishment depend on the circumstances of the crime.
5.1 Text as outcome: an experiment on immigration: Experiments 1 and 2 produced qualitatively similar results: criminal-history information significantly increased advocacy for more severe punishment or deportation despite differences in g and respondent questioning.Each experiment used equal train and test proportions, fitting several models in training data before applying one selected model to test data.
5.2 Text as treatment: Consumer Financial Protection Bureau: The CFPB application examines which complaint features are associated with a timely resolution, finding that detailed feedback and loan-repayment complaints increase prompt-response probability.Prompt responses were less common for complaints involving debt collectors, including explicit collection attempts or threats involving credit reports.
5.2 Text as treatment: Consumer Financial Protection Bureau: The authors propose further CFPB analyses using incoming data to study temporal trends, new categories, and demographic differences in complaint language and response.They distinguish reestimating effects with an existing g from estimating a new g as categories change over time.

6 Conclusion

The paper develops a framework for causal inference with text, identifying risks from latent representation discovery and proposing split-sample solutions. It extends the framework beyond text to other latent measures while acknowledging unresolved design and efficiency challenges.

Text’s high dimensionality requires low-dimensional representations, but those representations can make causal inference difficult even in experiments.The framework centers on learning a representation that links raw text to a treatment or outcome.
The paper identifies Analyst-induced SUTVA violation as an identification threat and overfitting as an estimation threat.Both risks arise when the discovered codebook is entangled with effect estimation.
A split between training data for discovering g and test data for estimating causal effects addresses both identification and estimation concerns.The procedure separates discovery from causal-effect estimation.
Sequential experiments can reserve research degrees of freedom for discovering measures before rigorously testing relationships within experiments.This design explicitly separates measure discovery from confirmatory testing.
The authors note unresolved work on other causal designs, optimal training/test splits, and improving discovery efficiency for smaller datasets.Existing work without a train/test split is not thereby declared invalid, but the authors hope for increased use of split samples.
The framework applies broadly to latent representations such as democracy scores, voting ideal points, and manual content-analysis measures.The authors recommend considering train/test splits when developing future low-dimensional measures from high-dimensional data.

A Online Appendix

The online appendix supplies proofs, technical clarifications, machine-learning connections, procedural details, model descriptions, and supplementary experiment materials.

Section A.1 contains proofs and additional technical clarifications referenced in the main text.
Section A.2 develops additional connections to the machine-learning literature.
Section A.3 outlines the procedure and clarifies variance estimation.
Sections A.5 and A.6 provide details for the STM and SIBP models, respectively.
Section A.7 assesses STM stability across training and test splits.
Section A.8 collects additional materials from the experiments reported in Section 5.1.

A.1.1 Estimation with a true g

The appendix rejects the idea that discovery is estimating a single true codebook function for a dataset. Instead, the usefulness of a discovered representation depends on the research question and the quantity of interest.

The procedure can be viewed as an estimator G whose performance might be assessed against a hypothetical true function ˇg.
The authors argue that assuming a true ˇg is misguided because different research questions can make different text representations relevant.Their example considers responses to a knock and encouragement to vote, illustrating that the target quantity shapes the useful representation.

A.1.2 Proof: Identifying ATE with text as dependent variable

With text as a dependent variable, the appendix shows that a fixed codebook can preserve identification of the average treatment effect. A weaker condition preserves the effect for the focal category even if other categories differ.

The appendix proves that applying a codebook function g to text as a dependent variable preserves the ATE under stated assumptions.
ATE_k is defined as E[z_i,1,k − z_i,0,k], the average difference in category-k prevalence across treatment potential outcomes.
Randomization identifies the category-specific effect by equating the difference in expected coded outcomes with E[z_i,1,k − z_i,0,k].
The proof requires g to be fixed; otherwise the relevant expectation is undefined.
Identification can survive a weaker condition in which alternative codebooks agree with g on the focal category for every document.Other categories may differ arbitrarily, provided their total proportion remains equal under both functions.

A.1.3 Technical Definition of AISV

The Analyst-Induced SUTVA Violation (AISV) occurs when the text-reduction function g depends on treatment assignments and responses used to discover it, making a unit’s representation change under other units’ re-randomization. This dependence can create new measured outcomes or treatments across randomizations.

Formal definition: AISV arises when g is written as depending on both observation i’s text and the training set’s treatments and text-based responses.Re-randomizing at least one other training-set unit can therefore alter the function applied to observation i.
Formal definition: An AISV exists when observation i receives different lower-dimensional representations across randomizations even though its text remains unchanged.The difference results from changing the treatment allocation and responses used to discover g.
Implication: This dependence changes the expectations underlying causal estimators because different treatment allocations can imply different text-based categories, outcomes, or treatments.The problem matters especially when characterizing estimator bias or large-sample properties.

A.1.4 Assuming the AISV Away

Assuming AISV away requires stability of the discovered codebook across treatment allocations, but such conditions can be difficult to obtain and do not resolve overfitting. The paper instead separates discovery and evaluation through train-test splitting, then examines model stability and treatment-effect estimates across splits.

Assumption: Assuming AISV away requires the discovered codebook to remain unchanged across treatment allocations and training sets.The paper notes that the conditions for this stability can be surprisingly difficult to obtain, and the assumption does not solve overfitting.
Split-sample procedure: The recommended procedure randomly divides units into non-overlapping training and test sets, discovers g using training data only, and applies it to the test set for causal estimation.The test set is kept separate from discovery and used only once for estimation.
Validation: The workflow validates theoretical label fidelity during discovery, checks performance in the test set, and ideally replicates the full procedure in a new sample.If replication fails, the researchers should clarify what they would alter in the next experiment.
Structural Topic Model: In the Structural Topic Model, g maps held-out documents to topic proportions using training-estimated topic-word parameters and a selected prior.Because treatment is included among the covariates, the application uses the average prior rather than a covariate-specific prior.
Stability results: At 5,000 documents, topic estimates are reasonably stable, whereas at 1,000 documents they are less stable; warm spectral initialization improves stability at 5,000 documents.Near-perfect stability with a warm oracle start suggests initialization contributes to instability, although some instability at 1,000 documents may be unavoidable.
Treatment-effect results: Treatment-effect estimates vary substantially across splits, while warm starts produce relatively tighter estimates; confidence intervals need not have proper coverage because the estimand changes across splits.The simulations suggest that more stable initialization strategies could reduce this variability.